Mohammad Rafayet Ali

PhD Student
Computer Science, University of Rochester
Email: mali7(at)cs(dot)rochester(dot)edu
Google Scholar Profile

I am a PhD student of Computer Science in University of Rochester. My research is focused on Human Computer Interaction (HCI). Currently I am working under Dr. Ehsan Hoque. I do research and develop technologies, which helps people improve social skills. My work includes understanding human emotions, using machine learning techniques to facilitate improving social skills, and designing effective feedback mechanism for helping people.

News and Talks

  • Talk - IUI 2019, Online Virtual Standardized Patient for Communication Skills Training. Los Angeles March 17, 2019. Slides
  • Talk - IUI 2018, Aging and Engaging: A Social Conversational Skills Training Program for Older Adults. Tokyo March 8, 2018.
  • Talk - 2017 REU symposium at RIT. June 30, 2017. Slides
  • Talk - The 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017) May 30 –June 3. Slides
  • Talk - National Science Foundation Research Traineeship Program (NRT) Slides
  • Talk - ROC Comment: Automated Descriptive and Subjective Captioning of Behavioral Videos.(UbiComp 2016 Germany) Slides
  • Talk - LISSA- Live Interactive Social Skill Assistance. (ACII 2015 China) Slides
  • Projects

    LISSA - Live Interactive Social Skills Assistance

    The LISSA system aims to help individuals improve their social skills by simulating face-to-face conversation and providing both real-time as well as post-session summary feedback on users’ nonverbal behavior. At the same time, it manages a fairly open-ended conversation with some degree of understanding. We have developed a HMM based model for generating real-time feedback.

    PARK: Parkinson's Analysis with Remotely-Accessed Knowledge

    PARK is a web-based interface to collect the data from Parkinson’s patients. Using the online tool, patients are able to engage in a voice test and five motor tasks, from the standardized unified Parkinson's disease rating scale (UPDRS) test for diagnosing Parkinson's, that they would otherwise complete in the doctor's office. These tasks are recorded via webcam and microphone of the user’s computer. Facial and motion features are extracted from the videos. The goal is to not only detect symptoms of Parkinson's, but to detect minute physical changes in the condition which provide important information about whether or not a patient requires a change in medication. Doctors would then be able to make timely and responsive decisions.

    ROC Comment

    ROC Comment is an automated interface for generating natural language comments on behavioral videos. We focus on the domain of public speaking, which many people consider their greatest fear. We collect a dataset of 196 public speaking videos from 49 individuals and gather 12,173 comments, generated by more than 500 independent human judges. We then train a k-Nearest-Neighbor (k-NN) based model by extracting prosodic (e.g., volume) and facial (e.g., smiles) features. Given a new video, we extract features and select the closest comments using k-NN model. We further filter the comments by clustering them using DBScan, and eliminating the outliers. We are working on generating comments using deep learning.

    Cancer Patient Voice Project

    We present a method of automatic analysis of doctor-patient communication and present findings after applying this methodology in a post hoc study of communication between oncologists and their cancer patients (N=122). We analyzed several features of each participant in the conversation including the number of words spoken, the average positive/negative sentiment expressed, the number of questions asked, and the word diversity (unique word count). We found that the number of words spoken by the doctor is correlated with the highest doctor communication ability ratings made by patients. We additionally found that unsupervised clustering of conversation features into “styles” identified that certain styles are associated with higher communication ratings. Two well-defined styles emerged when clustering based on doctor word diversity and doctor sentiment: a high word diversity-neutral sentiment style, which were associated with higher ratings, and a low word diversity-positive sentiment style with lower average ratings. Machine learning models were trained to automatically predict whether a doctor-patient interaction will be rated high or not with a best-performing 71% test set accuracy.

    Aging and Engaging

    This project focuses on the older adults, who are socially isolated. The number of older adults in the US is 700 million. Social isolation can lead to many problems e.g., depression, suicidal tendency. This project is specially designed for older adults, allowing them to practice conversation with a virtual agent and receive feedback, which might help them starting a conversation with others reducing their isolation.

    ROC Speak Real-time

    Imagine you need to give a public speech and you need to practice. You are too shy to share your practice video or maybe you don’t have friends, who can give you some objective feedback. Using this system you can practice your public speech and see how your nonverbal behaviors are changing in real-time.



    1. Raina Langevin, M. Rafayet Ali, Taylan Sen, Christopher Snyder, Taylor Myers, E. Ray Dorsey, Ehsan Hoque, “Could I have Parkinson’s? The PARK Framework for Automated Assessment of Parkinson’s Disease”, ACM International Joint Conference on Pervasive and Ubiquitous Computing. (UbiComp 2019)
    2. M. Rafayet Ali, "Online Virtual Standardized Patient for Communication Skills Training", ACM Intelligent User Interfaces (IUI 2019).
    3. Zahra Razavi, Lenhart Schubert, Benjamin Kane, M. Rafayet Ali, Kimberly A. Van Orden, and Tianyi Ma , "Dialogue Design and Management for Multi-Session Casual Conversation with Older Adults", User-Aware Conversational Agents in conjunction with ACM Intelligent User Interfaces (IUI 2019).
    4. 2018

    5. M. Rafayet. Ali, Taylan Sen, Dev Crasta, Viet-Duy Nguyen, Ronald Rogge, M Ehsan Hoque, “What, When, and Why of Facial Expressions: An Objective Analysis of Conversational Skills in Speed-Dating Videos”, 2018 IEEE Conference on Automatic Face and Gesture Recognition (FG 2018).
    6. M. Rafayet. Ali, K.V. Orden, K. Parkhurst, V. Nguyen, S. Liu, P. Duberstein, M. E. Hoque, "Aging and Engaging: A Social Conversational Skills Training Program for Older Adults", ACM Intelligent User Interfaces (IUI 2018).
    7. A. Singhal, M. Rafayet. Ali, R. Baten, C. Kurumada, E. Marvin, and M Ehsan Hoque, “Analyzing the Impact of Gender on the Automation of Feedback for Public Speaking” Automatic Face and Gesture Recognition (FG 2018).
    8. S. Samrose, R. Zhao, J. White, V. Li, L. Nova, Y. Lu, M. Rafayet. Ali, M. E. Hoque, "CoCo: Collaboration Coach for Understanding Team Dynamics during Video Conferencing", Interactive, Mobile, Wearable, and Ubiquitous Computing (UbiComp 2018).
    9. 2017

    10. Taylan Sen, Mohammad Rafayet Ali, Ron Epstein, Paul Duberstien, and Mohammed Ehsan Hoque, "Modeling Doctor-Patient Communication with Affective Text Analysis" Affective Computing and Intelligent Interaction (ACII 2017).
    11. Mohammad Rafayet Ali, and Mohammed Ehsan Hoque, "Social Skills Training with Virtual Assistant and Real-time Feedback" 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2017).
    12. Seyedeh Zahra Razavi, Lenhart K. Schubert, Mohammad Rafayet Ali, and Mohammed Ehsan Hoque, "Managing Casual Spoken Dialogue Using Flexible Schemas, Pattern Transduction Trees, and Gist Clauses," Conference on Advances in Cognitive Systems, (ACS 2017).
    13. Kimberly Van Orden, Mohammad Rafayet Ali, Kimberly Parkhurst, Paul Duberstein, Mohammed (Ehsan) Hoque, "Aging & Engaging: The Development of an Automated Tool to Teach Social Engagement Skills for Older Adults," Annual Meeting of the American Association for Geriatric Psychiatry, (AAGP 2017).
    14. 2016

    15. Mohammad Rafayet Ali, Facundo Ciancio, Ru Zhao, Iftekhar Naim, Mohammed (Ehsan) Hoque, "ROC Comment: Automated Descriptive and Subjective Captioning of Behavioral Videos." 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2016).
    16. Seyedeh Zahra Razavi, Mohammad Rafayet Ali, Lenhart Schubert, Mohammed (Ehsan) Hoque,"The LISSA Virtual Human and ASD Teens: An Overview of Initial Experiments." The Sixteenth International Conference on Intelligent Virtual Agents (IVA 2016).
    17. Dev Crasta, M. Rafayet Ali, Ronald D. Rogge, M. Ehsan Hoque, "Can Automated Conversation Skills Coaching Improve Interpersonal Attractiveness and Effectiveness?", 17th annual convention of Society for Personality and Social Psychology(SPSP), 2016
    18. 2015

    19. Ali, M.R.; Crasta, D.; Li Jin; Baretto, A.; Pachter, J.; Rogge, R.D.; Hoque, M.E., "LISSA- Live Interactive Social Skill Assistance," International Conference on Affective Computing and Intelligent Interaction (ACII), 2015, pp.173-179, 21-24 Sept. 2015
    20. Ali, Mohammad Rafayet. "Automated conversation skills assistant.", International Conference on IEEE Affective Computing and Intelligent Interaction (ACII), 2015.

    Get in touch

    Send me an email if you have any question.