Mohammad Rafayet Ali

Associate Data Scientist
Sysco Corporation
Email: rafayet.ali[at]
Google Scholar Profile
LinkedIn Profile

I am an associate data scientist working at Sysco corporation. My work focuses on the personalization of shopping experience. Previously, I worked as a Postdoctoral Associate in Computer Science at the University of Rochester. I received my Ph.D. and Master's degree from the University of Rochester in 2020 and 2016 respectively. I received a Bachelor's degree from BUET in 2013. During my Ph.D. and postdoc I worked with Dr. Ehsan Hoque. My research is focused on Human-Computer Interaction (HCI). My work includes sensing, understanding and analyzing human affect, using machine learning techniques. In addition to affective computing, I work on applications focusing on improving communication skills and healthcare.


  • LISSA - Live Interactive Social Skills Assistance

    The LISSA system aims to help individuals improve their social skills by simulating face-to-face conversation and providing both real-time as well as post-session summary feedback on users’ nonverbal behavior. At the same time, it manages a fairly open-ended conversation with some degree of understanding. We have developed a HMM based model for generating real-time feedback.

    Watch the Video
  • PARK: Parkinson's Analysis with Remotely-Accessed Knowledge

    PARK is a web-based interface to collect the data from Parkinson’s patients. Using the online tool, patients are able to engage in a voice test and five motor tasks, from the standardized unified Parkinson's disease rating scale (UPDRS) test for diagnosing Parkinson's. These tasks are recorded via webcam and microphone of the user’s computer. The goal is to not only detect symptoms of Parkinson's, but to detect minute physical changes in the condition which provide important information about whether or not a patient requires a change in medication.

  • Aging and Engaging: a social conversational skills training program for older adults

    This project focuses on the older adults, who are socially isolated. The number of older adults in the US is 700 million. Social isolation can lead to many problems e.g., depression, suicidal tendency. This project is specially designed for older adults, allowing them to practice conversation with a virtual agent and receive feedback, which might help them starting a conversation with others reducing their isolation.

  • SOPHIE - Standardized Online Patient for Healthcare Interaction Education

    SOPHIE is a virtual patient, capable of having conversation with oncologist and provide feedback on their communication skills. SOPHIE presents herself as a late-stage cancer patient and provides opportunities for showing empathy. The goal is to improve the prognosis understanding among terminally ill patients for better heath-care decision making.

    Watch the Video
  • Cancer Patient Voice Project

    In this project we aim to identify the styles of coomunication between doctor and patient that help improve patient outcome and decision making. We analyze the text transcripts and audio recording between 386 late stage patients and 38 oncologists. Our analysis include, sentiment analysis, clustering of sentiment progression, speech act analysis.

  • ROC Comment

    ROC Comment is an automated interface for generating natural language comments on behavioral videos. We focus on the domain of public speaking. We collect a dataset of 196 public speaking videos from 49 individuals and gather 12,173 comments, generated by more than 500 independent human judges. We then train a k-Nearest-Neighbor (k-NN) based model by extracting prosodic (e.g., volume) and facial (e.g., smiles) features. Given a new video, we extract features and select the closest comments using k-NN model. We further filter the comments by clustering them using DBScan, and eliminating the outliers. We are working on generating comments using deep learning.



  1. M. Rafayet Ali, Taylor Myers, Ellen Wagner, Harshil Ratnu, Ray Dorsey, Ehsan Hoque, “Facial expressions can detect Parkinson’s disease: preliminary evidence from videos collected online.” Nature partner Journal Digital Medecine 4, 129 (2021). doi:10.1038/s41746-021-00502-8.
  2. M. Rafayet Ali, Taylan Sen, Qianyi Li, Raina Langevin, Taylor Myers, E. Ray Dorsey, Saloni Sharma, and Ehsan Hoque. “Analyzing Head Pose in Remotely Collected Videos of People with Parkinson’s Disease.” ACM Transactions on Computing for Healthcare 2, no. 4 (2021): 1-13.
  3. M. Rafayet Ali, Taylan Sen, Benjamin Kane, Shagun Bose, Thomas M. Carroll, Ronald Epstein, Lenhart Schubert, and Ehsan Hoque. “Novel Computational Linguistic Measures, Dialogue System and the Development of SOPHIE: Standardized Online Patient for Healthcare Interaction Education.” IEEE Transactions on Affective Computing. (IEEETAC)
  4. Wasifur Rahman, Sangwu Lee, Md Saiful Islam, Victor Nikhil Antony, Harshil Ratnu, M. Rafayet Ali, Abdullah Al Mamun et al. "Detecting Parkinson Disease Using a Web-Based Speech Task: Observational Study." Journal of Medical Internet Research 23, no. 10 (2021): e26305.
  5. 2020

  6. M. Rafayet Ali, Ehsan Hoque, Paul Duberstein, Lenhart Schubert, Seyedeh Zahra Razavi, Benjamin Kane, Caroline Silva, Jennifer Daks, Jennifer Daks, Kim Van Orden, “Aging and Engaging: A Pilot Randomized Controlled Trial of an Online Conversational Skills Coach for Older Adults”, American Journal of Geriatric Psychiatry. (AJGP)
  7. M. Rafayet Ali, Seyedeh Zahra Razavi, Raina Langevin, Ab-dullah Al Mamun, Benjamin Kane, Reza Rawassizadeh, Lenhart K Schubert, Ehsan Hoque. "A Virtual Conversational Agent for Teens with Autism Spectrum Disorder: Experimental Results and Design Lessons", 20th ACM International Conference on Intelligent Virtual Agents (IVA 2020)
  8. Dorsey, E., Omberg, L., Waddell, E., Adams, J.L., Adams, R., M. Rafayet Ali, Amodeo, K., Arky, A., Augustine, E.F., Dinesh, K. and Hoque, M.E., 2020. "Deep Phenotyping of Parkinson’s Disease." Journal of Parkinson's Disease
  9. M. Rafayet Ali, Daniel McDuff, Javier Hernandez, E. Ray Dorsey, and Ehsan Hoque, "Spatio-Temporal Attention and Magnification for Classification of Parkinson's Disease from Videos Collected via the Internet", IEEE Conference on Automatic Face and Gesture Recognition. (FG 2020)
  10. M. Tran, T K. Sen, K. G. Haut, M. Rafayet Ali, M. E. Hoque, "Are you really looking at me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze from Conventional Video", IEEE Transactions on Affective Computing.
  11. 2019

  12. Raina Langevin, M. Rafayet Ali, Taylan Sen, Christopher Snyder, Taylor Myers, E. Ray Dorsey, Ehsan Hoque, "The PARK Framework for Automated Analysis of Parkinson's Disease Characteristics", ACM International Joint Conference on Pervasive and Ubiquitous Computing. (UbiComp 2019)
  13. M. Rafayet Ali, Taylan Sen, Viet-Duy Nguyen, Reza Rawassizadeh, Paul Duberstein, Ronald Epstein, and M Ehsan Hoque, "What Computers Can Teach Us About Doctor-Patient Communication: Leveraging Gender Differences in Cancer Care." International Conference on Affective Computing & Intelligent Interaction. (ACII 2019)
  14. M. Rafayet Ali, "Online Virtual Standardized Patient for Communication Skills Training", ACM Intelligent User Interfaces (IUI 2019).
  15. Zahra Razavi, Lenhart Schubert, Benjamin Kane, M. Rafayet Ali, Kimberly A. Van Orden, and Tianyi Ma , "Dialogue Design and Management for Multi-Session Casual Conversation with Older Adults", User-Aware Conversational Agents in conjunction with ACM Intelligent User Interfaces (IUI 2019).
  16. 2018

  17. M. Rafayet. Ali, Taylan Sen, Dev Crasta, Viet-Duy Nguyen, Ronald Rogge, M Ehsan Hoque, “What, When, and Why of Facial Expressions: An Objective Analysis of Conversational Skills in Speed-Dating Videos”, 2018 IEEE Conference on Automatic Face and Gesture Recognition (FG 2018).
  18. M. Rafayet. Ali, K.V. Orden, K. Parkhurst, V. Nguyen, S. Liu, P. Duberstein, M. E. Hoque, "Aging and Engaging: A Social Conversational Skills Training Program for Older Adults", ACM Intelligent User Interfaces (IUI 2018).
  19. A. Singhal, M. Rafayet. Ali, R. Baten, C. Kurumada, E. Marvin, and M Ehsan Hoque, “Analyzing the Impact of Gender on the Automation of Feedback for Public Speaking” Automatic Face and Gesture Recognition (FG 2018).
  20. S. Samrose, R. Zhao, J. White, V. Li, L. Nova, Y. Lu, M. Rafayet. Ali, M. E. Hoque, "CoCo: Collaboration Coach for Understanding Team Dynamics during Video Conferencing", Interactive, Mobile, Wearable, and Ubiquitous Computing (UbiComp 2018).
  21. 2017

  22. Taylan Sen, Mohammad Rafayet Ali, Ron Epstein, Paul Duberstien, and Mohammed Ehsan Hoque, "Modeling Doctor-Patient Communication with Affective Text Analysis" Affective Computing and Intelligent Interaction (ACII 2017).
  23. Mohammad Rafayet Ali, and Mohammed Ehsan Hoque, "Social Skills Training with Virtual Assistant and Real-time Feedback" 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2017).
  24. Seyedeh Zahra Razavi, Lenhart K. Schubert, Mohammad Rafayet Ali, and Mohammed Ehsan Hoque, "Managing Casual Spoken Dialogue Using Flexible Schemas, Pattern Transduction Trees, and Gist Clauses," Conference on Advances in Cognitive Systems, (ACS 2017).
  25. Kimberly Van Orden, Mohammad Rafayet Ali, Kimberly Parkhurst, Paul Duberstein, Mohammed (Ehsan) Hoque, "Aging & Engaging: The Development of an Automated Tool to Teach Social Engagement Skills for Older Adults," Annual Meeting of the American Association for Geriatric Psychiatry, (AAGP 2017).
  26. 2016

  27. Mohammad Rafayet Ali, Facundo Ciancio, Ru Zhao, Iftekhar Naim, Mohammed (Ehsan) Hoque, "ROC Comment: Automated Descriptive and Subjective Captioning of Behavioral Videos." 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2016).
  28. Seyedeh Zahra Razavi, Mohammad Rafayet Ali, Lenhart Schubert, Mohammed (Ehsan) Hoque,"The LISSA Virtual Human and ASD Teens: An Overview of Initial Experiments." The Sixteenth International Conference on Intelligent Virtual Agents (IVA 2016).
  29. Dev Crasta, M. Rafayet Ali, Ronald D. Rogge, M. Ehsan Hoque, "Can Automated Conversation Skills Coaching Improve Interpersonal Attractiveness and Effectiveness?", 17th annual convention of Society for Personality and Social Psychology(SPSP), 2016
  30. 2015

  31. Ali, M.R.; Crasta, D.; Li Jin; Baretto, A.; Pachter, J.; Rogge, R.D.; Hoque, M.E., "LISSA- Live Interactive Social Skill Assistance," International Conference on Affective Computing and Intelligent Interaction (ACII), 2015, pp.173-179, 21-24 Sept. 2015
  32. Ali, Mohammad Rafayet. "Automated conversation skills assistant.", International Conference on IEEE Affective Computing and Intelligent Interaction (ACII), 2015.


Download my CV