The National Academies feature ROC HCI Lab

September 1, 2020

Decoding the Unspoken Ways We Communicate

Feature Story | August 31, 2020

By Sara Frueh

Ehsan Hoque, a National Academy of Medicine Emerging Leader, harnesses nonverbal cues to improve health and human connection

Each of us has 43 muscles in our face, and we use those muscles to create 10,000 unique combinations of expressions — all of which subtly convey important information to other people during conversations, explains computer scientist Ehsan Hoque. Our physical gestures, language, and tone of voice add further layers of complexity.

“Anthropologist Edward Sapir said that when we speak to each other, there is a secret code that we all understand and comply with, but we don’t know how it works,” says Hoque, one of the 2020-2021 National Academy of Medicine Emerging Leaders in Health and Medicine.

In his research at the University of Rochester, he uses advanced computer technologies to analyze this unspoken language. “I want to be able to understand that code, using computation as a lens, while staying mindful of their ethical and appropriate uses.”

Advances in machine learning and artificial intelligence now allow the quick analysis of an unprecedented amount of human interaction data, enabling Hoque and his colleagues to gradually unravel the hidden patterns in communication, breaking new ground in human behavior research.

Hoque’s work aims not just to understand nonverbal communication but also to develop tools that channel those insights to improve human health and well-being. “As I started to do this, I realized there is a huge application on health … from autism, to Parkinson’s, to helping doctors improve quality of care and honor individual preferences of end-of-life patients, to helping elderly who might have difficulty with social communication due to social isolation, to PTSD, while improving equity and access.”

(clockwise, from upper left) Teenager diagnosed with Autism Spectrum Disorder interacting an online avatar called Live Interactive Social Skills Assistant (LISSA), which can sense and interpret the facial expressions, language, and intonation and can provide real-time feedback on smiles, eye-contact, gestures, and volume modulation; Hoque interacting with MACH (My Automated Conversation coacH); Hoque giving presentation; an older adult interacting with an online system called Aging and Engaging, which can carry on a conversation and provide feedback on eye contact, positivity, smile, and volume modulation; a physician practicing end-of-life communication with SOPHIE virtual patient

Building Skills to Connect with Others

One application Hoque has developed assists people with autism and developmental disabilities, who may find it challenging to navigate the unwritten rules of communication. “Many of these individuals don’t realize that they’re ‘monologuing,’ or that they’re not taking turns, or that they have a flat tone of voice.”

These individuals may want to seek help but fear the social stigma of doing so, Hoque explains. So he developed a virtual “conversation coach” to offer them practice. “We built technology where individuals — who may not want to seek help with a human just yet — can take the first step of getting some feedback on their nonverbal skills.” 

Using an avatar as a conversation partner, people can practice as often as they wish, get private feedback from the computer on their nonverbal communication signals, and control their own data.

The inspiration for this application was personal, Hoque explains. “I was blessed with a brother in 1999 who has Down syndrome and autism; he lives with me, and I am his primary caregiver,” he said. “So I deal with this every single day, and as a result I saw an opportunity to use technology to help someone with social skills.”

Similar technology, known as “aging and engaging,” is being used to help elderly people keep or build the skills they need to stay connected to others. Some aging people may become isolated due to immobility (or a pandemic) and forget the basic skills needed to successfully connect with other people, says Hoque; interacting with a virtual conversation partner can help them retain or regain their ability to do so.  

Using Sketches to Screen for PTSD

Another innovative application Hoque and his colleagues are developing uses a specific form of nonverbal communication — drawing — to help screen people for post-traumatic stress disorder (PTSD). The project is a collaboration with Bangladesh University of Engineering and Technology.  Bangladesh now houses over 900,000 stateless Rohingya refugees, half of whom are women and children, and who lack access to basic health care.

While PTSD is typically diagnosed through an interview in a clinical setting, many Rohingya refugees arrived from Myanmar and don’t speak the local language, says Hoque. “If there’s a language barrier, you won’t get the right information,” he says. “People may not open up to share their personal, and often traumatic, experiences.”  

Drawings, however, can sidestep language barriers by adding a nonverbal component. “Research shows, based on art therapy, that when people have been subjected to abuse, there are greater recurring graphic forms — disembodied eyes, and wedges — in the sketch that are visible and further quantifiable,” says Hoque.

Hoque and his colleagues asked a group of people who had PTSD to sketch something, and people without PTSD to sketch the same thing, and then identified differences in the patterns using image-processing algorithms. The algorithm was able to identify the subtle yet significant differences in the number of corners, the number of strokes, and their characteristics in the free-hand sketches to successfully distinguish between individuals with PTSD and without.

These findings can now be incorporated in touch-sensitive smartphones that allow people, particularly those from marginalized communities, to perform a free-hand sketch and potentially receive a referral to see a clinician.

Improving Communication at the End of Life

Another element of Hoque’s work seeks to strengthen communication in a difficult and delicate context — end-of-life health care.

Often there is a mismatch between a patient’s understanding of his or her prognosis and their doctor’s understanding of it. “There are huge discrepancies,” says Hoque. And these misunderstandings can then shape patient’s decisions. “Patients are opting for aggressive treatment that they later regret.”  For example, a patient with advanced cancer may not understand the severity of his prognosis and opt for treatment that worsens his quality of life without extending it very much. 

Hoque wants to change that by helping doctors communicate more effectively. “If I’m an oncologist, how do I give this information so that the patient understands it and is fully informed and part of the decision-making process?”

By analyzing over 400 conversations between oncologists and final stage cancer patients, Hoque and his team have identified which communication styles are more and less effective at improving patients’ understanding. “When doctors lecture patients — delivering a lot of information without giving the patient a chance to ask questions or to respond — miscommunication occurs,” he says. They have also found that doctors who use a more dynamic speaking style, with positive language at the beginning and at the end of a conversation, had more success in conveying prognoses to their patients.  

Hoque and his colleagues have built an online virtual patient, SOPHIE — Standardized Online Patient for Health-Care Interaction Education — so that doctors can practice communicating effectively using a laptop and learn whether they are inadvertently doing things that hinder patients’ understanding, such as lecturing or speaking too fast. 

Here, too, the motivation for the application sprang from one of Hoque’s close relationships. “I lost my mother to cancer, and when we lost her she was going through the third line of chemotherapy,” he says. “The oncologist was giving us cues — ‘every day is important,’ ‘she has an extensive disease’ — but we just didn’t get it, and we kept on pursuing aggressive chemotherapy, impacting the quality of her life.”

His mother also had Parkinson’s disease, which prompted Hoque to develop another tool, known as PARK (Parkinson’s Analysis with Remote Kinetic-tasks) that will let patients perform a neurological test to assess the severity of their symptoms, anytime and anywhere, using a webcam-enabled computer. This step will allow their medication doses to be fine-tuned more frequently, which helps the medication keep its therapeutic power longer. 

In his teaching, Hoque wants his students to draw from their own experiences and motivations in a similar way. “When you allow students to follow their intellectual curiosity, they become a force, and the best thing you can do is to simply get out of their way.”

Learn more about Hoque’s research and about the National Academy of Medicine Emerging Leaders in Health and Medicine Program.