Fu Foundation School of Engineering & Applied Science

What would you do if you walked up to a robot with a human-like head and it smiled at you first? You'd likely smile back and perhaps feel the two of you were genuinely interacting. But how does a robot know how to do this? Or a better question, how does it know to get you to smile back? What would you do if you walked up to a robot with a human-like head and it smiled at you first? You'd likely smile back and perhaps feel the two of you were genuinely interacting. But how does a robot know how to do this? Or a better question, how does it know to get you to smile back? While we're getting accustomed to robots that are adept at verbal communication, thanks in part to advancements in large language models like ChatGPT, their nonverbal communication skills, especially facial expressions, have lagged far behind. Designing a robot that can not only make a wide range of facial expressions but also know when to use them has been a daunting task. Tackling the challenge The Creative Machines Lab at Columbia Engineering has been working on this challenge for more than five years. In a new study published today in Science Robotics, the group unveils Emo, a robot that anticipates facial expressions and executes them simultaneously with a human. It has even learned to predict a forthcoming smile about 840 milliseconds before the person smiles, and to co-express the smile simultaneously with the person. The team, led by Hod Lipson, a leading researcher in the fields of artificial intelligence (AI) and robotics, faced two challenges: how to mechanically design an expressively versatile robotic face which involves complex hardware and actuation mechanisms, and knowing which expression to generate so that they appear natural, timely, and genuine. The team proposed training a robot to anticipate future facial expressions in humans and execute them simultaneously with a person. The timing of these expressions was critical -- delayed facial mimicry looks disingenuous, but facial co-expression feels more genuine since it requires correctly inferring the human's emotional state for timely execution. How Emo connects with you Emo is a human-like head with a face that is equipped with 26 actuators that enable a broad range of nuanced facial expressions. The head is covered with a soft silicone skin with a magnetic attachment system, allowing for easy customization and quick maintenance. For more lifelike interactions, the researchers integrated high-resolution cameras within the pupil of each eye, enabling Emo to make eye contact, crucial for nonverbal communication. The team developed two AI models: one that predicts human facial expressions by analyzing subtle changes in the target face and another that generates motor commands using the corresponding facial expressions. To train the robot how to make facial expressions, the researchers put Emo in front of the camera and let it do random movements. After a few hours, the robot learned the relationship between their facial expressions and the motor commands -- much the way humans practice facial expressions by looking in the mirror. This is what the team calls "self modeling" -- similar to our human ability to imagine what we look like when we make certain expressions. Then the team ran videos of human facial expressions for Emo to observe them frame by frame. After training, which lasts a few hours, Emo could predict people's facial expressions by observing tiny changes in their faces as they begin to form an intent to smile. "I think predicting human facial expressions accurately is a revolution in HRI. Traditionally, robots have not been designed to consider humans' expressions during interactions. Now, the robot can integrate human facial expressions as feedback," said the study's lead author Yuhang Hu, who is a PhD student at Columbia Engineering in Lipson's lab. "When a robot makes co-expressions with people in real-time, it not only improves the interaction quality but also helps in building trust between humans and robots. In the future, when interacting with a robot, it will observe and interpret your facial expressions, just like a real person." What's next The researchers are now working to integrate verbal communication, using a large language model like ChatGPT into Emo. As robots become more capable of behaving like humans, Lipson is well aware of the ethical considerations associated with this new technology. "Although this capability heralds a plethora of positive applications, ranging from home assistants to educational aids, it is incumbent upon developers and users to exercise prudence and ethical considerations," says Lipson, James and Sally Scapa Professor of Innovation in the Department of Mechanical Engineering at Columbia Engineering, co-director of the Makerspace at Columbia, and a member of the Data Science Institute. "But it's also very exciting -- by advancing robots that can interpret and mimic human expressions accurately, we're moving closer to a future where robots can seamlessly integrate into our daily lives, offering companionship, assistance, and even empathy. Imagine a world where interacting with a robot feels as natural and comfortable as talking to a friend."

These wide-ranging waves quickly link the specific constellations of brain regions that work in harmony to perform a task. In the space of just a few seconds, a person walking down a city block might check their phone, yawn, worry about making rent, and adjust their path to avoid a puddle. The smell from a food cart could suddenly conjure a memory from childhood, or they could notice a rat eating a slice of pizza and store the image as a new memory. For most people, shifting through behaviors quickly and seamlessly is a mundane part of everyday life. For neuroscientists, it's one of the brain's most remarkable capabilities. That's because different activities require the brain to use different combinations of its many regions and billions of neurons. How it manages to do this so rapidly has been an open question for decades. The Study In a paper published March 8 in Nature Human Behaviour, a team of researchers, led by Joshua Jacobs, associate professor of biomedical engineering at Columbia Engineering, shed new light on this question. By carefully monitoring neural activity of people who were recalling memories or forming new ones, the researchers managed to detect how a newly appreciated type of brainwave -- traveling waves -- influences the storage and retrieval of memories. "Broadly, we found that waves tended to move from the back of the brain to the front while patients were putting something into their memory," said the paper's co-author Uma R. Mohan, a postdoctoral researcher at NIH and former postdoctoral researcher in the Electrophysiology, Memory, and Navigation Laboratory at Columbia Engineering. "When patients were later searching to recall the same information, those waves moved in the opposite direction, from the front towards the back of the brain," she said. In the brains of some of the study's 93 participants, waves traveled in other directions. "There was a lot of diversity across patients, so we implemented a framework based on the direction an individual's oscillations 'preferred' to travel," Mohan said. The researchers say these findings advance fundamental neuroscience research and point toward diagnostic and therapeutic approaches for memory-related disorders. "We think the work may lead to new approaches for interfacing with the brain. By measuring the direction that a person's brain waves move, we may be able to predict their behavior," Jacobs said. The Challenge Brain waves are patterns of electrical oscillations that reflect the state of hundreds or thousands of individual neurons at a particular moment. One major question, which remains unsettled, is whether brain waves drive activity or simply occur as a byproduct of neural activity that was already happening. Researchers who study brain waves have tended to treat them as a stationary phenomenon that occurs in a particular region, noting when oscillations in multiple regions seem synchronized. In this study, Mohan and her colleagues contribute to a growing understanding of these oscillations differently, as "traveling waves" that spread across the brain's cortex, the outermost layer that supports higher cognitive processing. Mohan compares the traveling waves to the ripples that would spread outward after a pebble was thrown into a pond. "We're looking at neural oscillations not as independent stationary things but as things that are constantly and spontaneously moving across the brain in a dynamic way," Mohan said. This relatively new way of understanding brain waves is an exciting step in neuroscience because it offers a pathway to explaining how the brain quickly coordinates activity and shares information across multiple regions. The Experiments and Results This study drew on data from participants who were being treated for drug-resistant epilepsy at hospitals across the United States. The experiments occurred while the participants had grids or strips of electrodes temporarily implanted on the surface of the brain, beneath the skull, to determine where the patients' seizures arise. For the researchers, these electrodes offer the chance to perform experiments that wouldn't otherwise be feasible. "It's a rare opportunity to be able to see what's going on directly from the brain while the participants are engaged in different cognitive behaviors," Mohan said. During the experiments, researchers recorded the participants' brain activity while they performed tasks that required memorizing and recalling lists of words or letters. After the experiments, the researchers analyzed the brain activity from each participant in the context of what they were doing in the memory task and how well they performed. "I implemented a method to label waves traveling in one direction as basically 'good for putting something into memory.' Then we could see how the direction switched over the course of the task," Mohan said. This method builds on previous research from the Jacobs lab by expanding the mathematical framework used to make sense of the vast quantities of data these experiments produced. "The waves tended to go in the participant's encoding direction when that participant was putting something into memory and in the opposite direction right before they recalled the word," she said. " Overall, this new work links traveling waves to behavior by demonstrating that traveling waves propagate in different directions across the cortex for separate memory processes." The data also showed that participants tended to perform the memory task more accurately when the traveling waves were moving in the appropriate direction for memory storage and recall. "These findings shed light on the mechanisms that underlie memory processing. More broadly, they help us better understand how the brain supports a wide range of behaviors that involve precisely coordinated interactions between brain regions," Mohan said. Potential Impact and Future Directions As traveling waves are increasingly well understood, they could be the basis for a new class of diagnostic tools that recognize abnormal patterns in brain activity. There is also significant therapeutic potential. "If someone's waves are moving in the wrong direction when they're about to try to remember something, that might put them in a poor memory state," Mohan explained. "If you could apply stimulation in the right way, you could maybe push those waves to move in a different direction, bringing about a fundamentally different memory state." Advances in understanding traveling waves offer significant potential for human-computer interaction. In terms of both research and application, Mohan notes that memory is just the starting point. "I am interested in how characteristics of cortical traveling waves change to support a wide range of cognitive functions, including attention and associative memory," she said. "The direction of traveling wave propagation may tell us where information is moving across the brain at each moment, showing us how different parts of the brain transfer information during behavior," Jacobs said.

Researchers have used a soft, wearable robot to help a person living with Parkinson's walk without freezing. The robotic garment, worn around the hips and thighs, gives a gentle push to the hips as the leg swings, helping the patient achieve a longer stride. The device completely eliminated the participant's freezing while walking indoors, allowing them to walk faster and further than they could without the garment's help. Freezing is one of the most common and debilitating symptoms of Parkinson's disease, a neurodegenerative disorder that affects more than 9 million people worldwide. When individuals with Parkinson's disease freeze, they suddenly lose the ability to move their feet, often mid-stride, resulting in a series of staccato stutter steps that get shorter until the person stops altogether. These episodes are one of the biggest contributors to falls among people living with Parkinson's disease. Today, freezing is treated with a range of pharmacological, surgical or behavioral therapies, none of which are particularly effective. What if there was a way to stop freezing altogether? Researchers from the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and the Boston University Sargent College of Health & Rehabilitation Sciences have used a soft, wearable robot to help a person living with Parkinson's walk without freezing. The robotic garment, worn around the hips and thighs, gives a gentle push to the hips as the leg swings, helping the patient achieve a longer stride. The device completely eliminated the participant's freezing while walking indoors, allowing them to walk faster and further than they could without the garment's help. "We found that just a small amount of mechanical assistance from our soft robotic apparel delivered instantaneous effects and consistently improved walking across a range of conditions for the individual in our study," said Conor Walsh, the Paul A. Maeder Professor of Engineering and Applied Sciences at SEAS and co-corresponding author of the study. The research demonstrates the potential of soft robotics to treat this frustrating and potentially dangerous symptom of Parkinson's disease and could allow people living with the disease to regain not only their mobility but their independence. The research is published in Nature Medicine. For over a decade, Walsh's Biodesign Lab at SEAS has been developing assistive and rehabilitative robotic technologies to improve mobility for individuals' post-stroke and those living with ALS or other diseases that impact mobility. Some of that technology, specifically an exosuit for post-stroke gait retraining, received support from the Wyss Institute for Biologically Inspired Engineering, and was licensed and commercialized by ReWalk Robotics. In 2022, SEAS and Sargent College received a grant from the Massachusetts Technology Collaborative to support the development and translation of next-generation robotics and wearable technologies. The research is centered at the Move Lab, whose mission is to support advances in human performance enhancement with the collaborative space, funding, R&D infrastructure, and experience necessary to turn promising research into mature technologies that can be translated through collaboration with industry partners. This research emerged from that partnership. "Leveraging soft wearable robots to prevent freezing of gait in patients with Parkinson's required a collaboration between engineers, rehabilitation scientists, physical therapists, biomechanists and apparel designers," said Walsh, whose team collaborated closely with that of Terry Ellis, Professor and Physical Therapy Department Chair and Director of the Center for Neurorehabilitation at Boston University. The team spent six months working with a 73-year-old man with Parkinson's disease, who -- despite using both surgical and pharmacologic treatments -- endured substantial and incapacitating freezing episodes more than 10 times a day, causing him to fall frequently. These episodes prevented him from walking around his community and forced him to rely on a scooter to get around outside. In previous research, Walsh and his team leveraged human-in-the-loop optimization to demonstrate that a soft, wearable device could be used to augment hip flexion and assist in swinging the leg forward to provide an efficient approach to reduce energy expenditure during walking in healthy individuals. Here, the researchers used the same approach but to address freezing. The wearable device uses cable-driven actuators and sensors worn around the waist and thighs. Using motion data collected by the sensors, algorithms estimate the phase of the gait and generate assistive forces in tandem with muscle movement. The effect was instantaneous. Without any special training, the patient was able to walk without any freezing indoors and with only occasional episodes outdoors. He was also able to walk and talk without freezing, a rarity without the device. "Our team was really excited to see the impact of the technology on the participant's walking," said Jinsoo Kim, former PhD student at SEAS and co-lead author on the study. During the study visits, the participant told researchers: "The suit helps me take longer steps and when it is not active, I notice I drag my feet much more. It has really helped me, and I feel it is a positive step forward. It could help me to walk longer and maintain the quality of my life." "Our study participants who volunteer their time are real partners," said Walsh. "Because mobility is difficult, it was a real challenge for this individual to even come into the lab, but we benefited so much from his perspective and feedback." The device could also be used to better understand the mechanisms of gait freezing, which is poorly understood. "Because we don't really understand freezing, we don't really know why this approach works so well," said Ellis. "But this work suggests the potential benefits of a 'bottom-up' rather than 'top-down' solution to treating gait freezing. We see that restoring almost-normal biomechanics alters the peripheral dynamics of gait and may influence the central processing of gait control." The research was co-authored by Jinsoo Kim, Franchino Porciuncula, Hee Doo Yang, Nicholas Wendel, Teresa Baker and Andrew Chin. Asa Eckert-Erdheim and Dorothy Orzel also contributed to the design of the technology, as well as Ada Huang, and Sarah Sullivan managed the clinical research. It was supported by the National Science Foundation under grant CMMI-1925085; the National Institutes of Health under grant NIH U01 TR002775; and the Massachusetts Technology Collaborative, Collaborative Research and Development Matching Grant.