Newly-Developed Earphones Can Track Facial Expressions to Make Emojis

Over the decades, the input methods for technology have been becoming more user-friendly. From writing a whole program of command to mouse-click and the touch-screens– how we tell machines what we want has become less effort-intensive. Now imagine if you could make a facial expression that could be understood and translated by the computer too. This is now possible thanks to the newly-developed earphones that can track facial expressions.

Facial Expressions Tracker

It might be baffling to imagine a gadget that has been used for listening purposes becoming a facial expression tracker. The researchers at Cornell University decided to think outside the box and expanded the possible applications of the traditional wearable headphone or earphone device. Dubbed the C-Face, the ear-mountable system can monitor cheek contours and convert the wearer’s expression into an emoji. It is especially useful if the wearer does not want to hold up a camera in front, or has a messy background, or simply wants some privacy while conveying genuine expressions. The paper was presented at the Association for Computing Machinery Symposium on User Interface Software and Technology, held virtually on October 23rd, 2020.

How Does it Work?

The ear-mountable device employs two RGB cameras positioned below each ear. The cameras record changes in cheek contours corresponding to the user’s facial muscles movements in real-time. The cameras do this by capturing red and green bands of light, that change with facial muscle movements. These changes in light are in-turn interpreted by the camera to determine the kind of expression are made. This is interpreted as 2D images and is further analyzed by a convolutional neural network after they’ve been reconstructed. With the magic of computer vision and a deep learning model, the expression is translated from the 42 facial feature points representing the position and shape of the wearer’s mouth, eyes and eyebrows into an emoji or even a custom avatar of the user.

“This device is simpler, less obtrusive and more capable than any existing ear-mounted wearable technologies for tracking facial expressions,” has claimed Cheng Zhang, director of Cornell’s SciFi Lab and senior author of a paper on C-Face. “In previous wearable technology aiming to recognize facial expressions, most solutions needed to attach sensors on the face and even with so much instrumentation, they could only recognize a limited set of discrete facial expressions.”

C-Face relies only on the changes in facial contours to operate, thus it can capture facial expressions even when users are wearing masks. “When we perform a facial expression, our facial muscles stretch and contract. They push and pull the skin and affect the tension of nearby facial muscles. This effect causes the outline of the cheeks (contours) to alter from the point of view of the ear”, as explained by Zhang.

Applications of C-Face using Facial Expressions

The reconstructed facial expressions by the C-Face, represented by 42 feature points are so far translated to eight emojis, including “natural,” “angry” and “kissy-face”. Not only that but eight silent speech commands designed to control a music device, such as “play,” “next song” and “volume up” have been tested out as well. The mainstream use of remote learning could also be useful for instructors by allowing valuable student engagement during online lessons.

A device directly reading facial expressions means that commands can be quietly inputted into computers, like in libraries or other shared workspaces where people might not want to disturb others by speaking out loud. Physically challenged people can also benefit from C-Face if this technology is used with dedicated software designed to assist them with the routine use of smart things. Translating expressions into emojis could help those in virtual reality collaborations communicate more seamlessly, said Francois Guimbretière, professor of information science and a co-author of the C-Face paper. Coupled with a VR set, C-Face could give users a fully immersive experience in gaming as well.

Check out the official demo video here:

Experimental Results and Accuracy

The restrictions due to COVID-19 pandemic obstructed wide range testing. The researchers only managed to test the device on nine participants, including two of the study’s authors. They compared C-Face’s performance with a state-of-art computer vision library, which used front cameras to read facial landmarks the image of full face. They found the average error of the reconstructed landmarks to be under 0.8 mm. The participants reported the accuracy of C-Face and the accuracy was found to be 88% for emoji recognition and 85% for silent speech.

Limitations

Zhang expressed the earphones’ shortcomings admitting to its limited battery capacity. As its next step, the team plans to work on a sensing technology that uses less power.