Real-time smile sonification using surface EMG signal and the evaluation of its usability

Abstract

We propose a real-time, interactive system for smile recognition and sonification using surface electromyography (EMG) signals. When a user smiles, a sound is played. The surface EMG signal is mapped to pitch using a conventional scale. The timbre of the sound is a synthetic sound mimicking bubbles. In the experiment, eight participants evaluated the effects of smile-based sonification feedback. Participants expressed smiles in a condition that there was feedback or no feedback. We investigated what type of effects the feedback had on smiling by analyzing surface EMG signals and interviewing the subjects. The results suggest that the sonified feedback could facilitate the expression of spontaneous smiles. In addition, results suggested that both the attack and release of a smile were similarly perceived using visual or auditory feedback.