The Sonification and Learning of Human Motion
Smith, Kevin M.
MetadataShow full item record
This paper examines how sonification can be used to help a student emulate the complex motion of a teacher with increasing spatial and temporal accuracy. The system captures a teacher’s motion in real-time and generates a 3-D motion path, which is recorded along with a reference sound. A student then attempts to perform the motion and thus recreate the teacher’s reference sound. The student’s synthesized sound will dynamically approach the teacher’s sound as the student’s movement becomes more accurate. Several types of sound mappings which simultaneously represent time and space deviations are explored. For the experimental platform, a novel system that uses low-cost camera-based motion capture hardware and open source software has been developed. This work can be applied to diverse areas such as rehabilitation and physiotherapy, performance arts and aiding the visually impaired.