Designing, Playing, and Performing with a Vision-based Mouth Interface

Lyons, Michael J. and Haehnel, Michael and Tetsutani, Nobuji

Proceedings of the International Conference on New Interfaces for Musical Expression

The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.