Multimodal Musician Recognition

Hochenbaum, Jordan and Kapur, Ajay and Wright, Matthew

Proceedings of the International Conference on New Interfaces for Musical Expression

This research is an initial effort in showing how a multimodal approach can improve systems for gaining insight into a musician’s practice and technique. Embedding a variety of sensors inside musical instruments and synchronously recording the sensors’ data along with audio, we gather a database of gestural information from multiple performers, then use machine-learning techniques to recognize which musician is performing. Our multimodal approach (using both audio and sensor data) yields promising performer classification results, which we see as a first step in a larger effort to gain insight into musicians’ practice and technique.