Movement to emotions to music: using whole body emotional expression as an interaction for electronic music generation

Alexis Clay, Nadine Couture, Myriam Desainte-Catherine, Pierre-Henri Vulliard, Joseph Larralde, and Elodie Decarsin

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

The augmented ballet project aims at gathering research from several fields and directing them towards a same application case: adding virtual elements (visual and acoustic) to a dance live performance, and allowing the dancer to interact with them. In this paper, we describe a novel interaction that we used in the frame of this project: using the dancer's movements to recognize the emotions he expresses, and use these emotions to generate musical audio flows evolving in real-time. The originality of this interaction is threefold. First, it covers the whole interaction cycle from the input (the dancer's movements) to the output (the generated music). Second, this interaction isn't direct but goes through a high level of abstraction: dancer's emotional expression is recognized and is the source of music generation. Third, this interaction has been designed and validated through constant collaboration with a choreographer, culminating in an augmented ballet performance in front of a live audience.

Citation:

Alexis Clay, Nadine Couture, Myriam Desainte-Catherine, Pierre-Henri Vulliard, Joseph Larralde, and Elodie Decarsin. 2012. Movement to emotions to music: using whole body emotional expression as an interaction for electronic music generation. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1178237

BibTeX Entry:

  @inproceedings{Clay2012,
 abstract = {The augmented ballet project aims at gathering research from several fields and directing them towards a same application case: adding virtual elements (visual and acoustic) to a dance live performance, and allowing the dancer to interact with them. In this paper, we describe a novel interaction that we used in the frame of this project: using the dancer's movements to recognize the emotions he expresses, and use these emotions to generate musical audio flows evolving in real-time. The originality of this interaction is threefold. First, it covers the whole interaction cycle from the input (the dancer's movements) to the output (the generated music). Second, this interaction isn't direct but goes through a high level of abstraction: dancer's emotional expression is recognized and is the source of music generation. Third, this interaction has been designed and validated through constant collaboration with a choreographer, culminating in an augmented ballet performance in front of a live audience.},
 address = {Ann Arbor, Michigan},
 author = {Alexis Clay and Nadine Couture and Myriam Desainte-Catherine and Pierre-Henri Vulliard and Joseph Larralde and Elodie Decarsin},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1178237},
 issn = {2220-4806},
 keywords = {Interactive sonification, motion, gesture and music, interaction, live performance, musical human-computer interaction},
 publisher = {University of Michigan},
 title = {Movement to emotions to music: using whole body emotional expression as an interaction for electronic music generation},
 url = {http://www.nime.org/proceedings/2012/nime2012_180.pdf},
 year = {2012}
}