Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders

Tim Murray-Browne, and Panagiotis Tigas

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2021
• Location: Shanghai, China
• Article Number: 66
• DOI: 10.21428/92fbeb44.9d4bcd4b (Link to paper)
• PDF link
• Presentation Video

Abstract:

In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator [1] and MIMIC [2] allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.

Citation:

BibTeX Entry:

  @inproceedings{NIME21_66,
 abstract = {In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator [1] and MIMIC [2] allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.},
 address = {Shanghai, China},
 articleno = {66},
 author = {Murray-Browne, Tim and Tigas, Panagiotis},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.9d4bcd4b},
 issn = {2220-4806},
 month = {June},
 presentation-video = {https://youtu.be/zBOHWyIGaYc},
 title = {Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders},
 url = {https://nime.pubpub.org/pub/latent-mappings},
 year = {2021}
}