SoundMorphTPU: Exploring Gesture Mapping in Deformable Interfaces for Music Interaction

Zhen Wu, Ze Gao, Hua Xu, Xingxing Yang, and Tristan Braud

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Deformable interface is an emerging field with significant potential for use in computing applications, particularly in the design of Digital Music Instruments (DMIs). While prior works have investigated the design of gestural input for deformable interfaces and developed novel musical interactions, there remains a gap in understanding the tangible gestures as input and their corresponding output from the user's perspectives. This study explores the relationship between gestural input and the output of a deformable interface for multi-gestural music interaction. Following a pilot study to explore materials and their corresponding intuitive gestures with participants, we develop a TPU fabric interface as a probe to investigate this question in the context of musical interaction. Through user engagement with the probe as a sound control, we discovered that the input-output relationship between gestures and the sound can have meaningful implications for users' embodied interaction with the system. Our research deepens the understanding of designing deformable interfaces and their capacity to enhance embodied experiences in music interaction scenarios.

Citation:

Zhen Wu, Ze Gao, Hua Xu, Xingxing Yang, and Tristan Braud. 2024. SoundMorphTPU: Exploring Gesture Mapping in Deformable Interfaces for Music Interaction. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.13904891

BibTeX Entry:

  @article{nime2024_58,
 abstract = {Deformable interface is an emerging field with significant potential for use in computing applications, particularly in the design of Digital Music Instruments (DMIs). While prior works have investigated the design of gestural input for deformable interfaces and developed novel musical interactions, there remains a gap in understanding the tangible gestures as input and their corresponding output from the user's perspectives. This study explores the relationship between gestural input and the output of a deformable interface for multi-gestural music interaction. Following a pilot study to explore materials and their corresponding intuitive gestures with participants, we develop a TPU fabric interface as a probe to investigate this question in the context of musical interaction. Through user engagement with the probe as a sound control, we discovered that the input-output relationship between gestures and the sound can have meaningful implications for users' embodied interaction with the system. Our research deepens the understanding of designing deformable interfaces and their capacity  to enhance embodied experiences in music interaction scenarios.},
 address = {Utrecht, Netherlands},
 articleno = {58},
 author = {Zhen Wu and Ze Gao and Hua Xu and Xingxing Yang and Tristan Braud},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.13904891},
 editor = {S M Astrid Bin and Courtney N. Reed},
 issn = {2220-4806},
 month = {September},
 numpages = {12},
 pages = {395--406},
 presentation-video = {https://youtu.be/JkGwgeSRY4M?si=ohtTy-kiQ2JtjfvR},
 title = {SoundMorphTPU: Exploring Gesture Mapping in Deformable Interfaces for Music Interaction},
 track = {Papers},
 url = {http://nime.org/proceedings/2024/nime2024_58.pdf},
 year = {2024}
}