MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in eXtended Reality

Esther Gruy, and Florent Berthaut

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Extended Reality Interfaces open numerous opportunities for musical expression. One of them is 3D musical drawing, i.e., the ability to draw sonic and visual paths in virtual and physical spaces. However, in existing instruments (controllers, hand tracking) most of the interaction happens in the digital/virtual space which reduces primary audio and haptic feedback and might limit the expressiveness one has with physical drawing and acoustic instruments. In this paper, we propose a novel design approach which moves part of the 3D interaction back to the physical space for more intimate controls and more direct feedback, creating a physical interaction loop connected with the virtual interaction loop. We then present MagneTip, a first implementation of this approach for 3D musical drawing, which enables one or two handed interaction and combines co-localised and spatialised feedback. We believe that our physical interaction loop approach could be applied to other paradigms of 3D musical interaction.

Citation:

Esther Gruy, and Florent Berthaut. 2024. MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in eXtended Reality. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.13904764

BibTeX Entry:

  @article{nime2024_1,
 abstract = {Extended Reality Interfaces open numerous opportunities for musical expression. One of them is 3D musical drawing, i.e., the ability to draw sonic and visual paths in virtual and physical spaces. However, in existing instruments (controllers, hand tracking) most of the interaction happens in the digital/virtual space which reduces primary audio and haptic feedback and might limit the expressiveness one has with physical drawing and acoustic instruments. In this paper, we propose a novel design approach which moves part of the 3D interaction back to the physical space for more intimate controls and more direct feedback, creating a physical interaction loop connected with the virtual interaction loop. We then present MagneTip, a first implementation of this approach for 3D musical drawing, which enables one or two handed interaction and combines co-localised and spatialised feedback. We believe that our physical interaction loop approach could be applied to other paradigms of 3D musical interaction.},
 address = {Utrecht, Netherlands},
 articleno = {1},
 author = {Esther Gruy and Florent Berthaut},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.13904764},
 editor = {S M Astrid Bin and Courtney N. Reed},
 issn = {2220-4806},
 month = {September},
 numpages = {4},
 pages = {1--4},
 presentation-video = {},
 title = {MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in eXtended Reality},
 track = {Papers},
 url = {http://nime.org/proceedings/2024/nime2024_1.pdf},
 year = {2024}
}