SnakeSynth: New Interactions for Generative Audio Synthesis

Eric Easthope

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

I present "SnakeSynth," a web-based lightweight audio synthesizer that combines audio generated by a deep generative model and real-time continuous two-dimensional (2D) input to create and control variable-length generative sounds through 2D interaction gestures. Interaction gestures are touch and mobile-compatible and made with analogies to strummed, bowed, brushed, and plucked musical instrument controls. Point-and-click and drag-and-drop gestures directly control audio playback length and intensity. I show that I can modulate sound length and intensity by interacting with a programmable 2D grid and leveraging the speed and ubiquity of web browser-based audio and hardware acceleration to generate time-varying high-fidelity sounds with real-time interactivity. SnakeSynth adaptively reproduces and interpolates between sounds encountered during model training, notably without long training times, and I briefly discuss possible futures for deep generative models as an interactive paradigm for musical expression.

Citation

Eric Easthope. 2023. SnakeSynth: New Interactions for Generative Audio Synthesis. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.11189319 [PDF]

BibTeX Entry

@inproceedings{nime2023_90,
 abstract = {I present "SnakeSynth," a web-based lightweight audio synthesizer that combines audio generated by a deep generative model and real-time continuous two-dimensional (2D) input to create and control variable-length generative sounds through 2D interaction gestures. Interaction gestures are touch and mobile-compatible and made with analogies to strummed, bowed, brushed, and plucked musical instrument controls. Point-and-click and drag-and-drop gestures directly control audio playback length and intensity. I show that I can modulate sound length and intensity by interacting with a programmable 2D grid and leveraging the speed and ubiquity of web browser-based audio and hardware acceleration to generate time-varying high-fidelity sounds with real-time interactivity. SnakeSynth adaptively reproduces and interpolates between sounds encountered during model training, notably without long training times, and I briefly discuss possible futures for deep generative models as an interactive paradigm for musical expression.},
 address = {Mexico City, Mexico},
 articleno = {90},
 author = {Eric Easthope},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.11189319},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {8},
 pages = {612--619},
 title = {SnakeSynth: New Interactions for Generative Audio Synthesis},
 track = {Work in Progress},
 url = {http://nime.org/proceedings/2023/nime2023_90.pdf},
 year = {2023}
}