Real-time Timbre Remapping with Differentiable DSP

Jordie Shier, Charalampos Saitis, Andrew Robertson, and Andrew McPherson

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Timbre is a primary mode of expression in diverse musical contexts. However, prevalent audio-driven synthesis methods predominantly rely on pitch and loudness envelopes, effectively flattening timbral expression from the input. Our approach draws on the concept of timbre analogies and investigates how timbral expression from an input signal can be mapped onto controls for a synthesizer. Leveraging differentiable digital signal processing, our method facilitates direct optimization of synthesizer parameters through a novel feature difference loss. This loss function, designed to learn relative timbral differences between musical events, prioritizes the subtleties of graded timbre modulations within phrases, allowing for meaningful translations in a timbre space. Using snare drum performances as a case study, where timbral expression is central, we demonstrate real-time timbre remapping from acoustic snare drums to a differentiable synthesizer modeled after the Roland TR-808.

Citation:

Jordie Shier, Charalampos Saitis, Andrew Robertson, and Andrew McPherson. 2024. Real-time Timbre Remapping with Differentiable DSP. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.13904884

BibTeX Entry:

  @article{nime2024_55,
 abstract = {Timbre is a primary mode of expression in diverse musical contexts. However, prevalent audio-driven synthesis methods predominantly rely on pitch and loudness envelopes, effectively flattening timbral expression from the input. Our approach draws on the concept of timbre analogies and investigates how timbral expression from an input signal can be mapped onto controls for a synthesizer. Leveraging differentiable digital signal processing, our method facilitates direct optimization of synthesizer parameters through a novel feature difference loss. This loss function, designed to learn relative timbral differences between musical events, prioritizes the subtleties of graded timbre modulations within phrases, allowing for meaningful translations in a timbre space. Using snare drum performances as a case study, where timbral expression is central, we demonstrate real-time timbre remapping from acoustic snare drums to a differentiable synthesizer modeled after the Roland TR-808.},
 address = {Utrecht, Netherlands},
 articleno = {55},
 author = {Jordie Shier and Charalampos Saitis and Andrew Robertson and Andrew McPherson},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.13904884},
 editor = {S M Astrid Bin and Courtney N. Reed},
 issn = {2220-4806},
 month = {September},
 numpages = {9},
 pages = {377--385},
 presentation-video = {https://youtu.be/OISsdBFzSh8?si=0axzhHntFfx_XUaV},
 title = {Real-time Timbre Remapping with Differentiable DSP},
 track = {Papers},
 url = {http://nime.org/proceedings/2024/nime2024_55.pdf},
 year = {2024}
}