Towards Note-Level Prediction for Networked Music Performance

Oda, Reid and Finkelstein, Adam and Fiebrink, Rebecca

Proceedings of the International Conference on New Interfaces for Musical Expression

The Internet allows musicians and other artists to collaborate remotely.However, network latency presents a fundamental challenge for remotecollaborators who need to coordinate and respond to each other’s performancein real time. In this paper, we investigate the viability of predictingpercussion hits before they have occurred, so that information about thepredicted drum hit can be sent over a network, and the sound can be synthesizedat a receiver’s location at approximately the same moment the hit occurs atthe sender’s location. Such a system would allow two percussionists to playin perfect synchrony despite the delays caused by computer networks. Toinvestigate the feasibility of such an approach, we record vibraphone malletstrikes with a high-speed camera and track the mallet head position. We showthat 30 ms before the strike occurs, it is possible to predict strike time andvelocity with acceptable accuracy. Our method fits a second-order polynomial tothe data to produce a strike time prediction that is within the bounds ofperceptual synchrony, and a velocity estimate that will enable the soundpressure level of the synthesized strike to be accurate within 3 dB.