Collaborative Musical Performances with Automatic Harp Based on Image Recognition and Force Sensing Resistors

Yupu Lu, Yijie Wu, and Shijie Zhu

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

In this paper, collaborative performance is defined as the performance of the piano by the performer and accompanied by an automatic harp. The automatic harp can play music based on the electronic score and change its speed according to the speed of the performer. We built a 32-channel automatic harp and designed a layered modular framework integrating both hardware and software, for experimental real-time control protocols. Considering that MIDI keyboard lacking information of force (acceleration) and fingering detection, both of which are important for expression, we designed force-sensor glove and achieved basic image recognition. They are used to accurately detect speed, force (corresponding to velocity in MIDI) and pitch when a performer plays the piano.

Citation:

Yupu Lu, Yijie Wu, and Shijie Zhu. 2019. Collaborative Musical Performances with Automatic Harp Based on Image Recognition and Force Sensing Resistors. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.3672846

BibTeX Entry:

  @inproceedings{Lu2019,
 abstract = {In this paper, collaborative performance is defined as the performance of the piano by the performer and accompanied by an automatic harp. The automatic harp can play music based on the electronic score and change its speed according to the speed of the performer. We built a 32-channel automatic harp and designed a layered modular framework integrating both hardware and software, for experimental real-time control protocols. Considering that MIDI keyboard lacking information of force (acceleration) and fingering detection, both of which are important for expression, we designed force-sensor glove and achieved basic image recognition. They are used to accurately detect speed, force (corresponding to velocity in MIDI) and pitch when a performer plays the piano.},
 address = {Porto Alegre, Brazil},
 author = {Yupu Lu and Yijie Wu and Shijie Zhu},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.3672846},
 editor = {Marcelo Queiroz and Anna Xambó Sedó},
 issn = {2220-4806},
 month = {June},
 pages = {7--8},
 publisher = {UFRGS},
 title = {Collaborative Musical Performances with Automatic Harp Based on Image Recognition and Force Sensing Resistors},
 url = {http://www.nime.org/proceedings/2019/nime2019_paper002.pdf},
 year = {2019}
}