Matthias Dorfer, Florian Henkel, Gerhard Widmer,
"Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game."
: Proceedings of 19th International Society for Music Information Retrieval Conference (ISMIR), 2018
Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game.
Sprache des Titels:
Proceedings of 19th International Society for Music Information Retrieval Conference (ISMIR)
Score following is the process of tracking a musical performance
(audio) with respect to a known symbolic representation
(a score). We start this paper by formulating
score following as a multimodal Markov Decision Process,
the mathematical foundation for sequential decision making.
Given this formal definition, we address the score following
task with state-of-the-art deep reinforcement learning
(RL) algorithms such as synchronous advantage actor
critic (A2C). In particular, we design multimodal RL
agents that simultaneously learn to listen to music, read
the scores from images of sheet music, and follow the audio
along in the sheet, in an end-to-end fashion. All this
behavior is learned entirely from scratch, based on a weak
and potentially delayed reward signal that indicates to the
agent how close it is to the correct position in the score.
Besides discussing the theoretical advantages of this learning
paradigm, we show in experiments that it is in fact superior
compared to previously proposed methods for score
following in raw sheet music images.