The Science of Music

There is a research field called music information processing. It involves using computers to perform activities we normally enjoy, such as listening to and creating music. The methods differ depending on whether the target is analog (e.g., actual sound) or digital (e.g., sheet music). I will introduce this field with examples from our laboratory.

Let's consider the task of automatically transcribing currently playing music into sheet music. This is a challenge that has been pursued for over half a century. While it is simple for monophonic, single-instrument pieces, the difficulty increases significantly when dealing with complex mixtures of sounds like in an orchestra, as it requires sound source separation and beat recognition for each voice part. Research in this area continues to this day. In such music recognition, methods centered on signal processing have long been pursued. In our laboratory, a student is exploring the theory that processing instrument sounds with signal processing results in an impulse train. Furthermore, there have been new developments incorporating recent deep learning techniques. A 2024 master's thesis showed that recognition accuracy can be improved by considering the phase of the sound, which has been ignored in conventional signal processing. Additionally, acoustics can be visualized like a voiceprint, and a student who graduated in 2024 created a system that uses this visualization to easily assign chord types.

As an example of sheet music-to-sheet music conversion, a student created a system that arranges music to be playable on a music box. Music boxes have their own instrumental constraints, such as a narrow pitch range and the inability to play many notes in rapid succession. What was interesting about this research was that when we had professional arrangers evaluate the system's output, some were critical, while others suggested it was a good arrangement. This reaffirmed that arranging is, in a sense, a very human task with no single correct answer.

When practicing an instrument, instruction books are often used, but since the challenges and skill levels to be overcome differ for each person, etudes suited to a student's skill at a given time should be used. A teacher, based on their rich experience, can instantly select appropriate etudes, but when a student practices alone at home, both the efficiency and effectiveness of the practice decline. We are currently developing a method in collaboration with a company to provide etudes tailored to the student, acting in place of a teacher. This is also an example of sheet music-to-sheet music conversion, but unlike the music box example, it requires consideration of how to combine various practice challenges into the arrangement.

Thus, music information processing is a research field that uses computers, leveraging mathematics and music theory, to perform tasks that humans carry out using experience, knowledge, and sensibility.