Advancing BCI technology using sensory perception to create new entertainment and contribute to society

I was in the Global Studies course in high school, but I wanted to pursue a science-related faculty because I wanted to create something new myself. Among my options, I chose SFC because it has a well-established environment for hands-on learning compared to other universities, and I was also attracted to the interdisciplinary nature of the classes, which is unique to SFC.

After enrolling, I took a wide range of classes in the spring semester of my first year, which is where I became interested in neuro-informatics. Furthermore, I learned about a technology called BCI (Brain-Computer Interface) in that class and decided I wanted to pursue serious research, so I joined the Atsushi Aoyama Lab in the fall semester of my first year.

The benefit of joining a lab, besides being able to engage in serious individual research, is that you can work alongside your lab mates, even though everyone has different goals. Also, for research that requires human subjects, such as in neuro-informatics, there's the advantage that students can serve as subjects for each other. Especially during EEG measurements, securing subjects is a critical mission because the characteristics of brainwaves differ depending on the subject's proficiency with the experimental task.

My current research theme, "Brain Research on the Recall of Thermal and Cold Sensations in the Fingers," is a study in the field of BCI, an interface that allows the brain and computers to communicate directly. Currently, BCI research is advancing with the aim of assisting people whose physical freedom is limited by neurological disorders such as ALS (amyotrophic lateral sclerosis) and GBS (Guillain-Barré syndrome). The mainstream of this research focuses on motion control through motor imagery and text input using visual stimuli. However, the concept of "sensory recall," suggested by Professor Aoyama, is distinct from such research, and I began my study as a new challenge. Additionally, since there is little research that emphasizes entertainment, I believe I can explore new possibilities for BCI.

If this research is successful, it will become possible to do things like cast a fire spell in a VR game or the metaverse just by thinking of a "hot" sensation, or cast an ice spell just by thinking of a "cold" one.

Specifically, my "Brain Research on the Recall of Thermal and Cold Sensations in the Fingers" is a study investigating whether it's possible to classify brainwaves online (in real time) using machine learning when a person recalls (imagines) the sensation of warmth (hot) or coldness (cold) in their fingertips, and whether this can be applied to future entertainment.

In the study, I gathered subjects, had them wear an electroencephalograph (EEG) on their heads, and asked them to look at a monitor displaying a cross in a dark room. Depending on the color of the cross, they were instructed to recall the sensation of touching something hot or something cold with their fingertips. However, it is difficult to recall "hot" and "cold" sensations from nothing, and it is necessary to ensure consistency in the sensory images among subjects. Therefore, I built a thermal stimulation device using a Peltier element and, before the experiment, had subjects touch the Peltier element set to "hot" and "cold" temperatures to help them remember the sensations.

The EEG data obtained from the experiment was appropriately denoised using MATLAB (a numerical computing platform), and various features were extracted for model training. Examples include power spectral density and PAC (phase-amplitude coupling). After that, I repeatedly trained the model with various algorithms and features and evaluated the offline classification to improve its accuracy.

To achieve the final goal of online classification, I am experimenting with how to perform the online processing. While various processing steps can remove noise from brainwaves and improve accuracy, they also increase computational cost. As a result, even with online classification, the time from receiving the brainwave signal to outputting the classification result becomes long, reducing usability. Therefore, the current challenge is to extract only significant features while keeping the computational cost of brainwave processing as low as possible.

The goal of this research is to achieve real-time processing and provide an experience of casting fire and ice spells in a VR game. This is because imagining a fun future vision increases my motivation for the research.

My future goal is to further advance the EEG research I am currently working on and contribute to the development of practical BCI technology. In particular, I want to realize technology that provides new experiences utilizing "sensation" in the fields of VR and the metaverse.

Furthermore, my vision for the future is for BCI technology to become integrated into society, creating an environment where anyone can easily use it. For example, achieving accessibility where people with physical disabilities can control their surroundings with just their thoughts. Or interfaces that allow people to directly experience different cultures and values through sensation. The more I think about it, the more my imagination runs wild about a society where BCI technology is widespread.