Brain Science at Keio's Faculty of Science and Technology: The World of Brain-Machine Interfaces, Where Machines Move Just by Thinking

Bringing Science to Rehabilitation, and BMI to Patients

Junichi Ushiba is researching the application of brain-machine interfaces (BMI) to rehabilitation.

He discovered computers as something he could be passionate about and immersed himself in them from elementary school.

Another thing that captivated him during his junior high school years was the brain.

Both became the warp threads of Ushiba's career as a researcher, now woven into his BMI research

as he aims for the day when patients can use it.

Profile

Department of Biosciences and Informatics

Engaged in research on motor control mechanisms related to human voluntary movements and reflexes. Recently, he has been working on the development of brain-machine interfaces that apply his scientific findings to date. In 2003, he was a visiting researcher at the Center for Sensory-Motor Interaction, Aalborg University, Denmark. In 2004, he obtained his Ph.D. in Engineering and became an assistant professor at Keio University. Since 2007, he has been a senior assistant professor at the Keio University Faculty of Science and Technology, a position he holds to this day.

In the "New Edition of the Research Bulletin Kyurizukai," we feature one young researcher in each issue.

This issue features Senior Assistant Professor Junichi Ushiba, a young leader in BMI research, from which great results are expected in rehabilitation.

Connecting "Thought" to "Action"

A lone man walks silently along a frozen, snowy path. He seems to have difficulty walking, but even so, his movements are awkward. Just when you think he is moving straight ahead, he suddenly veers sharply to the right or left, sometimes even backward... It seems his destination is near. Mustering his last bit of strength, the man walks with all his might. Finally, he reaches a small cabin in the snowy mountains. A sigh of relief...

Suddenly, several young people rush out of the cabin. They surround the man, congratulating him with shouts of "Congratulations!" The man grasps their hands and replies powerfully, "Thank you."

The young people who rushed over are graduate and undergraduate students from the Tomita-Ushiba Laboratory in the Department of Biosciences and Informatics at the Keio University Faculty of Science and Technology. The man who reached the cabin is Mr. K, a 41-year-old former systems engineer. In fact, Mr. K suffers from muscular dystrophy. For the past 30 years, he has been almost unable to move his hands or feet. The man walking on the snowy path is the avatar (alter ego) of Mr. K in the world of Second Life. The students' avatars have gathered around him.

While the students are controlling their avatars with keyboards, Mr. K is moving his with the "thoughts" in his head. This magic wand for "moving things by thinking" is the BMI (Brain-Machine Interface). It is a new system that fuses knowledge from brain science and the medical sciences with the technology of information engineering. Leading this research and development is Senior Assistant Professor Junichi Ushiba, a brilliant 31-year-old researcher standing 184 cm tall.

Now, several electrodes are attached to Mr. K's head. When he forms an image in his mind, such as "go straight" or "turn right" (activating his brain), the pattern of his brain waves is captured as a signal by the electrodes. This is input into a computer to move the avatar in the virtual world. Using BMI, it is also possible to move devices and systems in the real world just by thinking.

However, for those with physical disabilities who have not moved for a long time, it becomes difficult for brain activity corresponding to the motor image to occur, and proper brain waves for the movement are not produced. The brain's activity also requires "rehabilitation." Mr. K attempted to use Second Life many times and finally succeeded in getting his avatar to its destination. That is why his young colleagues from the Tomita-Ushiba Laboratory rushed to congratulate him.

BMI reads the brain's motor commands from brain waves and analyzes them with a computer to directly control wheelchairs, home appliances, prosthetic hands, avatars, and more. It is expected to be a technology that can improve the quality of life for patients with spinal cord injuries, limb amputations, and other conditions.

Mr. Ushiba, who leads the BMI project, became fascinated with computers in elementary school and was captivated by the mysteries of the brain in junior high (see the interview with Mr. Ushiba on pages 4-5!). From the time he joined Professor Yutaka Tomita's laboratory for his undergraduate thesis research to the present day, he has conducted research in information and electrical engineering at the Faculty of Science and Technology on the Hiyoshi Campus in Yokohama, while also advancing his neurophysiological research at the School of Medicine on the Shinanomachi Campus in Shinjuku and at the Tsukigase Rehabilitation Center in Izu. "For me, researching both the brain and computers was a natural progression. I always felt that my place was at the intersection where these two fields merge."

It was in 2006 that Mr. Ushiba began working on BMI research, which directly connects the brain and computers. First, he verified what kind of brain waves are emitted from the brain's "somatosensory and motor cortex" (the area that controls sensation and movement) when a healthy person moves or imagines movement. The somatosensory and motor cortex has specific locations that control the movement and sensation of the hands, feet, shoulders, torso, and so on. "Indeed, when a person is actually moving their feet and when they are imagining the same movement, similar brain waves are emitted from the same location in the motor cortex."

In this way, he accumulated data on the correlation between types of movement and brain wave patterns and also developed a method for processing this data in real time. Next, with the aim of turning BMI into a communication tool for people unable to move their bodies due to conditions like spinal cord injury or ALS (amyotrophic lateral sclerosis), he tackled the challenge of moving an avatar with brain waves, as introduced at the beginning, and was able to achieve results in just six months. "For nearly 10 years, I have been conducting neurophysiological experiments with professors from the School of Medicine, delving into the differences in the sense of movement between healthy individuals and those with physical disabilities. This was likely the key to our smooth progress."

In patients with spinal cord injuries, the pathway that transmits motor commands (thoughts) from the brain to the muscles (actions) is severed. BMI acts as an alternative pathway (a bypass) to connect thoughts and actions.

In stroke patients, the brain's motor commands are not transmitted correctly to the muscles, and the muscles do not move. As a result, "sensory feedback from the muscles to the brain" also does not occur (a).

With BMI, the hand moves in response to the motor command, generating feedback. It is thought that maintaining the pathway from the brain to the muscles and from the muscles to the brain promotes rehabilitation (b).

In Mr. Ushiba's mind, perspectives, knowledge, methodologies, and technologies from a wide range of fields—including neuroscience, brain science, information science, and information engineering—are stored without being constrained by disciplinary boundaries. Furthermore, the raw sensibilities of the medical field are also incorporated. He has flexibly combined and deepened these elements to open up new horizons for BMI. One of these is the new concept of BMI as a "tool for rehabilitation," that is, to "restore the function of limbs and other body parts."

Traditionally, BMI in the medical field has been developed from the perspective of "substituting for the function of limbs" for people with physical disabilities. However, the leading cause of physical disability is stroke. Since patients with hemiplegia (paralysis on one side of the body) can still move the other half of their body, functional substitution is not as necessary. Also, unlike with spinal cord injuries or ALS, function can be restored to some extent with appropriate rehabilitation. "That's the crucial point. I had a sudden intuition that BMI could be useful for rehabilitation."

He immediately assembled a BMI system and began joint experiments with the School of Medicine. In this BMI rehabilitation system, the paralyzed hand is fixed on a box containing a motor. When the patient wills their fingers to extend, the brain wave signal is transmitted through the BMI to the motor, which then moves to extend the fingers. However, if the brain wave pattern does not match that of extending the fingers—that is, if the correct brain activity does not occur—the motor's switch will not turn on.

If a hand has been paralyzed for many years, it doesn't work well at first. It's difficult to form an image of the paralyzed hand in one's mind. When trying hard to will it, strange force is applied to the non-paralyzed side, changing the brain wave pattern. "Through trial and error, they learn to will it correctly while remaining relaxed... It's a learning process." When the brain is rehabilitated in this way, changes also occur in the muscles. In the muscles of the hand where no myoelectric potential was detected at all before BMI rehabilitation, a potential is detected when the patient correctly wills their fingers to extend during BMI rehabilitation.

As a result of this training, some patients have emerged who feel a slight improvement in their finger movements, and others who have become more conscious of actively using their paralyzed side. The global trend in BMI is now shifting toward rehabilitation, and it was Mr. Ushiba and his team who first proved that BMI is effective for rehabilitation.

Regarding future research, he lists three points: ① clarifying the mechanism by which recovery occurs in BMI rehabilitation, ② creating a more efficient BMI rehabilitation system based on those findings, and ③ making it inexpensive so that patients can easily use it as a rehabilitation tool. What will emerge next from Mr. Ushiba's Doraemon-like pocket of a brain as he works toward these goals?

(Interview and text by Nobuko Yuri)

An Interview with Senior Assistant Professor Junichi Ushiba

My family was completely in the humanities. My father was a scholar of French literature and a university faculty member, and my mother worked as a French conversation instructor and translator. My father was often in his study, and seeing him from behind, I thought, "Being a university professor seems nice." That might have been the first time I aspired to make research my career (laughs). In that household, I was raised to "do what you love, and master it responsibly."

When I was in the fifth grade, my school started offering a computer class. Several computers were installed, and they taught programming to interested students after school. A classmate invited me to join, and that was my first time touching a computer. It wasn't an era where every home had a computer, so I mostly used them at school.

During summer vacation, a university professor held a computer class at the Faculty of Science and Technology, so I started attending that as well. Seeing the graduate students writing programs, I was greatly inspired and thought, "Wow, that's amazing." Since then, I've been completely hooked on computers. At that time, manga and TV anime were popular among elementary school students, but my family didn't let me watch those things, so that might have pushed me even more toward computers.

At that time, artificial intelligence was the trend. A graduate student once brought a chatbot program to my elementary school. It was like a riddle game; if you gave it hints one by one, it would eventually give the correct answer. If you made a mistake and taught it the right answer, it would learn and give the correct response next time. I learned for the first time that you could create artificial intelligence, even though computers are supposed to only do what they are instructed to do. I was completely astonished.

In junior high, an alumnus, Dr. Katsuhiko Mikoshiba (currently at RIKEN), came to speak, and I heard about the brain for the first time. The way he spoke with such passion was very captivating. After that, I applied on my own to attend a lecture by brain scientist Dr. Gen Matsumoto (then at the Electrotechnical Laboratory). I was incredibly inspired by these two individuals, and the impression they made is still strong today.

At the annual school summer vacation independent research presentation, I would present programs I had created myself. I made them all on my own. In my first year of junior high, I did a simulation of urban reconstruction after the Gulf War; in my second year, I used an early hand scanner to research morphing; and in my third year, I created a ray-tracing program that estimated how the shadow of a polypyramid would appear depending on the position of the light source.

Every Saturday, I would go to a graduate student's boarding house to be taught algorithms. It was private tutoring. I really loved computers.

I don't think I was a particularly conspicuous presence at school. I wasn't good at sports, nor did I particularly attract attention. I got along with all types of classmates, but I did the things I loved by myself, both at school and at home. I guess I was pretty laid-back. I never felt any sense of alienation.

I went to a high school known for its strength in computers, but I also joined the brass band and played the trumpet, and even formed a rock band. My interest in computers waned a bit because Windows came out, making the system more complex and harder to handle.

On the other hand, I still had a strong interest in the brain. I liked browsing for difficult books at the library or bookstore, and I reacted strongly to words like "artificial intelligence" and "artificial life." I found out that a graduate student at the university next to my high school was translating a book on artificial life that had been published at the time, and I was inspired, thinking, "There's such an amazing student so close by." Even though computers are only supposed to do what they're told, functions like those performed by the brain or living organisms emerge. It was so mysterious to me that phenomena that weren't designed could be created. I wanted to know how that was possible.

For university, I was torn between aiming for the School of Medicine or the Faculty of Science and Technology. The Faculty of Science and Technology had professors specializing in bio-information, and I loved computers, so I ultimately decided to go to the Faculty of Science and Technology. Keio's Faculty of Science and Technology had just established a new department called the Department of Applied Physics and Physico-Informatics, so I enrolled there. This department had deep ties with the School of Medicine, and there were professors whose themes were nerves and muscles.

Although I went into the Faculty of Science and Technology, I was not good at math. I used to get Cs in junior high. I think I only really got motivated around my third year of university. Instead of learning the basics as just basics, I started to see their applications and understood the need to learn them, which finally gave me the motivation to study. I was the type of person who, upon seeing how something could be useful to society, would then study the necessary fundamentals for it.

The reason I joined Professor Yutaka Tomita's lab was largely because he was conducting rehabilitation research and had connections to the medical sciences. Right after I joined, he introduced me to a professor at the School of Medicine, saying he wanted to start a joint research project.

Of course, being raised in a family of academics was a factor, but thanks to computers, I had been in and out of the university since elementary school, so I felt a sense of familiarity with it.

I thought that a university was a wonderful world where everyone was doing creative work, and both young and experienced people got along in a liberal atmosphere. That feeling didn't change even when I became a university student myself. Before I had a chance to be attracted to companies, I was captivated by the charm of the university and ended up working here.

I also like Keio's liberal atmosphere, and I think one of Keio's strengths is its strong network, including Juku-wide collaborations where university professors visit elementary schools or junior high students visit the university, and alumni come back to Keio to speak. When I was looking for a job at a university, I looked around and was once again struck by Keio's appeal.

Since I was inspired by Dr. Mikoshiba's talk in junior high, I also go to junior high schools to talk about my own specialty, BMI research. I've been doing it for four years now. The other day, I also went to a girls' high school. I hope I can give back even a little.

It makes me happy to see students grow and do well out in the world, and when they say things like, "Your words back then encouraged me," I feel it's the greatest reward of being a teacher.

On the other hand, I always feel the difficulty of dealing with people. There have been times when my feelings didn't get through to students, and I lost confidence. I don't think I should give too many instructions, but some students want more detailed guidance, so there are many times I'm at a loss. Sometimes my lectures are called too difficult, other times too easy, so it's a process of trial and error to find the right balance.

In the undergraduate program, I am in charge of lectures on biocybernetics and statistics. I also teach lab classes. In the research setting, I might be someone who gives relatively detailed instructions to students. In addition to research and education, my roles have increased to include coordination with the School of Medicine, internal university work, lectures at various medical and engineering academic societies, and industry-academia collaboration work.

Ultimately, we have to make BMI a tool that patients can actually use, so we are teaming up with companies that support this idea to create bio-signal analysis algorithms and build machines together. The level of commitment to the market varies by company, from those who want to take their time and learn through advanced development to those who are aiming straight for the market. The industries are also diverse, including entertainment, home appliance manufacturers, and automotive.

It's the Strategic Research Program for Brain Sciences from MEXT. Besides Keio, participants include ATR, the University of Tokyo, Osaka University, and Shimadzu Corporation. The representative is Dr. Riu from the Department of Rehabilitation Medicine at the School of Medicine, and Keio is tackling this through a collaboration between the School of Medicine and the Faculty of Science and Technology.

In the short term, the goal is to produce solid evidence for the rehabilitation BMI we are currently developing within a few years. Rehabilitation is a new concept for BMI. I want to establish it as an academic path and disseminate it to the world from Keio. Globally, there are two major research centers in this field, but we are proud of our long history of medical-engineering collaborative research, so we hope to make it flourish in the field of BMI. I believe academic verification will progress to some extent in a few years, but it will take more time to connect it to medical care. However, for purposes like turning on an air conditioner or a TV, it's not out of the question for a device to be ready in a few years. Another goal is to pave the way for creating something that patients can use at a price they can afford.

In the medium term, I hope to play a part in establishing the science of rehabilitation based on brain science. Current rehabilitation still relies heavily on empirical rules. It is now moving in the direction of being systematized as a science, and I want to be a part of that.

The ultimate goal is to give back to education. This field requires learning across many domains in an integrated manner and interacting with diverse people, so I want to nurture individuals who can do these things spontaneously. I myself have to study, but I also want the students to grow with me and become people who not only have their own solid "warp threads" but can also weave "weft threads" between their own and their neighbors' warp threads.

I was raised in the family I described earlier, and I was able to find something I could be passionate about while learning various things in elementary, junior high, and high school. Fortunately, I was able to become a faculty member, so I want to continue, as my life's work, to help not only university students but also elementary, junior high, and high school students find their dreams and inspirations.

Lately, I don't really have anything specific. Besides work, I have two small children, so I'm in a tizzy taking care of them. Playing Kamen Rider with them might be my change of pace (laughs). The band I was in during my student days continued until a few years ago, renting out live houses and such, but it has since amicably disbanded. My wife is a clinical physician in rehabilitation, so we talk about research at home too. I recently built a study, which I had dreamed of since I was a child, and I was thinking I'd like to relax and read a book, but the reality is I'm busy writing documents for research grant applications.

Thank you very much.

◎A Few Words◎

From a student: He's sharp and has amazing foresight. He's very expressive, which might be rare for a science type. He's also good at getting people excited. It's a fun lab.

From his secretary: I wonder if his difficulty with tidiness is because reality can't keep up with his fast-thinking brain... He gives work instructions kindly and politely. He also has an endearing side as a family man.

From the interviewer: I got the impression that you're the type who achieves results without being too intense. I can just picture you as a young computer whiz. You have a long road ahead, so please be careful not to overwork yourself and get worn out.

(Interview and text by Etsuko Furugori)