The Imagination of Keio Science and Technology: Human-Agent Interaction and SF

I want you to pioneer research that creates what lies ahead, rather than following current Japanese trends.

The high affinity of Japanese people for communication robots appears unique. Osawa is well-versed in multiple media, including SF and manga, and feels the richness of Japanese society's sensibilities. He lightly crosses the boundary between the humanities and sciences, leading the world in AI research that leverages these sensibilities.

Featured this time is Associate Professor Hirotaka Osawa, who researches interactions between humans and "social agents" such as AI and robots.

The research field that studies and designs interactions between humans and "social agents" such as AI and robots is called "Human-Agent Interaction" (HAI). Agents do not merely provide knowledge or technology. They are also social entities, acting as conversation partners or working and talking within groups. Osawa pursues the possibilities of robots, believing that the essence of robots and AI characters lies in how humans perceive and feel about them.

  The research field of Human-Agent Interaction (HAI) is closely related to communicating robots. Recently, opportunities to interact closely with pet robots or robots that deliver food in restaurants have increased. Humans naturally read emotions, sense intentions, and feel attachment toward such robots. HAI broadly studies the interactions between "humans" and "artifacts" that evoke a sense of humanity or intention. Especially today, as "AI agents" that autonomously handle planning and execution—such as customer support, automation of routine tasks, and data analysis—are being introduced in many companies, the importance of HAI research is rising rapidly. Osawa has a series of studies exploring new forms of interaction between anthropomorphized agents and humans. A representative example is the "agent that conveys emotions and information through human eye movements" announced in 2017. Originally working on anthropomorphizing home appliances, he was inspired by the thought, "Perhaps we can provide psychological support by anthropomorphizing humans themselves..."

  Focusing on the "eyes" that express emotions richly, he developed "Agency Glass," a device that substitutes the communicative functions of the eyes (Figure 1). An attached camera captures the other person's face and gaze, automatically moving the eyes accordingly. When worn by a visually impaired person, the eyes appear to move naturally, reducing any sense of awkwardness. Additionally, since vibrations are transmitted to the face when the eyes move, the wearer can turn their head accordingly. Although the wearer cannot see the images, visually impaired users can align their gaze with their conversation partner, enabling smoother communication.

The "Book Introduction Robot" actually used in an elementary school in Tsukuba City is a success story of activating children's reading communities through a robot (Figure 2). When students type the book they want to introduce and a description into a computer, the book's name is registered. Clicking on the book's name on the screen causes the robot to introduce the book via voice, incorporating facial expressions and movements. The ability to set the robot's expressions (facial movements and colors) was very popular. "I was happy to see older students spontaneously teaching younger ones how to make and use it," says Osawa. Osawa calls robots that not only help by themselves but also activate the environment by their presence "catalytic agents."

In addition, a learning system for children is under development. It is a system where children learn by teaching the agent, rather than being taught by it. "Such agents do not necessarily need to be smarter than humans. In fact, being a bit more clumsy than a human is just right. The relationship with catalytic agents is one of my future research themes."

The theme of "social intelligence" in agents is the second major pillar of Osawa's research. This research primarily focuses on software using programming and algorithms, with the challenge being whether AI can understand the intentions of others and cooperate. For example, in the party game "Werewolf," players try to find out who the "Werewolf" is while talking to each other. He also researches a game called "Hanabi," where players cooperate by estimating their own cards based on others' actions while their own cards are hidden. The key is reading the psychology of others—reading intentions, cooperating, and sometimes betraying—and he is researching how to make AI perform these tasks.

While it is difficult for AI to read intentions like a human, performance improves when it acts according to the other person's personality. "We need AI designs tailored to human individuality, such as bold risk-takers versus cautious types, or those who decide instantly versus those who take time to deliberate," says Osawa. He is working to improve performance through repeated testing while actually playing games with humans.

The third pillar of his research is the application of SF. Even with agents developed to support humans, various problems arise in actual use. SF writers have actually been exploring these issues in advance. Through deepening his interactions with them, Osawa thought he could gain new ideas for agents using SF.

Figure 3 classifies AI characters appearing in SF. Using statistical methods such as clustering and principal component analysis based on parameters like autonomy, friendliness, network breadth, and linguistic ability, he found that AI depicted in SF can be broadly divided into four types. "Humanoid and machine types have existed for a long time, but we can see the potential for new agents in the buddy and infrastructure types that lie in between," says Osawa. Currently, he has established the "Science Fiction Research and Development / Implementation Center (SU)" within the university as a base for his research.

Osawa is also researching a problem-discovery method called "SF Prototyping" using SF. Typically, companies use "forecasting" to discover future challenges based on the current situation. In SF prototyping, stakeholders such as SF writers and companies imagine stories of the future and perform "backcasting" to calculate the route to get there. This brings challenges and visions of the future into clear focus that were difficult to find with conventional methods.

The difficulty of Osawa's research lies in the fact that evaluation criteria are not simple. It is hard to establish standards for "acceptability" or "human-likeness." Furthermore, there is significant variation in evaluations due to individual differences. Nevertheless, Osawa believes, "If even one in ten people can be saved, it is worth doing." He says, "It is difficult because the subject is human, but that is also the most interesting part."

We look forward to Osawa's research leading toward a happy future for humans and agents.

(Interview and composition by Yuko Hiratsuka)

Interview with Associate Professor Hirotaka Osawa

In elementary school, I learned about programming through a friend's older brother and realized I could make games myself. My dream was to create games like "Super Mario Bros.," which I loved at the time. In junior high and high school, I did programming in a club and enjoyed playing games I made with my friends. I found it interesting to make characters in games move vividly, and while researching various things, I learned about and became interested in artificial intelligence through a book called "Embodiment and Computers (a special issue of bit magazine)." I then enrolled in the Faculty of Science and Technology at Keio University, wanting to pursue that field.

At the university, I was active in the Robot Technology Society and the SF Research Society. When it came time to choose a laboratory in my third year of undergraduate studies, a senior in the Robot Technology Society recommended the Imai Lab, saying, "If you want to do what you're interested in, the Imai Lab might be good," so I joined the Anzai-Imai Lab.

At the time, I was interested in AI algorithms and learning, so when I told Professor Imai, "I want to do neural networks," he said, "AI is most interesting when you think about how to interpret and respond to unpredictable things that happen where people interact. You should do that, Osawa-kun." At the time, I wondered what he meant, but my thinking changed through subsequent experiences. In the lab, there was "Robovie," a robot created for communication research. When you spread your arms toward Robovie and say, "Give me a hug," the robot detects it with sensors, approaches you, says "I love you," and hugs you. When I tried it, I was deeply moved. I thought I understood the mechanism of the movement well enough, but I never expected to be so moved, so the gap was significant. There are things you won't understand unless you experience them, even if you're supposed to know them. Even if you can't reach a conclusion by thinking in your head, you can sometimes reach it by actually designing and operating it. Realizing that might be the origin of my current HAI (Human-Agent Interaction) research.

For a while, I used the lab's communication robots to research defining the meaning of actions based on the robot's utterances and information obtained from sensors. Later, during my master's course, the HAI Symposium was established, and I won an award for research on the anthropomorphism of home appliances. The appliances would explain how to use themselves, but when I measured the eye movements of the users, it was an unexpected discovery that the understanding changed even with the same utterance depending on the character's position and settings.

Gratefully, my research on anthropomorphism was accepted for the Japan Society for the Promotion of Science (JSPS) Research Fellowship and the Japan Science and Technology Agency (JST) "PRESTO" project. I continued my research as a visiting researcher at the National Institute of Informatics, a special researcher at the MIT AgeLab in the United States, an assistant professor at Keio University, and an assistant professor at the University of Tsukuba, before returning to Keio in 2022, where I am today.

I considered anthropomorphism a keyword for advancing AI research, but to go further, I wanted to focus on "social intelligence for adapting to others," such as cooperation and betrayal. During my time at the University of Tsukuba, I collaborated with people from diverse research backgrounds to refine research ideas using the "Werewolf" game, formed a voluntary organization, and proceeded with the research. Those members are now active at the forefront of various research fields.

The HAI Symposium has been held every year since the first one in 2006. Since 2013, it has evolved into the HAI International Conference, held annually in Europe and Asia, including Spain, South Korea, Singapore, Germany, New Zealand, Sweden, and the UK. It was also held online during the COVID-19 pandemic. This year, I am the general chair, and it will be held at Keio's Hiyoshi Campus in the fall of 2025.

In AI research, the investment amounts in the US and China are on a different scale. It is often said that Japan is one lap behind, but I believe Japan has a significant advantage in AI research that collaborates with fields where Japan is strong, such as quantum computers, materials science, chemistry, and biology. The HAI field is one of them. Japan is a leader in research and practice of communication robots, and the synergy with character culture and virtual agents is also a strength.

Japan has a significant accumulation of visual and interactive fiction. Manga is undoubtedly the best in the world in terms of both quality and quantity, and animation and video games also have a world-leading impact. While the United States, Europe, and China are also excellent, from a comprehensive perspective that includes the derivative work scene, the environment surrounding Japanese creation is extremely diverse.

Just as Comic Market, which symbolizes doujin culture, branched off from the Japan SF Convention, SF culture lies at the origin of Japan's diverse works. SF has many charms, one of which is that it develops stories based on the world rather than humans. Because stories unfold based on the nature of the set world without being bound by depicting humans as they are now, it has the advantage of being a place for free thought experiments.

From the perspective of artificial intelligence research, we are considering SF as the next theme for nurturing AI. I believe one challenge is how machines can support or even replace the work of creating visions, such as in SF prototyping. For that purpose, research into SF itself is necessary, and I expect that this will create new synergies.

SF research is being undertaken in the United States as business-related research and in China as brain science or psychology-related research. However, if I may say so myself, I believe Keio University's SF Center is the best in terms of diversity. I don't think there are many other places where engineers, cognitive scientists, aestheticians, philosophers, management scholars, and science communication researchers collaborate on research such as creating AI visions from "AI mapping in SF" or evaluating the results of SF prototyping.

  The "Science Fiction Research and Development / Implementation Center" (SF Center) was just launched in 2024. Based on the history of SF in art and creation and its transformations, researchers in literature, engineering, and art are collaborating to explore the value of stories and investigate the role of SF as a methodology for leading to innovation in human society. There are six groups, such as the AI group and the cognitive science group; for example, Professor Susumu Niijima of the Faculty of Economics serves as the head of the literature group. The flexibility of Keio as a comprehensive university, where the humanities and sciences can easily collaborate on themes like SF, is unique. I feel that the framework for tackling new things is well-established. "SF Studies," the study of SF itself, was proposed by Sakyo Komatsu, but until now, there have been few places that have treated it as an academic discipline. We are currently preparing for initiatives at the SF Center. This is something that can probably only be done by Keio, which is arguably Japan's top private university. It is an advantage of private schools to be able to independently take risks and explore the goals of the next era, rather than just doing what everyone else is aiming for at the time while watching the government's intentions.   

Basically, I respect what each individual wants to do and want them to conduct research along those lines. However, since fourth-year undergraduates sometimes find it difficult to have a clear vision of what they should do in the first place, I provide direct paths or have them work in groups with senior students. Since they can choose from several groups, I have them decide while consulting with the group leaders. From the Master's level onwards, I respect their autonomy in deciding their own research themes. I provide advice on selecting topics such as sociality, agents, and interaction.

Since I was the same way, my honest feeling is that I want them to do what they like freely. In particular, I want to tell them, "It's ultimately more fun to do new things without fear." Of course, it's also important to build up your skills in existing fields. Truly new things are not recognized by anyone at first, but in my experience, if you work hard for a few years, someone will recognize you, and the number of people who become your allies will increase. If it's something you've questioned yourself about and can explain the reason for to your own satisfaction, I want you to continue without giving up, and I believe Keio is a place where that is possible.

(Interview/Composition: Yuko Hiratsuka)