Published papers

Through the use of multiple avatars and robots, the construction of an alter ego whose senses and movements are synchronized with those of the operator has been explored. There are two possible states of operation when manipulating those bodies as synchronous operation: multiple bodies are synchronized simultaneously, and asynchronous operation: a specific body is selectively operated. We propose a system that allows intuitive switching between the active robot and the operation state by Eye gaze and EMG. A two-stage pick-and-place task was performed with the proposed system, and task performance and subjective evaluation of embodiment were analyzed. To consider the coordinates of the tracker in the asynchronous operation of the robot arm, we performed a similar study for a coordinate condition: the absolute coordinate condition, where the coordinates of the tracker are stored in space, and the relative coordinate condition, where only the motion of the tracker is added.

In this paper, we present a technique to conduct on-site laboratory experiments using VRChat, a social Virtual Reality platform. COVID-19 pandemic has created challenges and restrictions on re-searchers in running experiments for Human-Computer Interaction research. Hence, universities and institutions set guidelines for social distancing and encourage people to work from home. Therefore, participants are harder to recruit for experiments that run in a physical laboratory. Our technique uses VRChat as a substitution for running on-site experiments despite social restrictions. We exploit features in VRChat to enable researchers to run experiments with domestically recruited participants. Furthermore, we propose a hassle-free method to retrieve study data. This paper also discusses and offers design considerations to run an on-site experiment using VRChat over a homemade system.

Visual perspective taking is inferring how the world looks to another person. To clarify this process, we investigated whether employing a humanoid avatar as the viewpoint would facilitate an imagined perspective shift in a virtual environment, and which factor of the avatar is effective for the facilitation effect. We used a task that involved reporting how an object looks by a simple direction judgment, either from the avatar's position or from the position of an empty chair. We found that the humanoid avatar's presence improved task performance. Furthermore, the avatar's facilitation effect was observed only when the avatar was facing the visual stimulus to be judged; performance was worse when it faced backwards than when there was only an empty chair facing forwards. This suggests that the avatar does not simply attract spatial attention, but the posture of the avatar is crucial for the facilitation effect. In addition, when the directions of the head and the torso were opposite (i.e., an impossible posture), the avatar's facilitation effect disappeared. Thus, visual perspective taking might not be facilitated by the avatar when its posture is biomechanically impossible because we cannot embody it. Finally, even when the avatar's head of the possible posture was covered with a bucket, the facilitation effect was found with the forward-facing avatar rather than the backward-facing avatar. That is, the head/gaze direction cue, or presumably the belief that the visual stimulus to be judged can be seen by the avatar, was not required. These results suggest that explicit perspective taking is facilitated by embodiment towards humanoid avatars.

Research has been conducted on attaching a robotic arm to a human body and using it as a supernumerary limb. However, it remains unclear what type of behaviors contribute to the induction of embodiment in a supernumerary limb. In this study, we created a virtual environment where we could observe behaviors that contribute to the embodiment of avatars and tools, such as "looking at one's own body in a mirror,""moving the supernumerary limb quickly and accurately,"and "paying constant attention to the supernumerary limb,"and verified whether these behaviors are also effective in inducing the embodiment of the supernumerary limb. After spending 40 min in a behavior observation space, the left hand, the side with the supernumerary limb, was used significantly less frequently. In addition to these behaviors, "self-touching,""transferring objects between their own hand and the supernumerary limb,"and "whole-body movement"were also observed. The observations suggest that these behaviors may change the awareness of the supernumerary limb and contribute to the improvement of its embodiment.

As one of the facial expression recognition techniques for Head-Mounted Display (HMD) users, embedded photo-reflective sensors have been used. In this paper, we investigate how gaze and face directions affect facial expression recognition using the embedded photo-reflective sensors. First, we collected a dataset of five facial expressions (Neutral, Happy, Angry, Sad, Surprised) while looking in diverse directions by moving 1) the eyes and 2) the head. Using the dataset, we analyzed the effect of gaze and face directions by constructing facial expression classifiers in five ways and evaluating the classification accuracy of each classifier. The results revealed that the single classifier that learned the data for all gaze points achieved the highest classification performance. Then, we investigated which facial part was affected by the gaze and face direction. The results showed that the gaze directions affected the upper facial parts, while the face directions affected the lower facial parts. In addition, by removing the bias of facial expression reproducibility, we investigated the pure effect of gaze and face directions in three conditions. The results showed that, in terms of gaze direction, building classifiers for each direction significantly improved the classification accuracy. However, in terms of face directions, there were slight differences between the classifier conditions. Our experimental results implied that multiple classifiers corresponding to multiple gaze and face directions improved facial expression recognition accuracy, but collecting the data of the vertical movement of gaze and face is a practical solution to improving facial expression recognition accuracy.

Recognizing foot gestures can be useful for subtle inputs to appliances and machines in everyday life, but for a system to be useful, it must allow users to assume various postures and work in different spaces. Camera-based and pressure-based systems have limitations in these areas. In this paper, we introduce AnkleSens, a novel ankle-worn foot sensing device that estimates a variety of foot postures using photo reflective sensors. Since our device is not placed between the foot and the floor, it can predict foot posture, even if we keep the foot floating in the air. We developed a band prototype with 16 sensors that can be wrapped around the leg above the ankle. To evaluate the performance of the proposed method, we used eight foot postures and four foot states as preliminary classes. After assessing a test dataset with the preliminary classes, we integrated the eight foot postures into five. Finally, we classified the dataset with five postures in four foot states. For the resulting 20 classes, the average classification accuracy with our proposed method was 79.57% with user-dependent training. This study showed the potential of foot posture sensing as a new subtle input method in daily life.

We propose a virtual reality (VR) baseball batting system for assessing individual decision-making skill based on swing judgement of pitch types and the underlying prediction ability by manipulating combinations of pitching motion and ball trajectory cues. Our analy-sis of data from 10 elite baseball players revealed highly accurate swing motions in conditions during which the batter made precise swing decisions. Delays in swing motion were observed in conditions during which predictive cues were mismatched. Our findings indicated that decision-making based on pitch type influences the inherent stability of decision and accuracy of hitting, and that most batters made decisions based on pitching motion cues rather than on ball trajectory.

Pseudo physical touch is used for communications in virtual environments such as VRChat. We aimed to test if the pseudo-touch communication affects social impression in a virtual environment. Nineteen participants performed the controlled experiment with a partner who was an experimenter with three types of communications: no touch, pseudo touch, and actual touch. Subjective ratings of attractiveness and the communication easiness with the partner increased in all conditions, suggesting that the communication in virtual environments improves interpersonal attraction and communicability either with or without physical or pseudo touch.

Smiling plays a crucial role in human communication. It is the most frequent expression shown in daily life. Smile analysis usually employs computer vision-based methods that use data sets annotated by experts. However, cameras have space constraints in most realistic scenarios due to occlusions. Wearable electromyography is a promising alternative; however, issue of user comfort is a barrier to long-term use. Other wearable-based methods can detect smiles, but they lack consistency because they use subjective criteria without expert annotation. We investigate a wearable-based method that uses optical sensors for consistent smile intensity estimation while reducing manual annotation cost. First, we use a state-of-art computer vision method (OpenFace) to train a regression model to estimate smile intensity from sensor data. Then, we compare the estimation result to that of OpenFace. We also compared their results to human annotation. The results show that the wearable method has a higher matching coefficient (r=0.67) with human annotated smile intensity than OpenFace (r=0.56). Also, when the sensor data and OpenFace output were fused, the multimodal method produced estimates closer to human annotation (r=0.74). Finally, we investigate how the synchrony of smile dynamics among subjects and their average smile intensity are correlated to assess the potential of wearable smile intensity estimation.

An inter-personal shared body is an extended body that allows an avatar or robot to be controlled by multiple people. It is possible for operators to share control over the actions of the shared body. A shared body can be operated well when the same actions are performed by its operators. However, when their actions are not the same, we need to have an adaptive control method for the operators to maintain a sense of agency and ownership. In this study, we propose dynamic adjustment methods to control a shared body using EMG sensors and movements of the operators.

Visual-tactile synchronization or visual-motor synchronization makes a not innate body feel as if it was one's own body (illusory body ownership). Although recent studies have shown that body ownership can be induced in multiple bodies, it is unclear whether the body ownership is switched between bodies, or we feel that multiple bodies belong to us simultaneously. If the body ownership is induced in multiple bodies simultaneously, we expected the body schema to be extended to correspond to the number of bodies. In this study, we investigated whether body ownership is induced in multiple bodies simultaneously by measuring the body schema when the bodies are replicated and split. Experiment 1 investigated whether two side-by-side virtual bodies moved synchronously with the participant's movements to extend the body schema. In Experiment 2, virtual split bodies in which a virtual body was split into left and right sides were presented. As a result, the body schema was extended only in the split bodies. However, it changed even in the no body ownership condition, suggesting that the appearance of the split bodies simply affected the body schema, not that body ownership of the split bodies occurred at the same time. The results of the questionnaire indicated that the split bodies might be perceived as a single body.

Virtually reality can induce body ownership (Sense of being one's own body) of a not innate body or multiple bodies. However, the relationship between the number of bodies and body ownership has not been clarified. In a study that investigated body ownership for one, two, and four virtual bodies, the greater the number of bodies, the closer the body ownership tended to approach the strength of ownership for a single body. Therefore, in the present study, we investigated whether increasing the number of bodies strengthens body ownership for multiple bodies. In the experiment, multiple virtual bodies that moved in synchronization with the participant's movement were lined up in a single file line, and the participant observed the multiple virtual bodies through a head-mounted display from the position of the hindmost virtual body. We measured body ownership and drifts in self-location. Our results showed that contrary to expectations, the greater the number of bodies, the weaker body ownership.

In the illusory body ownership, humans feel as if a rubber hand or an avatar in a virtual environment is their own body through visual-tactile synchronization or visual-motor synchronization. Despite the onset time and duration of illusory body ownership has been investigated, it is not clear how the onset time and duration change when a part of the body is missing from the full-body. In this study, we investigated the completeness of the full-body for the illusion onset and duration by comparing the following conditions: complete avatar, avatar missing hands and feet, and avatar with hands and feet only. Our results suggest that avatar hands and feet only shorten the duration of the illusion, and missing body parts, such as only hands and feet or no hands and feet, reduce the sense of body ownership and of agency. However, the effects of avatar completeness on the onset time are unclear, and no conclusions can be made in either direction based on the current findings.

We developed an 'unnatural avatar' of which the lower arms could be bent in reversed/unnatural directions. Participants controlled the avatar in a virtual environment to perform reaching tasks. We measured their senses of body ownership and agency. Compared to a normal avatar, the embodiment towards the unnatural avatar was significantly lower. However, senses of body ownership and agency towards both avatars were higher when the avatar's movements were synchronized with the movements of the participants compared to when there was a delay of one second. These results suggest that visuo-motor synchrony induces embodiment even towards avatars with reversed/unnatural limb movements.

The supernumerary robotic limb system expands the motor function of human users by adding extra artificially designed limbs. It is important for us to embody the system as if it is a part of one’s own body and to maintain cognitive transparency in which the cognitive load is suppressed. Embodiment studies have been conducted with an expansion of bodily functions through a “substitution” and “extension”. However, there have been few studies on the “addition” of supernumerary body parts. In this study, we developed a supernumerary robotic limb system that operates in a virtual environment, and then evaluated whether the extra limb can be regarded as a part of one’s own body using a questionnaire and whether the perception of peripersonal space changes with a visuotactile crossmodal congruency task. We found that the participants can embody the extra-limbs after using the supernumerary robotic limb system. We also found a positive correlation between the perceptual change in the crossmodal congruency task and the subjective feeling that the number of one’s arms had increased (supernumerary limb sensation). These results suggest that the addition of an extra body part may cause the participants to feel that they had acquired a new body part that differs from their original body part through a functional expansion.