Published papers

The robotic limbs using a body remapping approach often follow either the user's reference frame 1) fixed outside the user's body (Space frame), 2) centered on a user's torso (Torso frame), or 3) centered on a user's head (Head frame). In this study, we investigate the effect of the reference frame on the sense of embodiment and the user's motor control process. We asked 12 participants to perform some point-to-point tasks with the virtual robotic limbs remapped to the participants' feet. The virtual robotic limbs tip followed the participants' feet and each reference frame condition (Space frame, Torso frame, and Head frame). As a result, the reaction time, the movement straightness, and the movement priority were significantly high rate in the Torso frame condition. The subjective score on the sense of embodiment showed that the reference frame condition contributed to the sense of embodiment.

Virtual reality (VR) is gaining interest as a platform for sports skills training. VR allows for information manipulation and feedback that would be difficult in reality. This is particularly useful in open skill sports where players must adjust their behavior in response to environmental factors. Auditory feedback (sonification) is constructive for sports training in VR. However, this has not been well studied in open skill-specific situations due to the difficulty of accounting for environmental factors in reality. This study focuses on a serve return, an important scene in tennis. It investigates the effects of sonification on the forehand return stroke in VR by comparing score displays and trajectory visualizations. We designed the sonification based on the difference between experienced and novice players' movements in VR. We then conducted a between-subjects experiment to investigate the effect of the sonification (N=20). The results showed that the system with sonification effectively improved the timing of hip movement for preparing a slow serve return compared to the system without sonification.

In spatial navigation, adding haptic cues to visual information lets users understand the spatial information better. Most haptic devices stimulate various body parts, while few devices target our heads that are sensitive to mechanical stimuli. This paper presents Virtual Whiskers, a spatial directional guidance technique using cheek haptics in a virtual space. We created a cheek haptic stimulation device by attaching two tiny robot arms to a Head-Mounted Display. The robot arms trace the cheek with proximity sensors to estimate the cheek surface. Target azimuthal and elevational directions are translated into a point on the cheek surface. The robot arms touch the point to present target directional cues. We demonstrate our technique in two applications.

Spatial cues are an important element of navigating people in physical/virtual spaces. In terms of spatial navigation, integrating vision with other modalities, such as haptics, can guide users more effectively. Haptic cues are presented on the body parts that are sensitive to stimuli such as hands and a head. The head is reported to be superior to the body for spatial directional perception. In this paper, we propose Virtual Whiskers, a spatial directional guidance technique by haptic stimulation of the cheeks using tiny robot arms attached to a Head-Mounted Display (HMD). We deploy photo reflective sensors attached to the tip of 2 robotic arms to detect the distance between the tip and the cheek surface. Using the robot arms, we stimulate a point on the cheek obtained by calculating an intersection between the cheek surface and the target direction. We experimentally investigated how accurately participants identify the target direction provided by our guidance method. We evaluated an error between the actual target direction and the participant's pointed direction. The experimental result shows that our method achieves the average absolute directional error of 2.76 degrees in the azimuthal plane and 7.32 degrees in the elevation plane. We also conducted a spatial guidance experiment to evaluate task performance in a target search task. We compared the condition of only vision and vision with haptics for task completion time. The average of task completion time in visual-only condition was M=12.45 s, SD=14.51 s, and visual with haptic condition resulted in M=6.91 s, SD=5.48 s. Statistical test revealed a significant difference in task completion time between the visual condition and the visual+haptic condition.

In this demonstration, we present X-Wing, a force feedback device using ducted fans attached to a Virtual Reality (VR) Head Mounted Display (HMD). Applying haptic technologies to VR has become a standard solution to enhance the immersive experience. It is also essential for virtual interactions where user experience and performance are crucial. This research focuses on a head-based, wearable device that provides forces to the VR user using exertion force by electric thrusters with controllable strengths and directions. Our system allows the user to experience forces based on their virtual momentum and velocity. It enables a VR user to receive force feedback based on different thrust power converted into translational and rotational force for a unique VR experience such as virtual aerial simulation. We present the prototype to the attendees of SIGGRAPH Asia 2021 through a live demonstration.

Spatial cues play an important role in navigating people in both physical and virtual spaces. In spatial navigation, visual information with additional cues, such as haptic cues, enables effective guidance. Most haptic devices are applied to various body parts to make mechanical stimuli, while few devices stimulate a head despite the excellent sensitivity. This article presents Virtual Whiskers, a spatial directional guidance technique by cheek haptic stimulation using tiny robot arms attached to a Head-Mounted Display (HMD). The tip of the robotic arm has photo reflective sensors to detect the distance between the tip and the cheek surface. Using the robot arms, we stimulate a point on the cheek obtained by calculating an intersection between the cheek surface and the target direction. In the directional guidance experiment, we investigated how accurately participants identify the target direction provided by our guidance method. We evaluated an error between the actual target direction and the participant's pointed direction. The experimental result shows that our method achieves the average absolute directional error of 2.54° in the azimuthal plane and 6.54° in the elevation plane. We also conducted a spatial guidance experiment to evaluate task performance in a target search task. We compared the condition of visual information, visual and audio information, and visual information and cheek haptics for task completion time, System Usability Scale (SUS) score, NASA-TLX score. The averages of task completion time were M = 6.39 s, SD = 3.34 s, and M = 5.62 s, SD = 3.12 s, and M = 4.35 s, SD = 2.26 s, in visual-only condition, visual+audio condition, and visual+haptic condition, respectively. In terms of the SUS score, visual condition, visual+audio condition, and visual+haptic condition achieved M = 55.83, SD = 20.40, and M = 47.78, SD = 20.09, and M = 80.42, SD = 10.99, respectively. As for NASA-TLX score, visual condition, visual+audio condition, and visual+haptic condition resulted in M = 75.81, SD = 16.89, and M = 67.57, SD = 14.96, and M = 38.83, SD = 18.52, respectively. Statistical tests revealed significant differences in task completion time, SUS score, and NASA-TLX score between the visual and the visual+haptic condition and the visual+audio and the visual+haptic condition.

We display a virtual avatar changing its biceps appearance to fit the user's biceps contraction. The avatar's biceps changes its size and color based on either: (EMG) the biceps activity (recorded with a electromyography [EMG] sensor); or (ANG) the elbow angle; or (EMG + ANG) the biceps activity and elbow angle. The users also wear an ad hoc haptic device around their biceps to give the illusion of stronger biceps contractions. In the Virtual Environment, the participants (N=10) look at their right biceps and do elbow flexion. They rate EMG, ANG and EMG + ANG (with and without the haptic device) along Body Ownership and Agency. We found that ANG (without the haptic device) was rated noticeably lower on the Agency than EMG and EMG + ANG (both without the haptic device). It is likely better to change the appearance of a muscle using the user's muscle contraction rather than the user's movement when considering the user's muscle ownership and agency. The addition of the ad hoc haptic device appeared to not affect the Sense of Embodiment.

Parallel view is a technique that allows a VR user to see multiple locations at a time. It enables the user to control several remote or virtual body parts while seeing parallel views to solve synchronous tasks. However, these techniques only explored the benefits and drawbacks of a user performing different tasks. In this paper, we explored enhancements on a singular or asynchronous task by utilizing information obtained in parallel views. We developed three prototypes where parallel views are fixed, moving in symmetric order, or following the user's eye gaze. We conducted a user study to compare each prototype against traditional VR (without parallel views) in three types of tasks: object search and interaction tasks in a 1) simple environment and 2) complex environment, and 3) object distances estimation task. We found parallel views improved multi-embodiment while each technique helped different tasks. No parallel view provided a clean interface, thus improving spatial presence, mental effort, and user performance. However, participants' feedback highlighted potential usefulness and a lower physical effort by using parallel views to solve complicated tasks.

As one of the techniques to recognize head-mounted display (HMD) user's facial expressions, the photo-reflective sensor (PRS) has been employed. Since the classification performance of PRS-based method is affected by rewearing an HMD and difference in facial geometry for each user, the user have to perform dataset collection for each wearing of an HMD to build a facial expression classifier. To tackle this issue, we investigate how transfer learning improve within-user and cross-user accuracy and reduce training data in the PRS-based facial expression recognition. We collected a dataset of five facial expressions (Neutral, Smile, Angry, Surprised, Sad) when participants wore the PRS-embedded HMD five times. Using the dataset, we evaluated facial expression classification accuracy using a neural network with/without fine tuning. Our result showed fine tuning improved the within-user and cross-user facial expression classification accuracy compared with non-fine-tuned classifier. Also, applying fine tuning to the classifier trained with the other participant dataset achieved higher classification accuracy than the non-fine-tuned classifier.

Generative adversarial networks (GANs) generate high-dimensional vector spaces (latent spaces) that can interchangeably represent vectors as images. Advancements have extended their ability to computationally generate images indistinguishable from real images such as faces, and more importantly, to manipulate images using their inherit vector values in the latent space. This interchangeability of latent vectors has the potential to calculate not only the distance in the latent space, but also the human perceptual and cognitive distance toward images, that is, how humans perceive and recognize images. However, it is still unclear how the distance in the latent space correlates with human perception and cognition. Our studies investigated the relationship between latent vectors and human perception or cognition through psycho-visual experiments that manipulates the latent vectors of face images. In the perception study, a change perception task was used to examine whether participants could perceive visual changes in face images before and after moving an arbitrary distance in the latent space. In the cognition study, a face recognition task was utilized to examine whether participants could recognize a face as the same, even after moving an arbitrary distance in the latent space. Our experiments show that the distance between face images in the latent space correlates with human perception and cognition for visual changes in face imagery, which can be modeled with a logistic function. By utilizing our methodology, it will be possible to interchangeably convert between the distance in the latent space and the metric of human perception and cognition, potentially leading to image processing that better reflects human perception and cognition.

Recent technological advances have enabled the use of AI agents to assist with human tasks and augment human cognitive abilities in a variety of contexts, including decision making. It is critical that users trust these AI agents in order to use them effectively. Given that people tend to trust other people who are similar to themselves, incorporating features of one's own face into the AI agent's face may improve one's trust in the AI agent. However, it is still unclear how impressions differ when comparing agents with the same appearance as one's own and some similarities under the same conditions. Recognizing the appropriate level of similarity when using a self-similar agent is important for establishing a trustworthy agent relationship between people and the AI agent. Therefore, we investigated the effect of the degree of self-similarity of the face of the AI agent on the user's trust in the agent. We examined users' impressions of four AI agents with different degrees of face self-similarity in different scenarios. The results showed that the AI agent, whose similarity to the user's facial feature was slightly recognizable but not obvious, received higher ratings on the feeling of closeness, attractiveness, and facial preferences. These self-similar AI agents were also more trustworthy in everyday non-serious decisions and were more likely to improve people's trustworthiness in such situations. Finally, we discuss the potential applications of our findings to design real-world AI agents.

In comics, Textual Sound Effects (TE) can describe sounds, but also actions, events, etc. TE could be used in Virtual Environment to efficiently create an easily recognizable scene and add more information to objects at a relatively low design cost. We investigate the impact of TE in a Virtual Environment on objects’ material perception (on category and properties) and on sound perception (on volume [dB] and spatial position). Participants (N=13, repeated measures) categorized metallic and wooden spheres and significantly changed their reaction time depending on the TE congruence with the spheres’ material/sound. They then rated a sphere’s properties (i.e., wetness, warmness, softness, smoothness, and dullness) and significantly changed their rating depending on the TE. When comparing 2 sound volumes, they perceived a sound associated with a shrinking TE as less loud and a sound associated with a growing TE as louder. When locating an audio source location, they located it significantly closer to a TE.

Visual-tactile synchronization or visual-motor synchronization makes a non-innate body feel like one's own body (illusory body ownership). A recent study has shown that body ownership is induced to an empty space between the hands and feet from the motion of only hands and feet (dynamic invisible body illusion). However, it is unclear whether both hands and feet are necessary to induce the illusion. In this study, we investigated the minimal condition for the dynamic invisible body illusion by manipulating the presentation of hands and feet. Our results suggest that both hands and feet are necessary for the illusion.

Human behavior and perception are optimized for a single body. Yet, the human brain has plasticity, which allows us to extend our body schema. By utilizing technology like robotics or virtual reality (VR), we can modify our body parts or even add a new body to our own while retaining control over these parts. However, the update of body cognition when controlling multiple bodies has not been well examined. In this study, we explore the task performance and body cognition of humans when they have multiple full bodies as an extended embodiment. Our experimental system allows a participant to control up to four bodies at the same time and perceive sensory information from them. The participant experiences synchronizing behavior and vision perception in a virtual environment. We set up two tasks for multiple bodies and evaluated the cognition of these bodies by their task performances and subjective ratings. We found that humans can have the sense of body ownership and agency for each body when controlling multiple bodies simultaneously. Distributed embodiment has the potential to extend human behavior in cooperative work, parallel work, group behavior, and so on.

Human behavior and perception are optimized for a single body. Yet, the human brain has plasticity, which allows us to extend our body schema. By utilizing technology like robotics or virtual reality (VR), we can modify our body parts or even add a new body to our own while retaining control over these parts. However, the update of body cognition when controlling multiple bodies has not been well examined. In this study, we explore the task performance and body cognition of humans when they have multiple full bodies as an extended embodiment. Our experimental system allows a participant to control up to four bodies at the same time and perceive sensory information from them. The participant experiences synchronizing behavior and vision perception in a virtual environment. We set up three tasks for multiple bodies and evaluated the cognition of these bodies with their gazing information, task performances, and subjective ratings. We found that humans can have the sense of body ownership and agency for each body when controlling multiple bodies simultaneously. Furthermore, it was observed that people manipulate multiple bodies by actively switching their attention in a static environment and passively switching their attention in a dynamic environment. Distributed embodiment has the potential to extend human behavior in cooperative work, parallel work, group behavior, and so on.

Peripersonal space (PPS) is important for humans to perform body–environment interactions. However, many previous studies only focused on the specific direction of the PPS, such as the front space, despite suggesting that there were PPSs in all directions. We aimed to measure and compare the peri-trunk PPS in four directions (front, rear, left, and right). To measure the PPS, we used a tactile and an audio stimulus because auditory information is available at any time in all directions. We used the approaching and receding task-irrelevant sounds in the experiment. Observers were asked to respond as quickly as possible when a tactile stimulus was applied to a vibrator on their chest. We found that peri-trunk PPS representations exist with an approaching sound, irrespective of the direction.

When humans generate stimuli voluntarily, they perceive the stimuli more weakly than those produced by others, which is called sensory attenuation (SA). SA has been investigated in various body parts, but it is unclear whether an extended body induces SA. This study investigated the SA of audio stimuli generated by an extended body. SA was assessed using a sound comparison task in a virtual environment. We prepared robotic arms as extended bodies, and the robotic arms were controlled by facial movements. To evaluate the SA of robotic arms, we conducted two experiments. Experiment 1 investigated the SA of the robotic arms under four conditions. The results showed that robotic arms manipulated by voluntary actions attenuated audio stimuli. Experiment 2 investigated the SA of the robotic arm and innate body under five conditions. The results indicated that the innate body and robotic arm induced SA, while there were differences in the sense of agency between the innate body and robotic arm. Analysis of the results indicated three findings regarding the SA of the extended body. First, controlling the robotic arm with voluntary actions in a virtual environment attenuates the audio stimuli. Second, there were differences in the sense of agency related to SA between extended and innate bodies. Third, the SA of the robotic arm was correlated with the sense of body ownership.

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

In the medical field, surgical videos can be used to introduce surgical skills. Medical students and residents watch the videos to study the surgical skills and increase learning speed by compensating for the lack of experience in surgical rooms due to limited opportunity to join in surgery. To record egocentric surgical videos by a wearable camera is a solution to record surgical skills of a surgeon in detail. However, most egocentric surgical videos are of quite long duration. For example, in the case of tumor removal in breast surgery, a video recording time often reaches 2h. With that length, it is time consuming to see important scenes in the video, particularly because many surgical videos include nonessential scenes such as sterilization and preparation of tools. For extracting specific scenes from a long video, we can apply scene estimation by machine learning. Furthermore, it is important to know where the surgeon is looking to observe the area of the incision in detail. In particular, it is vital to be able to zoom in on key elements, allowing viewers to see the incision area and the fine details of the necessary surgical skills. In this study, we aimed to highlight incision scenes from egocentric surgical videos in the spatiotemporal domain by utilizing two neural networks for the temporal and spatial highlights. For the temporal highlights, we designed a neural network that estimates the incision scenes by learning gaze speed, hand movements, number of hands, and background movements in egocentric surgical videos. For the spatial highlights, in order to estimate the important area to zoom in, we designed a neural network that learns the surgeon's gaze on natural features of surgical scenes to form a probability map as a representation of the estimated gaze area. The estimated gaze area was also used to calculate the appropriate zoom-in position and zoom-in ratio. To control the highlighted parameters in accord with user preferences, we also made a user interface that allows for the selection of playback speed gain and zoom ratio gain. For the evaluation, we verified the performance of the networks by a quantitative assessment and conducted a user study with medical doctors by showing an actual surgical video to obtain a qualitative assessment on the proposed system.

Is it possible to feel part of one’s own body even when the body part is separated from the body? If so, we could exist in different locations by splitting the self-body and using our body in each location. In a study in which the illusion that two bodies are one’s own body was induced using virtual reality (VR), the participants felt as if they were at two locations through the two bodies. However, this illusion was weak and reported only subjectively. We hypothesized that this was because two presented bodies moved synchronously with one participant’s movement or the simultaneous stroking of one participant’s body and two presented bodies switched their attention and weakened body ownership. In this study, we investigated whether splitting one body into left and right in VR could induce body ownership and extend the self-location while maintaining a one-to-one correspondence between the participant’s body and the presented body. The results showed that weaker body ownership was induced in the split body than in the normal body and self-location was extended to the right side of the body. The participants did not report a sense of having more than one body but reported a sense of body spreading, suggesting that the split bodies were perceived as a single body extending to the right side.