論文

The relationship between audience and performers is crucial to what makes live events so special. The aim of this work is to develop a new approach amplifying the link between audiences and performers. Specifically, we explore the use of wearable sensors in gathering real-time audience data to augment the visuals of a live dance performance. We used the J!NS MEME, smart glasses with integrated electrodes enabling eye movement analysis (e.g. blink detection) and inertial motion sensing of the head (e.g. nodding recognition). This data is streamed from the audience and visualised live on stage during a performance, alongside we also collected heart rate and eye gaze of selected audience. In this paper we present the recorded dataset, including accelerometer, electrooculography(EOG), and gyroscope data from 23 audience members.

Existing stage monitor mixing systems are inefficient and cannot accommodate the communication between the musicians and sound engineers. We introduce ARMixer, which allows musicians to perform self-stage monitor mixing through gestures in augmented reality to provide an intuitive mixing experience. We performed two usability tests and found that ARMixer is acceptable to the user and has excellent psychoacoustic intuitiveness in terms of mixing parameter controls by gestures and identifying mixing target.

Anxiety and stress have gradually become commonplace mental health problems afflicting many people worldwide. We propose asmVR, a novel approach to enhance the autonomous sensory meridian response (ASMR) experience by combining multimodal triggers including visual, auditory, tactile, and emotional stimuli. asmVR helps users enhance ASMR tingling sensations through online and offline modes, provides realistic VR environments and remote avatar ASMRtist for users, showcasing its potential for stress relief, emotion regulation, and customization. Additionally, it reveals new possibilities for the future application of VR in the field of psychotherapy.

Individuals are besieged by anxiety and stress throughout the world. Solutions to improve mental well-being can vary, and Autonomous Sensory Meridian Response (ASMR) is one potential method that has proven to be able to reduce stress. To that end, we introduce asmVR, a novel approach to enhancing ASMR experiences using multi-modal triggers. Combining online and offline modes, asmVR enhances ASMR tingling, offering immersive VR environments and remote ASMRist embodiments. Initial user testing shows tingles enhance and stress relief potential, along with new possibilities for VR in psychological therapy.

Haptic Augmented Reality (HAR) is a method that actively modulates the perceived haptics of physical objects by presenting additional haptic feedback using a haptic display. However, most of the proposed HAR research focuses on modifying the hardness, softness, roughness, smoothness, friction, and surface shape of physical objects. In this paper, we propose an approach to augment the perceived stickiness of a physical object by presenting additional tactile feedback at a particular time after the finger lifts off from the physical object using a thin and soft tactile display suitable for HAR. To demonstrate this concept, we constructed a thin and soft tactile display using a Dielectric Elastomer Actuator suitable for HAR. We then conducted two experiments to validate the effectiveness of the proposed approach. In Experiment 1, we showed that the developed tactile display can augment the perceived stickiness of physical objects by presenting additional tactile feedback at appropriate times. In Experiment 2, we investigated the stickiness experience obtained by our proposed approach and showed that the realism of the stickiness experience and the harmony between the physical object and the additional tactile feedback are affected by the frequency and presentation timing of the tactile feedback. Our proposed approach is expected to contribute to the development of new applications not only in HAR, but also in Virtual Reality, Mixed Reality, and other domains using haptic displays.

The recent emergence of point cloud streaming technologies has spawned new ways to digitally perceive and manipulate live data of users and spaces. The graphical rendering limitations prevent state-of-the-art interaction techniques from achieving segmented bare-body user input to manipulate live point cloud data. We propose BridgedReality, a toolkit that enables users to produce localized virtual effects in live scenes, without the need for an HMD nor any wearable devices or virtual controllers. Our method uses body tracking and an illusory rendering technique to achieve large scale, depth-based, real time interaction with multiple light field projection display interfaces. This toolkit circumvented time-consuming 3D object classification, and packaged multiple proximity effects in a format understandable by middle schoolers. Our work can offer a foundation for multidirectional holographic interfaces, GPU simulated interactions, teleconferencing and gaming activities, as well as live cinematic quality exhibitions.

Humans have spatiotemporal limitations. Robotic avatar technology enables the expansion of an individual's innate physical characteristics, physical capabilities, and perceptual and cognitive abilities. Our research group has focused on scalability and co-creativity through co-operation and collaboration between humans, and has developed a system in which two operators are physically integrated into a single robotic arm. In particular, to enable more human-like tasks, such as receiving and delivering, we extended the system to a dual-arm robotic avatar and built a system in which multiple persons can integrate their bodies, with the expectation of further task efficiency and human extension. Specifically, we investigated the workability and operator cognition of the dual-arm robotic avatar when it was operated by one, two, or three individuals. The results showed that stability improved with the three-person operation. Furthermore, it was found that co-operation among multiple persons can overcome the original individual abilities for operation. In the three-person condition, all participants obtained a higher feeling of control than the actual amount of manipulation, which has potential applications in skill transfer. The three-person condition tended to be better, validating the usefulness of multi-person body integration.

In this study, we developed a system in which two users are integrated into one actual robot arm to collaborate with each other. The method of dividing the roles allows flexible movements that is not dependent on the range of movement and rotational freedom of the human arm, and enables movements that are impossible for a single person. The mixing of movements in adjustable ratio allows an expert to support a beginner to make more stable movements, and induces collaboration between users with different perspectives. Additionally, we implement tactile feedback to enable interaction between users and between users and the robot. We investigate the effects of these on usability and cognitive behavior. This system is expected to become a new method of collaboration in the cyber-physical society.

Current video call systems can facilitate formal conversation, yet they lack the ability to provide users with an informal sense of presence. To this end, we propose Comado, a communication system that creates an ambient connection using online interactive haptic sharing. Outside of face-to-face communication, Comado provides haptic feedback to the desk to convey a sense of presence. From preliminary results, we plan to further improve the design of Comado to be deployed in the wild.

In this study, we propose Conversation Echo, a system that reflects the topics of conversation in a VR environment in real time. This method uses AI to convert speech data into text, extract conversation topics, and generate panoramic images to generate a VR environment, which dynamically changes the environment in real-time. This method aims to realize an experience that generates conversation topics and inspiration.

To enhance the dining experience, prior studies in Human-Computer Interaction (HCI) and gastrophysics have demonstrated that modifying the static shape of solid foods can amplify taste perception. However, the exploration of dynamic shape-changing mechanisms in liquid foods remains largely untapped. In the present study, we employ cymatics, a scientific discipline focused on utilizing sound frequencies to generate patterns in liquids and particles-to augment the drinking experience. Utilizing speakers, we dynamically reshaped liquids exhibiting five distinct taste profiles and evaluated resultant changes in taste perception and drinking experience. Our research objectives extend beyond merely augmenting taste from visual to tactile sensations; we also prioritize the experiential aspects of drinking. Through a series of experiments and workshops, we revealed a significant impact on taste perception and overall drinking experience when mediated by cymatics efects. Building upon these findings, we designed and developed tableware to integrate cymatics principles into gastronomic experiences.

Dementia describes a syndrome of cognitive degeneration, and Behavioural and Psychological Symptoms of Dementia (BPSD) is the non-cognitive symptom. BPSD can be improved by care services. To aid better care service, we explore the potential of using Augmented Reality (AR) to support dementia education for medical workers in three steps: (1) We explore medical workers' perspective on dementia care lived experience and XR, (2) we co-design an educational experience containing an AR-based application and a 5-min activity with medical workers, (3) we evaluate the effectiveness of the system through a mixed method study. Our result shows that the AR experience successfully touches participants, and motivates them to reflect on the provision of care service. On this basis, we discuss the elements and challenges of designing XR-enabled dementia education for users unfamiliar with novel technology, and the potential of using XR in clinical education.

Dementia is a global health crisis, of which there is a need to understand the patients' perception towards improving their quality of life. We propose Dementia Eyes, a mobile AR experience that simulates common visual symptoms of senile dementia based on the known pathology and caregivers' actual experience with patients. Leveraging an iPhone and a Head-Mounted Display (HMD), we developed a real-time application which allows users to see the world from the perspective of an Alzheimer's type of dementia (AD) patient. The experience was validated by professional medical workers in Japan, and the result advocates for the efficacy of the empathy we intended to bring to them.

Pedestrian navigation systems on smartphones have become integral to modern society. However, a crucial problem in pedestrian navigation is that visual and/or auditory cues can distract situational awareness while walking, which may lead to serious consequences. To address this problem, various haptic navigation devices have been developed. Among these, a pseudo-force sensation elicited by asymmetric vibrations on the skin is considered a promising method owing to its intuitive directional cues. Nevertheless, pseudo-force sensation for pedestrian navigation has yet to be well validated in actual walking situations. Here, we propose an untethered pseudo-force haptic device for pedestrian navigation. Our device controls the direction of pseudo-force sensation by delivering asymmetric vibrations to the fingertips using a pair of vibrotactile actuators. Perceptual and navigation experiments were conducted to examine the effectiveness of the proposed pseudo-force navigation device. Our results suggest that pseudo-force navigation is effective during actual walking.