Published papers

For a companion robot that follows a person as an assistant, predicting human walking is important to produce a proactive movement that is helpful to maintain an appropriate area decided by the human personal space. However, fully trusting the prediction may result in obstructing human walking because it is not always accurate. Hence, we consider the estimation of uncertainty (i.e., entropy) of the prediction to enable the robot to move without causing overconfident motion and without being late for the person it follows. To consider this uncertainty of the prediction to the controller, we introduce a reliability value that changes based on the entropy of the prediction. This value expresses the extent the controller should trust the prediction result, and it affects the cost function of our controller. We propose an uncertainty-aware robot controller based on nonlinear model predictive control to realize natural human-followings. We found that our uncertainty-aware control system can produce an appropriate robot movement, such as not obstructing the human walking and avoiding delay, in both simulations using actual human walking data and real-robot experiments.

This study aims to evaluate effects of different lower limb joint torque constraints on the motion trajectory of an sit-to-stand movement (STS) grasping an assistive device by simulating the STS with an assistive device using a human model with different lower limb joint torque constraints at various second lengths. This is an initial study for estimating lower limb muscle strength from the motion trajectory of the STS grasping an assistive device. In this study, instead of measuring many human subjects, optimal motion trajectories are calculated and simulated. Equations of the STS using the assistive device are first established using a rigid link model with both hands grasping the assistive device. Then, equations of motion and physical constraints such as joint range of motion and torque limitations are used as constraint conditions, and the sum of squares of joint torque, which calculates the action with the minimum muscle load on the body, is used as an evaluation function to optimize the STS. During optimization, calculations are performed with several types of joint torque conditions and end-time, i.e., the time required to complete the operation. By comparing several types of joint torque condition movements, the differences in the effects of different joint torque constraints on the motion trajectory of the STS grasping the assistive device can be determined.

We present UnrealEgo, i.e., a new large-scale naturalistic dataset for egocentric 3D human pose estimation. UnrealEgo is based on an advanced concept of eyeglasses equipped with two fisheye cameras that can be used in unconstrained environments. We design their virtual prototype and attach them to 3D human models for stereo view capture. We next generate a large corpus of human motions. As a consequence, UnrealEgo is the first dataset to provide in-the-wild stereo images with the largest variety of motions among existing egocentric datasets. Furthermore, we propose a new benchmark method with a simple but effective idea of devising a 2D keypoint estimation module for stereo inputs to improve 3D human pose estimation. The extensive experiments show that our approach outperforms the previous state-of-the-art methods qualitatively and quantitatively. UnrealEgo and our source codes are available on our project web page (https://4dqv.mpi-inf.mpg.de/UnrealEgo/ ).

For intelligent systems, it is important to understand the action possibility for agent in real space. As the action possibility varies with the subsystem configuration of the agent and its states, the possibilities should be understood based on the world state comprising the agent’s state as well as the environmental state. However, most conventional methods consider only the environmental state. Therefore, this study proposes a world state-dependent action graph based on knowledge representation using scene graphs which allows the capturing of the action possibility of agents, which implies the feasible actions and their positions in real space, and their recursive variations depending on the world state. Moreover, the effectiveness of the proposed method was verified with simulations, assuming a coffee shop environment.

Recent studies have explored the integration of large language models (LLMs) into caregiving robots. The use of LLMs facilitates the generation of human-like natural dialogues and diverse, varied expressions. However, in case of a lack of direct and specific instructions from a user, gaining an accurate understanding of the user's intent is challenging, which may result in miscommunication. Thus, this study proposed a method to achieve robust communication even in the absence of direct instructions by considering the uncertainty in destination prediction probabilities. The proposed system leveraged GPT-4 to generate various Japanese speech content candidates and employed neighborhood components analysis to facilitate dimensionality reduction. Consequently, notable features of the rich linguistic embeddings were preserved while transforming them into manageable data. The system selected appropriate utterances based on the destination prediction probabilities. Furthermore, this study investigated changes in the speech content according to the variations in these probabilities, and the incorporation of uncertainty in communication was validated.

To facilitate the integration of autonomous mobile robots into environments shared with humans, ensuring the comfort of pedestrians and observers (individuals observing the interaction from a third-party perspective) is crucial. This study explores the correlation between pedestrian and observer comfort and the robot's distance and pose angle during pedestrian encounters. The findings indicate that when a robot passes a pedestrian after completing its avoidance motion, it enhances the comfort of both pedestrians and observers. In addition, when a robot maintains a large distance from pedestrians while passing by, it improves pedestrian comfort. Furthermore, designing the robot's pose during passing so that it appears to move away from the pedestrian from the observer's perspective further increases the observer's comfort.

This study proposes an image-based action generation method using a controller that is trained with experiences autonomously collected by a robot. Operating a robot using its state or point in coordinate space is not always easy for human operators to control. This is because humans do not interpret the world as a coordinate space. Depending on the task, inputs such as voice-input or image-input are easier for human operators to use when managing a robot. Accordingly, we studied an image-based control method of robotic agents. Designing an image-based controller by hand for different tasks and input images is highly complex. Therefore, a controller which can be automatically trained from experiences collected by a robot is strongly preferred. In addition, when considering the operation of a robot in a real environment, the controller should guarantee the safety of a robot’s behavior in a way that is configurable and understandable by the human operator. However, most previous approaches trained the controller end-to-end, which does not guarantee the safety of the behavior learned by the controller. Instead of training the controller end-to-end, we train state prediction and cost estimation functions to solve the action generation as a path planning problem. By doing so, we can explicitly design and specify the undesired state of a robot in the configuration space. The results show that the proposed method can be used to provide safe navigation to different goal positions in a realistic living room-like environment with an hour of training data.

Reducing traffic accidents pertaining to autonomous vehicles has garnered attention. Merging on a highway is one of the most challenging problems that must be addressed for the realization of autonomous vehicles. It is difficult because an agent must decide where to merge in a complex and ever-changing environment. Merging with congested highway traffic involves significant interaction with vehicles in the main lane. If there is no space for the autonomous vehicle to merge, it needs to work on vehicles in the main lane to create space and subsequently decide to merge or not. Reinforcement learning (RL) is a promising method for solving decision-making problems. However, it is difficult to guarantee the safety of the controller obtained using RL. Therefore, we propose a combined method in which decision-making is performed by RL and vehicle control by model predictive control (MPC) to ensure safety. The performance of the proposed system is tested by simulations. The proposed system made appropriate decisions according to the situation, and by controlling the vehicle in consideration of collision avoidance constraints, it showed a high merge success rate evenin a crowded situation.

Autonomous service robots are increasingly necessary to move without impeding the movement of pedestrians. Previous studies have determined optimal input for robots by minimizing a multi-objective function that includes the cost of reaching the destination and avoiding surrounding pedestrians. However, it is challenging to adjust the weights of each term in the cost function since they depend on the users and environment. In this study, we used the Social Force Model (SFM) as the base cost function and proposed a method to estimate SFM weights preferred by general user based on population density from human feedback. To achieve this, first we use Bayesian optimization and derive each user's evaluation map of SFM in a virtual reality environment that provides a realistic and immersive experience for subjects to provide feedback on the robot's movement. Second, we aggregated each user's evaluation map to estimate a general user's evaluation map. Finally, we have derived a functional relationship between the preferred SFM weights of general users and population density by Gaussian process regression. This relationship empowers the robot to navigate in a manner preferred by the general public, contingent on population density, even in the absence of human feedback obtained through virtual reality experimentation.

Recently, an alternative pressure mattress has been proposed to prevent pressure ulcers. In this study, we aim to design a control system for body pressure distribution using the mattress. This study focuses on modeling the dynamics between the body pressure distribution and the mattress. As modeling based on first principles is challenging, we identify the model from experimental data. However, when high-dimensional data such as body pressure distribution are directly used for identification, the model accuracy can deteriorate. Therefore, we need to determine the suitable model structure or dataset. Herein, we examined how the prediction accuracy of body pressure distribution varies with the dataset and model structure. We also proposed a method for representing the model as the dynamics of the parameters of the Gaussian Mixture Model to suppress the estimated parameters. To the best of our knowledge, this study is the first to propose the body pressure model as dynamics of the parameters of the Gaussian Mixture Model.

Collaborative delivery robots in hospitals are required to move safely and efficiently in a short time, without colliding with people. Hence, they must consider the risk of people rushing out from blind spots or rooms, including field context such as the role and usage of the location. However, these factors are difficult to extract solely from geometric information. Therefore, we propose a method for generating a rush-out risk map considering the field context from the hospital staf's operation data of an electric wheelchair. We convert the wheelchair's speed operated by staff into rush-out risk, and then place rush-out risk potentials at positions where rush-outs may occur. Subsequently, we optimize the mapping position of rush-out risk and parameters of each potential to minimize the error to obtain a rush-out risk map. We collected actual staff operation data in the hospital and confirmed that we could generate a rush-out risk map with small errors.

Sleep apnea hypopnea syndrome (SAHS) is one of the most common sleeping disorders. However, numerous people suffering from SAHS are left undiagnosed. SAHS causes arousal reactions, sleep fragmentation, and hypoxemia, thus leading to excessive daytime sleepiness, lack of concentration, and a high risk of cardiovascular complications. Although polysomnography is the standard way of SAHS screening, its high cost, limited accessibility, and requirement of operational skills hinder its applicability for SAHS diagnoses. Therefore, this letter proposes a novel approach for SAHS detection using a patch-type photoplethysmography sensor. As an initial study, both respiratory and pulse information are simultaneously measured, and the trend of pulse variation, which reportedly has a strong correlation to oxygen saturation (SpO2), is thereby clarified. In addition, more real-time measurement of the pulse rate was possible on the chest compared to the fingertip. SAHS detection by incorporating respiratory and pulse information is beneficial for complete and credible screening and risk assessment of cardiovascular complications with less invasive devices. Therefore, this simple patch-type sensor can replace invasive, expensive, and limiting screening methods to facilitate the diagnosis of SAHS.

Lately, there has been a need to improve the efficiency of material movements within factories and multi-agents are required to perform these tasks. In this study, graphical representation and mixed-integer programming have been adopted for simultaneous optimization of task allocation and path planning for each agent to achieve the following three goals. First, this study realizes time and capacity constrained multi-agent pickup and delivery (TCMAPD) that simultaneously considers time constraints, capacity constraints, and collision avoidance. Previous studies have not considered these constraints simultaneously. Thus, we can solve the problems associated with using multi-agents in actual factories. Second, we achieved TCMAPD that optimizes the collision avoidance between multi-agents. In conventional research, only a single collision avoidance method can be used. However, an appropriate route was selected from a variety of avoidance methods in this study. Hence, we could achieve a more efficient task allocation and path planning with collision avoidance. Third, the proposed method simultaneously optimizes task allocation and path planning for each agent. Previous studies have separately considered the approach of optimizing task allocation and path planning or used the cost of path planning after task allocation to again perform task allocation and path planning. To simultaneously optimize them in a single plan, we have developed a solution-derivable formulation using mixed-integer programming to derive a globally optimal solution. This enables efficient planning with a reduced total time traveled by the agents.

Social robots are used to perform mobile manipulation tasks, such as tidying up and carrying, based on instructions provided by humans. A mobile manipulation planner, which is used to exploit the robot's functions, requires a better understanding of the feasible actions in real space based on the robot's subsystem configuration and the object placement in the environment. This study aims to realize a mobile manipulation planner considering the world state, which consists of the robot state (subsystem configuration and their state) required to exploit the robot's functions. In this paper, this study proposes a novel environmental representation called a world state-dependent action graph (WDAG). The WDAG represents the spatial and temporal order of feasible actions based on the world state by adopting the knowledge representation with scene graphs and a recursive multilayered graph structure. The study also proposes a mobile manipulation planning method using the WDAG. The planner enables the derivation of many effective action sequences to accomplish the given tasks based on an exhaustive understanding of the spatial and temporal connections of actions. The effectiveness of the proposed method is evaluated through practical machine experiments performed. The experimental result demonstrates that the proposed method facilitates the effective utilization of the robot's functions.

The conventional joystick, the most common input device for powered wheelchairs, occupies one hand during operation and interferes with activities of daily living. Existing hands-free trunk-motion interfaces require extensive trunk movements, making them unsuitable for users with limited stability. Moreover, they increase the risk of falling and psychological load, and calibration schemes that explicitly account for left-right asymmetry in trunk kinematics are lacking. This study introduces a hands-free belt-type interface that senses subtle belt tensions via a six-axis force/torque sensor mounted on the backrest and maps them to translational and angular velocity commands. The interface frees the user's hands while minimizing the need for large postural adjustments during wheelchair operation. To handle asymmetrical motor function frequently observed in users with physical disabilities, we also introduced a body-axis calibration method.An empirical study involving path-following tasks was conducted with twelve able-bodied participants and two participants with physical disabilities. Statistical non-inferiority testing confirmed that, in the able-bodied group, path-tracking accuracy with the belt interface was not inferior to that with joystick control. In the group with physical disabilities, body-axis calibration significantly enhanced the belt interface's path-tracking accuracy, narrowing the performance gap to joystick control.

Active debris removal has become crucial. The removal of the upper stages of the rocket, which are the most common type of debris, is receiving increased attention. In the case of debris removal from the upper stage of a rocket, it is considered to recognize the circular structure of the upper stage of the rocket, which appears as an ellipse on the image plane, as a payload attach fitting (PAF) using the navigation sensor, and to approach it. Approaching the debris requires tracking the ellipses during the approach after initially recognizing the inner and outer ellipses of the PAF. Thus, the navigation sensor must be robust to sunlight for stable ellipse recognition. Consequently, we propose an initial PAF recognition method using an event camera due to its robust features as a navigation sensor. In the initial recognition, image processing is applied to event frames that accumulate asynchronous output events. As the inner ellipse of the PAF is suitable for feature extraction, we proposed a method to recognize the outer ellipse using the value of the inner ellipse after initial recognition of the inner ellipse. Additionally, when the recognition of the inner or outer ellipse fails, we propose extending the event accumulation time and repeating the process. The effectiveness of the proposed method was verified by numerical analysis using a simulator.

Landing dynamics play a crucial role in the success of planetary missions, particularly in low-gravity environments, and determining optimal landing sites continues to be challenging. This paper reports a method for landing safety evaluation based on spacecraft dynamics for landings on the surface layer of small bodies in the Martian moons exploration project. Unlike conventional short-duration contact explorations, spacecraft designed for the exploration of Martian moons will remain on the surface of Phobos for several hours post landing. Owing to low gravitational forces, small bodies exhibit an inadequate restoring force, which can result in overturning and rebounding of the spacecraft; moreover, the gravitational forces on Martian moons are several tens of times greater than that on other small bodies, such as Ryugu and Itokawa, necessitating a consideration of the impact acceleration during landing. In this study, a distinctive model was constructed based on low-gravity experimental results, with the capacity to replicate the flexibility of the spacecraft structure, tank sloshing, shock absorption, and interaction with the surface layer. Subsequently, we confirmed the unique behaviors of the spacecraft during its landing simulation on authentic terrain, and candidate landing areas were evaluated based on nine safety criteria. This evaluation process entailed the identification of safe landing areas, yielding a 20% reduction in the number of safe landing sites compared with that based exclusively on terrain.

Chronic obstructive pulmonary disease (COPD) is the third leading cause of death in the world, and doctors expect the number of patients to increase by 23% in 2050. Patients experience dyspnea during physical exertion due to compromised pulmonary gas exchange capacity. Deterioration of pulmonary function is expected to appear in the Inspiratory to Expiratory ratio (I:E ratio). However, the I:E ratio is often unmonitored even in hospitals. The I:E ratio is not yet a standard index for health monitoring; therefore, it remains untouched by most studies. Therefore, we aim to estimate the I:E ratio from the proposed patch-type multimodal sensor for pulmonary function screening of prediagnosed COPD patients. We conducted experiments with three types of I:E ratios on 14 healthy participants. The first objective was a trend analysis of photoplethysmography (PPG) and three-axis acceleration respiratory signals acquired from the proposed multimodal sensor. From the results, the best results of the four signals differed depending on signal amplitude, baseline wander, inconsistency, and vibration. Hence, by formulating a signal quality evaluation step, we proposed a signal selection method to systematically determine the best of the four signals for I:E ratio estimation.

The recent advancements in space exploration technology have led to significant progress in sampling techniques within microgravity environments. In particular, there is a growing demand for employing redundant joint manipulators in sample collection. Collecting a larger number of samples at greater depths from the targeted ground holds immense scientific and engineering value. However, when sampling in a microgravity environment, substantial force is exerted on the manipulator tip upon sampler ejection, resulting in acceleration fluctuations (AFs). These fluctuations can cause the sample to drop from the sampler, significantly impacting mission feasibility. Consequently, there is a critical need for a sampling approach that minimizes AF to the largest extent possible. This study introduces a methodology for determining the initial posture of a manipulator and a control technique to ensure successful sample collection within a microgravity environment, assuming a probe equipped with a redundant joint manipulator. The initial posture determination evaluates the acceleration variation based on the inverse inertia matrix, which is a kinematic index, presenting a method that circumvents dynamic simulation. However, despite establishing a posture with minimal AFs in the initial posture determination, AFs still persist. Therefore, we propose a control method that further suppresses AFs utilizing optimal regulator control. The effectiveness of the proposed system is assessed using a simulator developed utilizing the MATLAB Simscape Multibody library, providing validation of its capabilities.

Recently, lunar exploration has been actively carried out. This paper focuses on the Pressurized Rover (PR) mission. For PR mission, two astronauts need to live for a month in a small, enclosed space without private rooms, even though human needs our own private space. This paper examines the possibilities of using spotlights to solve this problem. The use of non-uniform lighting is known to affect human behavior and communication style. Based on this theory, this paper discusses how to design a spotlight system that adjusts lighting based on feedback from vital signs so that astronauts can keep comfortable in space. To verify the basics of the proposed system, an experiment was conducted in which two subjects stayed in a limited space, which size is the same as a PR. The main objective of this experiment is to investigate the influence of spotlights on human behavior. The second objective is to verify how human pleasure and displeasure appear in their vital signs. The experimental procedure is as follows. Only one subject is placed under spotlight and sits in the same place as a non-moving subject, and the other moves a chair to sit under various illumination areas. We measured the distance between them and the pulse rate and gaze of the non-moving subject. It was confirmed that in proportion to the illumination radius the distance between the two increased, and the direction of gaze moved. An amount of change of illumination radius variation also affected the distance and the direction of gaze. It was also confirmed that the distance between the two people was shorter with spotlight than without. It means that the spotlight can affect the distance between two people. Therefore, it is suggested that even if sufficient space cannot be secured in a small, enclosed space, the two people live comfortably together to decrease illumination radius, which lead to shorten the distance between two people and reduce the degree of attention from the other person. When non-mover is displeasure, the direction of gaze shifted toward the mover. At the same time, it was confirmed that pulse rate was decreased. In conclusion, the study suggested that changing the spotlight in a limited space may help control behavior and mood, leading to a comfortable life.