Published papers

We are developing a service platform to achieve use cases such as finding people in trouble and sharing them with AR applications to ask for help from the people around them, finding information about a sale at a store and sharing it with a map service, or alerting the police when a crime occurs, as well as quickly alerting the people around them. Our system detects “objects” and “events” in the real world using edge devices that can share the detected information in cyberspace so that the users can remotely obtain real-world information from distant locations. We believe our system can make life more safe and convenient. In this paper, we report on the findings from our experiments on searching for lost dogs by mapping object recognition results, on mapping bird habitat areas by environmental sound recognition results, and on mapping public facility announcement voices by speech recognition results.

We are developing a system that detects "objects"and "events"in the real-world using edge devices that can share the detected information among them in the cyber-space (virtual world) so that the users can remotely obtain real-world information from distant locations. We aim for the system can serve as a foundation for services that construct and utilize a digital twin of the real world. In this paper, we report our findings by implementing "objects"and "things"detection in edge devices (smartphones) and making experiments for sharing the detected "objects"and "things"registered on a digital map.

Surgical tool detection is essential for analyzing and evaluating minimally invasive surgery videos. Current approaches are mostly based on supervised methods that require large, fully instance-level labels (i.e., bounding boxes). However, large image datasets with instance-level labels are often limited because of the burden of annotation. Thus, surgical tool detection is important when providing image-level labels instead of instance-level labels since image-level annotations are considerably more time-efficient than instance-level annotations. In this work, we propose to strike a balance between the extremely costly annotation burden and detection performance. We further propose a co-occurrence loss, which considers a characteristic that some tool pairs often co-occur together in an image to leverage image-level labels. Encapsulating the knowledge of co-occurrence using the co-occurrence loss helps to overcome the difficulty in classification that originates from the fact that some tools have similar shapes and textures. Extensive experiments conducted on the Endovis2018 dataset in various data settings show the effectiveness of our method.

Electrical cables consist of numerous wires. The 3D shape of individual wires in the cables significantly affects their characteristics. With the increasing diversity of wire structures, understanding the 3D shape of each wire in an electrical cable is crucial for analyzing characteristics such as bending stiffness. To accurately estimate the bending stiffness of actual electrical cables, a detailed 3D representation of each wire is required. Therefore, it is important to obtain the 3D shape of actual electrical cable wires using non-destructive inspection. In this study, we propose a new method to associate wires using particle tracking techniques. The proposed method performs wire tracking in two steps, linking detected wire positions between adjacent frames and connecting segmented wires across frames. The effectiveness of the proposed method was evaluated quantitatively and qualitatively. In the quantitative evaluation, the correctness of the tracking was evaluated using the data with artificial noise added to the real data detection results. The proposed method achieves highly accurate wire tracking when there is little undetected noise, and shows robustness to over-detection.

Automatic fall detection is a crucial task in healthcare as falls pose a significant risk to the health of elderly individuals. This paper presents a lightweight acceleration-based fall detection method that can be implemented on edge devices. The proposed method uses Autoencoders, a type of unsupervised learning, within the framework of anomaly detection, allowing for network training without requiring extensive labeled fall data. One of the challenges in fall detection is the difficulty in collecting fall data. However, our proposed method can overcome this limitation by training the neural network without fall data, using the anomaly detection framework of Autoencoders. Additionally, this method employs an extremely lightweight Autoencoder that can run independently on an edge device, eliminating the need to transmit data to a server and minimizing privacy concerns. We conducted experiments comparing the performance of our proposed method with that of a baseline method using a unique fall detection dataset. Our results confirm that our method outperforms the baseline method in detecting falls with higher accuracy.

Augmented reality (AR) provides ways to visualize missing view samples for novel view synthesis. Existing approaches present 3D annotations for new view samples and task users with taking images by aligning the AR display. This data collection task is known to be mentally demanding and limits capture areas to pre-defined small areas due to the ideal but restrictive underlying sampling theory. To free users from 3D annotations and limited scene exploration, we propose using locally reconstructed light fields and visualizing errors to be removed by inserting new views. Our results show that the error-peaking visualization is less invasive, reduces disappointment in final results, and is satisfactory with fewer view samples in our mobile view synthesis system. We also show that our approach can contribute to recent radiance field reconstruction for larger scenes, such as 3D Gaussian splatting.

Cervical myelopathy (CM) is a pathology of the spinal cord that causes upper limb disorders. CM is often screened by conducting the 10-s grip and release (G&R) test, which mainly focuses on hand dysfunction caused by CM. This test has patients repeat gripping and releasing their hands as quickly as possible. Spine surgeons observe the quickness of this repetition to screen for CM. We propose an automatic screening method of CM that involves patients’ hands recorded as videos when they are performing the G&R test. The videos are used to estimate feature values, i.e., the positions of each part of the hand, which are obtained through image processing. A support vector machine classifier classifies CM patients and controls with these feature values after pre-processing. We validated our method with 10-fold cross-validation and the videos of 20 CM patients and 15 controls. The results indicate that sensitivity, specificity, and area under the receiver operating characteristic curve were 90.0 %, 93.3 %, and 0.947, respectively.

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.

In this paper, we present an end-to-end unsupervised anomaly detection framework for 3D point clouds. To the best of our knowledge, this is the first work to tackle the anomaly detection task on a general object represented by a 3D point cloud. We propose a deep variational autoencoder based unsupervised anomaly detection network adapted to the 3D point cloud and an anomaly score specifically for 3D point clouds. To verify the effectiveness of the model, we conducted extensive experiments on ShapeNet dataset. Through quantitative and qualitative evaluation, we demonstrate that the proposed method outperforms the baseline method.

In recent years, much attention has been paid to the prediction of pedestrian trajectories, as they are one of the key factors for a better society, such as automatic driving, a guide for blind people, and social robots interacting with humans. To tackle this task, many methods have been proposed but few are from the first-person perspective because of the lack of a publicly available dataset. Therefore, we propose a method that uses egocentric vision, which does not need to be trained with a first-person video dataset. We made it possible to utilize existing methods, which predict from a bird’s-eye view. In addition, we propose a novel way to consider semantic information without changing the shape of the input to apply to all existing bird’s-eye methods that use only past trajectories. Therefore, there is no need to create a new dataset from egocentric vision. The experimental results demonstrate that the proposed method makes it possible to predict from an egocentric view via existing methods of bird’s-eye view. The proposed method qualitatively improves trajectory predictions without aggravating quantitative accracy, and the effectiveness of predicting the trajectories of multiple people simultaneously.

Reliably and accurately forecasting future trajectories of pedestrians is necessary for systems like autonomous vehicles or visual assistive devices to function correctly. While previous state-of-the-art methods relied on modeling social interactions with LSTMs, with videos captured with a static camera from a bird’s-eye view, our paper presents a new method that leverages the Transformers architecture and offers a reliable way to model future trajectories in first-person videos captured by a body-mounted camera, without having to model any social interactions. Accurately forecasting future trajectories is a challenging task, mainly due to how unpredictably humans move. We tackle this issue by using information about target persons’ previous locations, scales, and dynamic poses, as well as information about the camera wearer’s ego-motion. The model we propose predicts future trajectories in a simple way, modeling each target’s trajectory separately, without the use of complex social interactions between humans or interactions between targets and the scene. Experimental results show that our method overall outperforms previous state-of-the-art methods, and yields better results in challenging situations where previous state-of-the-art methods fail.

Extended reality (XR) refers to a technological group namely Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). While the operator cannot view the actual environment in VR, AR can allow him to view the actual surroundings with superimposed virtual items. However, MR is anything in between VR and AR. The Microsoft Hololens headset is a key contributor by focusing on MR technology and it is expensive. Therefore, this research proposes especially for proper investment using VR and AR platforms for complicated motorcycle engine assembly. Consequently, the authors built a fully immersive VR platform using Unreal Engine and HTC Vive. Two AR (marker-less and marker-based) platforms were also developed using Unity 3D and EPSON MOVERIO BT-300. Fourteen students from engineering education participated in the four experiments, which were separated into four groups: (i) video, (ii) VR, (iii) marker-less AR, and (iv) marker-based AR. All participants were then requested to complete all experiments to test their satisfaction levels. In comparison, the outputs showed that the marker-less AR was 34.4% superior, followed by the VR (31.6% superior), and finally the marker-based AR (14.4% superior) to the video-based method. In addition, students prefer the immersive XR (VR and AR) platforms to perform the engine assembly properly.

Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery.

Electrical cables consist of numerous wires, the three-dimensional (3D) shape of which significantly impacts the cables' overall properties, such as bending stiffness. Although X-ray computed tomography (CT) provides a non-destructive method to assess these properties, accurately determining the 3D shape of individual wires from CT images is challenging due to the large number of wires, low image resolution, and indistinguishable appearance of the wires. Previous research lacked quantitative evaluation for wire tracking, and its overall accuracy heavily relied on the accuracy of wire detection. In this study, we present a long short-term memory-based approach for wire tracking that improves robustness against detection errors. The proposed method predicts wire positions in subsequent frames based on previous frames. We evaluate the performance of the proposed method using both actual annotated cables and artificially noised annotations. Our method exhibits greater tracking accuracy and robustness to detection errors compared with the previous method.

Ball tracking based on sports videos captured by high-speed cameras has had limited applicability due to the need for sufficient lighting and the necessity of abundant computational resources to process the large amount of captured data. To overcome these problems, this paper addresses ball tracking using an event camera. An event camera outputs luminance changes in a scene as "events" and is characterized by high temporal resolution, high dynamic range, and data efficiency. The proposed method selects the ball position from the bounding boxes output by the object detector by taking into account the ball motion estimated from the events and the past ball positions using Contrast Maximization (CMax). Considering past ball positions enables us to estimate the ball trajectory consistently over time and detect the ball position more accurately. Additionally, CMax is applied to the events within the tracked ball bounding box, allowing for the estimation of the ball spin. Experimental results in this paper show that the proposed method is quantitatively and qualitatively more accurate than conventional ball tracking methods and demonstrates the potential ability to estimate ball spin.

For human-safe robot control in human-to-robot handover, the physical properties of containers and fillings should be accurately estimated. In this paper, we propose a Transformer encoder that shares the same architecture and parameters for filling level and type estimation. We also propose a mask-based geometric algorithm to estimate 3D models of containers for the estimation of their capacity and dimensions. We further use these estimations to estimate their mass in a Convolutional Neural Network model. Experiments show that our Transformer model produced encouraging results in both estimations. While challenges remain in our mask-based algorithm and Convolutional Neural Network model, their results revealed several ways for improvement.

In this paper, we propose a framework for 3D human pose estimation using a single 360° camera mounted on the user's wrist. Perceiving a 3D human pose with such a simple setup has remarkable potential for various applications (e. g., daily-living activity monitoring, motion analysis for sports training). However, no existing method has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured, and because of a lack of training data. We propose a method for translating wrist-mounted 360° camera images into 3D human poses. Since we are the first to try this task, we cannot use existing datasets. To address this issue, we use synthetic data to build our own dataset. This solution, however, creates a different problem, that of a domain gap between synthetic data for training and real image data for inference. To resolve this problem, we propose silhouette-based synthetic data generation created for this task. Extensive experiments comparing our method with several baseline methods demonstrated the effectiveness of our silhouette-based pose estimation approach.

In this paper, we propose a framework for 3D human pose estimation with a single 360◦ camera mounted on the user’s wrist. Perceiving a 3D human pose with such a simple setting has remarkable potential for various applications (e.g., daily-living activity monitoring, motion analysis for sports enhancement). However, no existing work has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured and the lack of training data. Therefore, we propose an effective method for translating wrist-mounted 360◦ camera images into 3D human poses. We also propose silhouette-based synthetic data generation dedicated to this task, which enables us to bridge the domain gap between real-world data and synthetic data. We achieved higher estimation accuracy quantitatively and qualitatively compared with other baseline methods.

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

Conventional safe route-finding systems display routes on a map to avoid dangerous areas based on offline data such as statistical data on crime rates. However, such dangerous level changes in a dynamic manner, which can be detected by people walking around. This paper presents a system that shares the detected data measured by edge devices owned by walking people for dynamic safe route finding in real-time. This system evaluates the safety of routes in real time using and displays the route in AR on HMDs and smartphones. This paper reports findings from a simulation and edge device (smartphone) implementation of spatial safety assessment, safe route finding, and route visualization.