Published papers

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.

Extended reality (XR) refers to a technological group namely Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). While the operator cannot view the actual environment in VR, AR can allow him to view the actual surroundings with superimposed virtual items. However, MR is anything in between VR and AR. The Microsoft Hololens headset is a key contributor by focusing on MR technology and it is expensive. Therefore, this research proposes especially for proper investment using VR and AR platforms for complicated motorcycle engine assembly. Consequently, the authors built a fully immersive VR platform using Unreal Engine and HTC Vive. Two AR (marker-less and marker-based) platforms were also developed using Unity 3D and EPSON MOVERIO BT-300. Fourteen students from engineering education participated in the four experiments, which were separated into four groups: (i) video, (ii) VR, (iii) marker-less AR, and (iv) marker-based AR. All participants were then requested to complete all experiments to test their satisfaction levels. In comparison, the outputs showed that the marker-less AR was 34.4% superior, followed by the VR (31.6% superior), and finally the marker-based AR (14.4% superior) to the video-based method. In addition, students prefer the immersive XR (VR and AR) platforms to perform the engine assembly properly.

Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery.

Electrical cables consist of numerous wires, the three-dimensional (3D) shape of which significantly impacts the cables' overall properties, such as bending stiffness. Although X-ray computed tomography (CT) provides a non-destructive method to assess these properties, accurately determining the 3D shape of individual wires from CT images is challenging due to the large number of wires, low image resolution, and indistinguishable appearance of the wires. Previous research lacked quantitative evaluation for wire tracking, and its overall accuracy heavily relied on the accuracy of wire detection. In this study, we present a long short-term memory-based approach for wire tracking that improves robustness against detection errors. The proposed method predicts wire positions in subsequent frames based on previous frames. We evaluate the performance of the proposed method using both actual annotated cables and artificially noised annotations. Our method exhibits greater tracking accuracy and robustness to detection errors compared with the previous method.

Ball tracking based on sports videos captured by high-speed cameras has had limited applicability due to the need for sufficient lighting and the necessity of abundant computational resources to process the large amount of captured data. To overcome these problems, this paper addresses ball tracking using an event camera. An event camera outputs luminance changes in a scene as "events" and is characterized by high temporal resolution, high dynamic range, and data efficiency. The proposed method selects the ball position from the bounding boxes output by the object detector by taking into account the ball motion estimated from the events and the past ball positions using Contrast Maximization (CMax). Considering past ball positions enables us to estimate the ball trajectory consistently over time and detect the ball position more accurately. Additionally, CMax is applied to the events within the tracked ball bounding box, allowing for the estimation of the ball spin. Experimental results in this paper show that the proposed method is quantitatively and qualitatively more accurate than conventional ball tracking methods and demonstrates the potential ability to estimate ball spin.

In this paper, we present an end-to-end unsupervised anomaly detection framework for 3D point clouds. To the best of our knowledge, this is the first work to tackle the anomaly detection task on a general object represented by a 3D point cloud. We propose a deep variational autoencoder based unsupervised anomaly detection network adapted to the 3D point cloud and an anomaly score specifically for 3D point clouds. To verify the effectiveness of the model, we conducted extensive experiments on ShapeNet dataset. Through quantitative and qualitative evaluation, we demonstrate that the proposed method outperforms the baseline method.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

For human-safe robot control in human-to-robot handover, the physical properties of containers and fillings should be accurately estimated. In this paper, we propose a Transformer encoder that shares the same architecture and parameters for filling level and type estimation. We also propose a mask-based geometric algorithm to estimate 3D models of containers for the estimation of their capacity and dimensions. We further use these estimations to estimate their mass in a Convolutional Neural Network model. Experiments show that our Transformer model produced encouraging results in both estimations. While challenges remain in our mask-based algorithm and Convolutional Neural Network model, their results revealed several ways for improvement.

In this paper, we propose a framework for 3D human pose estimation using a single 360° camera mounted on the user's wrist. Perceiving a 3D human pose with such a simple setup has remarkable potential for various applications (e. g., daily-living activity monitoring, motion analysis for sports training). However, no existing method has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured, and because of a lack of training data. We propose a method for translating wrist-mounted 360° camera images into 3D human poses. Since we are the first to try this task, we cannot use existing datasets. To address this issue, we use synthetic data to build our own dataset. This solution, however, creates a different problem, that of a domain gap between synthetic data for training and real image data for inference. To resolve this problem, we propose silhouette-based synthetic data generation created for this task. Extensive experiments comparing our method with several baseline methods demonstrated the effectiveness of our silhouette-based pose estimation approach.

In this paper, we propose a framework for 3D human pose estimation with a single 360◦ camera mounted on the user’s wrist. Perceiving a 3D human pose with such a simple setting has remarkable potential for various applications (e.g., daily-living activity monitoring, motion analysis for sports enhancement). However, no existing work has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured and the lack of training data. Therefore, we propose an effective method for translating wrist-mounted 360◦ camera images into 3D human poses. We also propose silhouette-based synthetic data generation dedicated to this task, which enables us to bridge the domain gap between real-world data and synthetic data. We achieved higher estimation accuracy quantitatively and qualitatively compared with other baseline methods.

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

We propose a lightweight and robust real-time tracking system for tennis balls using an event camera. The method integrates spatial positive/negative polarity correlation with cosine similarity, applies a PCA-based shape filter assuming linear motion, and performs hierarchical, dynamically thresholded search guided by Kalman filter prediction. The algorithm enables robust tracking in real-world environments without substantial computing resources. Evaluation on two real-world sequences captured on a real tennis court (40 rallies) yielded an F1-score of 99.2% and an average throughput over 126 FPS. The system maintains stable performance even in the presence of player motion and background noise, demonstrating practical applicability for interactive training systems.

Conventional safe route-finding systems display routes on a map to avoid dangerous areas based on offline data such as statistical data on crime rates. However, such dangerous level changes in a dynamic manner, which can be detected by people walking around. This paper presents a system that shares the detected data measured by edge devices owned by walking people for dynamic safe route finding in real-time. This system evaluates the safety of routes in real time using and displays the route in AR on HMDs and smartphones. This paper reports findings from a simulation and edge device (smartphone) implementation of spatial safety assessment, safe route finding, and route visualization.

Achieving high-frequency motion estimation with a fast-moving camera is an important task in the field of computer vision. Contrast Maximization (CMax), a method of motion estimation using an event camera, is the de-facto standard. However, CMax requires the processing of a large number of events at a single time, a computationally expensive task. That makes it difficult to perform high-frequency estimates. Specifically, past events that have already been used once for estimation need to be evaluated again. In this paper, we propose 'Recursive Contrast Maximization (R-CMax)' to estimate motions at high frequencies. The proposed method approximates multiple events by two 'compressed events' using estimated trajectories of events from the previous time step, which can be updated recursively. By using a small number of 'compressed events,' motion estimation can be updated efficiently. Comparing R-CMax with CMax and its extensions, we experimentally show that R-CMax can perform motion estimation with a fraction of the computational complexity while maintaining comparable accuracy.

Objective: Early detection and intervention are essential for the mitigation of degenerative cervical myelopathy (DCM). However, although several screening methods exist, they are difficult to understand for community-dwelling people, and the equipment required to set up the test environment is expensive. This study investigated the viability of a DCM-screening method based on the 10-second grip-and-release test using a machine learning algorithm and a smartphone equipped with a camera to facilitate a simple screening system. Methods: Twenty-two participants comprising a group of DCM patients and 17 comprising a control group participated in this study. A spine surgeon diagnosed the presence of DCM. Patients performing the 10-second grip-and-release test were filmed, and the videos were analyzed. The probability of the presence of DCM was estimated using a support vector machine algorithm, and sensitivity, specificity, and area under the curve (AUC) were calculated. Two assessments of the correlation between estimated scores were conducted. The first used a random forest regression model and the Japanese Orthopaedic Association scores for cervical myelopathy (C-JOA). The second assessment used a different model, random forest regression, and the Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire. Results: The final classification model had a sensitivity of 90.9%, specificity of 88.2%, and AUC of 0.93. The correlations between each estimated score and the C-JOA and DASH scores were 0.79 and 0.67, respectively. Conclusions: The proposed model could be a helpful screening tool for DCM as it showed excellent performance and high usability for community-dwelling people and non-spine surgeons.

3D change detection plays a crucial role in a wide range of applications, including disaster management, as well as in robotics for search, rescue, security, and surveillance purposes. Although previous works exist, most of them are limited to detecting a few specific targets or are restricted to 2D images. Additionally, some assume prior knowledge of the object positions of interest. This paper presents a novel change detection algorithm that combines panoptic segmentation and K-NN, enabling the detection of changes without relying on positional information about the objects of interest. Experimental evaluations on indoor point clouds demonstrate the algorithm’s capability to detect the removal of densely and closely placed objects, an aspect overlooked by previous approaches due to their inherent limitations. Despite variations in settings and datasets, our algorithm achieves a recall improvement of 0.06 for the removed class, surpassing the performance of existing related works.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Drowsiness affects the drivers' sensory, cognitive, and psychomotor abilities, which are necessary for safe driving. Drowsiness detection is a critical technique to avoid traffic accidents. Federated learning (FL) can solve the problem of insufficient driver facial data by utilizing different industrial entities' data. However, in the FL system, the privacy information of the drivers might be leaked. In addition, reducing the communication costs and maintaining the model performance are also challenges in industrial scenarios. In this work, we propose a federated transfer learning method with the privacy-preserving protocol for driver drowsiness detection, named PFTL-DDD. We use fine-tuning transfer learning on the initial model of the drowsiness detection FL system. Furthermore, a CKKS-based privacy-preserving protocol is applied to preserve the drivers' privacy data by encrypting the exchanged parameters. The experimental results show that the PFTL-DDD method is superior in terms of accuracy and efficiency compared to the conventional federated learning on the NTHU-DDD and YAWDD datasets. The theoretical analysis demonstrates that the proposed transfer learning method can reduce the communication cost of the system, and the CKKS-based security protocol can protect personal privacy.