Published papers

In this paper, we present an end-to-end unsupervised anomaly detection framework for 3D point clouds. To the best of our knowledge, this is the first work to tackle the anomaly detection task on a general object represented by a 3D point cloud. We propose a deep variational autoencoder based unsupervised anomaly detection network adapted to the 3D point cloud and an anomaly score specifically for 3D point clouds. To verify the effectiveness of the model, we conducted extensive experiments on ShapeNet dataset. Through quantitative and qualitative evaluation, we demonstrate that the proposed method outperforms the baseline method.

In recent years, much attention has been paid to the prediction of pedestrian trajectories, as they are one of the key factors for a better society, such as automatic driving, a guide for blind people, and social robots interacting with humans. To tackle this task, many methods have been proposed but few are from the first-person perspective because of the lack of a publicly available dataset. Therefore, we propose a method that uses egocentric vision, which does not need to be trained with a first-person video dataset. We made it possible to utilize existing methods, which predict from a bird’s-eye view. In addition, we propose a novel way to consider semantic information without changing the shape of the input to apply to all existing bird’s-eye methods that use only past trajectories. Therefore, there is no need to create a new dataset from egocentric vision. The experimental results demonstrate that the proposed method makes it possible to predict from an egocentric view via existing methods of bird’s-eye view. The proposed method qualitatively improves trajectory predictions without aggravating quantitative accracy, and the effectiveness of predicting the trajectories of multiple people simultaneously.

In this paper, we propose a framework for 3D human pose estimation using a single 360° camera mounted on the user's wrist. Perceiving a 3D human pose with such a simple setup has remarkable potential for various applications (e. g., daily-living activity monitoring, motion analysis for sports training). However, no existing method has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured, and because of a lack of training data. We propose a method for translating wrist-mounted 360° camera images into 3D human poses. Since we are the first to try this task, we cannot use existing datasets. To address this issue, we use synthetic data to build our own dataset. This solution, however, creates a different problem, that of a domain gap between synthetic data for training and real image data for inference. To resolve this problem, we propose silhouette-based synthetic data generation created for this task. Extensive experiments comparing our method with several baseline methods demonstrated the effectiveness of our silhouette-based pose estimation approach.

In this paper, we propose a framework for 3D human pose estimation with a single 360◦ camera mounted on the user’s wrist. Perceiving a 3D human pose with such a simple setting has remarkable potential for various applications (e.g., daily-living activity monitoring, motion analysis for sports enhancement). However, no existing work has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured and the lack of training data. Therefore, we propose an effective method for translating wrist-mounted 360◦ camera images into 3D human poses. We also propose silhouette-based synthetic data generation dedicated to this task, which enables us to bridge the domain gap between real-world data and synthetic data. We achieved higher estimation accuracy quantitatively and qualitatively compared with other baseline methods.

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

Extended reality (XR) refers to a technological group namely Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). While the operator cannot view the actual environment in VR, AR can allow him to view the actual surroundings with superimposed virtual items. However, MR is anything in between VR and AR. The Microsoft Hololens headset is a key contributor by focusing on MR technology and it is expensive. Therefore, this research proposes especially for proper investment using VR and AR platforms for complicated motorcycle engine assembly. Consequently, the authors built a fully immersive VR platform using Unreal Engine and HTC Vive. Two AR (marker-less and marker-based) platforms were also developed using Unity 3D and EPSON MOVERIO BT-300. Fourteen students from engineering education participated in the four experiments, which were separated into four groups: (i) video, (ii) VR, (iii) marker-less AR, and (iv) marker-based AR. All participants were then requested to complete all experiments to test their satisfaction levels. In comparison, the outputs showed that the marker-less AR was 34.4% superior, followed by the VR (31.6% superior), and finally the marker-based AR (14.4% superior) to the video-based method. In addition, students prefer the immersive XR (VR and AR) platforms to perform the engine assembly properly.

Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery.

Objective: Early detection and intervention are essential for the mitigation of degenerative cervical myelopathy (DCM). However, although several screening methods exist, they are difficult to understand for community-dwelling people, and the equipment required to set up the test environment is expensive. This study investigated the viability of a DCM-screening method based on the 10-second grip-and-release test using a machine learning algorithm and a smartphone equipped with a camera to facilitate a simple screening system. Methods: Twenty-two participants comprising a group of DCM patients and 17 comprising a control group participated in this study. A spine surgeon diagnosed the presence of DCM. Patients performing the 10-second grip-and-release test were filmed, and the videos were analyzed. The probability of the presence of DCM was estimated using a support vector machine algorithm, and sensitivity, specificity, and area under the curve (AUC) were calculated. Two assessments of the correlation between estimated scores were conducted. The first used a random forest regression model and the Japanese Orthopaedic Association scores for cervical myelopathy (C-JOA). The second assessment used a different model, random forest regression, and the Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire. Results: The final classification model had a sensitivity of 90.9%, specificity of 88.2%, and AUC of 0.93. The correlations between each estimated score and the C-JOA and DASH scores were 0.79 and 0.67, respectively. Conclusions: The proposed model could be a helpful screening tool for DCM as it showed excellent performance and high usability for community-dwelling people and non-spine surgeons.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

For human-safe robot control in human-to-robot handover, the physical properties of containers and fillings should be accurately estimated. In this paper, we propose a Transformer encoder that shares the same architecture and parameters for filling level and type estimation. We also propose a mask-based geometric algorithm to estimate 3D models of containers for the estimation of their capacity and dimensions. We further use these estimations to estimate their mass in a Convolutional Neural Network model. Experiments show that our Transformer model produced encouraging results in both estimations. While challenges remain in our mask-based algorithm and Convolutional Neural Network model, their results revealed several ways for improvement.

Conventional safe route-finding systems display routes on a map to avoid dangerous areas based on offline data such as statistical data on crime rates. However, such dangerous level changes in a dynamic manner, which can be detected by people walking around. This paper presents a system that shares the detected data measured by edge devices owned by walking people for dynamic safe route finding in real-time. This system evaluates the safety of routes in real time using and displays the route in AR on HMDs and smartphones. This paper reports findings from a simulation and edge device (smartphone) implementation of spatial safety assessment, safe route finding, and route visualization.

Achieving high-frequency motion estimation with a fast-moving camera is an important task in the field of computer vision. Contrast Maximization (CMax), a method of motion estimation using an event camera, is the de-facto standard. However, CMax requires the processing of a large number of events at a single time, a computationally expensive task. That makes it difficult to perform high-frequency estimates. Specifically, past events that have already been used once for estimation need to be evaluated again. In this paper, we propose 'Recursive Contrast Maximization (R-CMax)' to estimate motions at high frequencies. The proposed method approximates multiple events by two 'compressed events' using estimated trajectories of events from the previous time step, which can be updated recursively. By using a small number of 'compressed events,' motion estimation can be updated efficiently. Comparing R-CMax with CMax and its extensions, we experimentally show that R-CMax can perform motion estimation with a fraction of the computational complexity while maintaining comparable accuracy.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Drowsiness affects the drivers' sensory, cognitive, and psychomotor abilities, which are necessary for safe driving. Drowsiness detection is a critical technique to avoid traffic accidents. Federated learning (FL) can solve the problem of insufficient driver facial data by utilizing different industrial entities' data. However, in the FL system, the privacy information of the drivers might be leaked. In addition, reducing the communication costs and maintaining the model performance are also challenges in industrial scenarios. In this work, we propose a federated transfer learning method with the privacy-preserving protocol for driver drowsiness detection, named PFTL-DDD. We use fine-tuning transfer learning on the initial model of the drowsiness detection FL system. Furthermore, a CKKS-based privacy-preserving protocol is applied to preserve the drivers' privacy data by encrypting the exchanged parameters. The experimental results show that the PFTL-DDD method is superior in terms of accuracy and efficiency compared to the conventional federated learning on the NTHU-DDD and YAWDD datasets. The theoretical analysis demonstrates that the proposed transfer learning method can reduce the communication cost of the system, and the CKKS-based security protocol can protect personal privacy.

In many sports events, replay footage is commonly used to enhance audience engagement. However, the process of selecting and editing these replays is typically manual, placing a significant operational burden on on-site staff. While recent advancements in computer vision have facilitated sports match analysis, most existing methods lack real-time capabilities, rendering them unsuitable for automatic replay generation. This paper proposes a real-time ball tracking and action classification method using an event camera for volleyball matches, addressing the gap in automatic replay generation. By leveraging the unique advantages of event cameras, our method avoids machine learning techniques, instead utilizing event-based data streams for lightweight computation in ball tracking and action classification. When ball identification fails, temporal and spatial patterns of key events in volleyball are used for localization. The tracked ball trajectory and classified actions are then employed to identify key moments for replay generation. Experimental results demonstrate the effectiveness of the proposed method, achieving real-time operation and successfully identifying moments suitable for replay generation.

We propose a lightweight and robust real-time tracking system for tennis balls using an event camera. The method integrates spatial positive/negative polarity correlation with cosine similarity, applies a PCA-based shape filter assuming linear motion, and performs hierarchical, dynamically thresholded search guided by Kalman filter prediction. The algorithm enables robust tracking in real-world environments without substantial computing resources. Evaluation on two real-world sequences captured on a real tennis court (40 rallies) yielded an F1-score of 99.2% and an average throughput over 126 FPS. The system maintains stable performance even in the presence of player motion and background noise, demonstrating practical applicability for interactive training systems.

Difference-in-level detection in outdoor scenes has various possible applications, including walking assistance for blind people, robot walking assistance, and mapping the hazards of factory premises. It is difficult to detect all outdoor differences in level, such as RGB or RGB-D images, not only including road curbs, which are often targeted for detection in automated driving, but also differences in level on factory premises and sidewalks, because the pattern of outdoor differences in level is abundant and complex. This paper proposes a novel method for detecting differences in level from RGB-D images with segmented ground masks. First, image patches of differences in level were extracted from outdoor images to create the dataset. The change in the normal vector of the contour part on the detected plane is used to generate image patches of the difference in level, but this method strongly depends on the accuracy of planar detection, and it detects only some differences in level. Then, we created the dataset, consisting of image patches and including the extracted differences in level. The dataset is used for training a deep learning model for detecting differences in level in outdoor images without limitations. In addition, because the purpose of this paper is to detect differences in level in outdoor walking areas, regions in the image other than the target areas were excluded by the segmented ground mask. For the performance evaluation, we implemented our algorithm using a modern smartphone with a high-performance depth camera. To evaluate the effectiveness of the proposed method, the results from various inputs, such as RGB, depth, grayscale, normal, and combinations of them, were qualitatively and quantitatively evaluated, and Blender was used to generate synthetic test images for a quantitative evaluation of the difference in level. We confirm that the suggested method successfully detects various types of differences in level in outdoor images, even in complex scenes.

This study focuses on reconstructing game states in American football by generating minimap representations of plays from video footage. Our approach follows a five-step pipeline: detecting and tracking players and referees, identifying players via jersey numbers, transforming camera perspectives, stitching multiview images to create a panoramic image, and generating a minimap. Obtaining accurate player trajectories requires a sideline perspective wide view, providing a comprehensive field overview. Conversely, jersey number recognition demands an end-zone perspective tight view, where players face each other before playing. To integrate these perspectives, we apply image stitching to merge multiple camera views into a seamless scene. Camera transformation techniques ensure accurate spatial alignment across perspectives. Our pipeline achieved 59.9 mean localization error with 45.6% accuracy for team classification for players and referees. This work introduces an efficient pipeline for automated minimap generation, enabling effective game state reconstruction from video footage.