Published papers

Achieving high-frequency motion estimation with a fast-moving camera is an important task in the field of computer vision. Contrast Maximization (CMax), a method of motion estimation using an event camera, is the de-facto standard. However, CMax requires the processing of a large number of events at a single time, a computationally expensive task. That makes it difficult to perform high-frequency estimates. Specifically, past events that have already been used once for estimation need to be evaluated again. In this paper, we propose 'Recursive Contrast Maximization (R-CMax)' to estimate motions at high frequencies. The proposed method approximates multiple events by two 'compressed events' using estimated trajectories of events from the previous time step, which can be updated recursively. By using a small number of 'compressed events,' motion estimation can be updated efficiently. Comparing R-CMax with CMax and its extensions, we experimentally show that R-CMax can perform motion estimation with a fraction of the computational complexity while maintaining comparable accuracy.

Objective: Early detection and intervention are essential for the mitigation of degenerative cervical myelopathy (DCM). However, although several screening methods exist, they are difficult to understand for community-dwelling people, and the equipment required to set up the test environment is expensive. This study investigated the viability of a DCM-screening method based on the 10-second grip-and-release test using a machine learning algorithm and a smartphone equipped with a camera to facilitate a simple screening system. Methods: Twenty-two participants comprising a group of DCM patients and 17 comprising a control group participated in this study. A spine surgeon diagnosed the presence of DCM. Patients performing the 10-second grip-and-release test were filmed, and the videos were analyzed. The probability of the presence of DCM was estimated using a support vector machine algorithm, and sensitivity, specificity, and area under the curve (AUC) were calculated. Two assessments of the correlation between estimated scores were conducted. The first used a random forest regression model and the Japanese Orthopaedic Association scores for cervical myelopathy (C-JOA). The second assessment used a different model, random forest regression, and the Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire. Results: The final classification model had a sensitivity of 90.9%, specificity of 88.2%, and AUC of 0.93. The correlations between each estimated score and the C-JOA and DASH scores were 0.79 and 0.67, respectively. Conclusions: The proposed model could be a helpful screening tool for DCM as it showed excellent performance and high usability for community-dwelling people and non-spine surgeons.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Data analysis in net sports, such as badminton, is becoming increasingly important. This research aims to analyze data so that players can gain an advantage in the fast rally development of badminton matches. We investigate the novel task of predicting future shuttle trajectories in badminton match videos and propose a method that uses shuttle and player position information. In an experiment, we detected players from match videos and trained a time-sequence model. The proposed method outperformed baseline methods that use only the shuttle position information as the input and other methods that use time-sequence models.

Drowsiness affects the drivers' sensory, cognitive, and psychomotor abilities, which are necessary for safe driving. Drowsiness detection is a critical technique to avoid traffic accidents. Federated learning (FL) can solve the problem of insufficient driver facial data by utilizing different industrial entities' data. However, in the FL system, the privacy information of the drivers might be leaked. In addition, reducing the communication costs and maintaining the model performance are also challenges in industrial scenarios. In this work, we propose a federated transfer learning method with the privacy-preserving protocol for driver drowsiness detection, named PFTL-DDD. We use fine-tuning transfer learning on the initial model of the drowsiness detection FL system. Furthermore, a CKKS-based privacy-preserving protocol is applied to preserve the drivers' privacy data by encrypting the exchanged parameters. The experimental results show that the PFTL-DDD method is superior in terms of accuracy and efficiency compared to the conventional federated learning on the NTHU-DDD and YAWDD datasets. The theoretical analysis demonstrates that the proposed transfer learning method can reduce the communication cost of the system, and the CKKS-based security protocol can protect personal privacy.

In many sports events, replay footage is commonly used to enhance audience engagement. However, the process of selecting and editing these replays is typically manual, placing a significant operational burden on on-site staff. While recent advancements in computer vision have facilitated sports match analysis, most existing methods lack real-time capabilities, rendering them unsuitable for automatic replay generation. This paper proposes a real-time ball tracking and action classification method using an event camera for volleyball matches, addressing the gap in automatic replay generation. By leveraging the unique advantages of event cameras, our method avoids machine learning techniques, instead utilizing event-based data streams for lightweight computation in ball tracking and action classification. When ball identification fails, temporal and spatial patterns of key events in volleyball are used for localization. The tracked ball trajectory and classified actions are then employed to identify key moments for replay generation. Experimental results demonstrate the effectiveness of the proposed method, achieving real-time operation and successfully identifying moments suitable for replay generation.

We propose a lightweight and robust real-time tracking system for tennis balls using an event camera. The method integrates spatial positive/negative polarity correlation with cosine similarity, applies a PCA-based shape filter assuming linear motion, and performs hierarchical, dynamically thresholded search guided by Kalman filter prediction. The algorithm enables robust tracking in real-world environments without substantial computing resources. Evaluation on two real-world sequences captured on a real tennis court (40 rallies) yielded an F1-score of 99.2% and an average throughput over 126 FPS. The system maintains stable performance even in the presence of player motion and background noise, demonstrating practical applicability for interactive training systems.

Difference-in-level detection in outdoor scenes has various possible applications, including walking assistance for blind people, robot walking assistance, and mapping the hazards of factory premises. It is difficult to detect all outdoor differences in level, such as RGB or RGB-D images, not only including road curbs, which are often targeted for detection in automated driving, but also differences in level on factory premises and sidewalks, because the pattern of outdoor differences in level is abundant and complex. This paper proposes a novel method for detecting differences in level from RGB-D images with segmented ground masks. First, image patches of differences in level were extracted from outdoor images to create the dataset. The change in the normal vector of the contour part on the detected plane is used to generate image patches of the difference in level, but this method strongly depends on the accuracy of planar detection, and it detects only some differences in level. Then, we created the dataset, consisting of image patches and including the extracted differences in level. The dataset is used for training a deep learning model for detecting differences in level in outdoor images without limitations. In addition, because the purpose of this paper is to detect differences in level in outdoor walking areas, regions in the image other than the target areas were excluded by the segmented ground mask. For the performance evaluation, we implemented our algorithm using a modern smartphone with a high-performance depth camera. To evaluate the effectiveness of the proposed method, the results from various inputs, such as RGB, depth, grayscale, normal, and combinations of them, were qualitatively and quantitatively evaluated, and Blender was used to generate synthetic test images for a quantitative evaluation of the difference in level. We confirm that the suggested method successfully detects various types of differences in level in outdoor images, even in complex scenes.

This study focuses on reconstructing game states in American football by generating minimap representations of plays from video footage. Our approach follows a five-step pipeline: detecting and tracking players and referees, identifying players via jersey numbers, transforming camera perspectives, stitching multiview images to create a panoramic image, and generating a minimap. Obtaining accurate player trajectories requires a sideline perspective wide view, providing a comprehensive field overview. Conversely, jersey number recognition demands an end-zone perspective tight view, where players face each other before playing. To integrate these perspectives, we apply image stitching to merge multiple camera views into a seamless scene. Camera transformation techniques ensure accurate spatial alignment across perspectives. Our pipeline achieved 59.9 mean localization error with 45.6% accuracy for team classification for players and referees. This work introduces an efficient pipeline for automated minimap generation, enabling effective game state reconstruction from video footage.

3D change detection plays a crucial role in a wide range of applications, including disaster management, as well as in robotics for search, rescue, security, and surveillance purposes. Although previous works exist, most of them are limited to detecting a few specific targets or are restricted to 2D images. Additionally, some assume prior knowledge of the object positions of interest. This paper presents a novel change detection algorithm that combines panoptic segmentation and K-NN, enabling the detection of changes without relying on positional information about the objects of interest. Experimental evaluations on indoor point clouds demonstrate the algorithm’s capability to detect the removal of densely and closely placed objects, an aspect overlooked by previous approaches due to their inherent limitations. Despite variations in settings and datasets, our algorithm achieves a recall improvement of 0.06 for the removed class, surpassing the performance of existing related works.

Human Activity Recognition (HAR) technology is pivotal in medical and surveillance systems. Traditional HAR methods using a single sensor are limited in the types of actions they can classify due to the inherent constraints of the sensor. This paper addresses multimodal HAR, which enables the classification of a broader range of activities by utilizing both a camera and an IMU, thereby compensating for each sensor's shortcomings. The proposed method employs images, acceleration, and angular velocity as inputs to a network that integrates an attention mechanism. This mechanism dynamically adjusts the importance of each sensor feature based on the input data. Furthermore, the proposed network is lightweight and high-performance, designed to operate on edge devices with limited computational resources. Experimental results demonstrate that our network can classify five actions with approximately 97% precision, significantly outperforming single-sensor baselines. For a robust evaluation, we conducted experiments using the UESTC-MMEA-CL dataset, confirming that our multimodal approach is more accurate than single-sensor methods.

We present a novel framework of motion tracking from event data using implicit expression. Our framework uses pre-trained event generation MLP called the implicit event generator (IEG) and carries out motion tracking by updating its state (position and velocity) based on the difference between the observed event and generated event from the current state estimation. The difference is computed implicitly by the IEG. Unlike the conventional explicit approach, which requires dense computation to evaluate the difference, our implicit approach realizes the update of the efficient state directly from sparse event data. Our sparse algorithm is especially suitable for mobile robotics applications in which computational resources and battery life are limited. To verify the effectiveness of our method on real-world data, we applied it to the AR marker tracking application. We have confirmed that our framework works well in real-world environments in the presence of noise and background clutter.

Recording surgery in operating rooms is one of the essential tasks for education and evaluation of medical treatment. However, recording the fields which depict the surgery is difficult because the targets are heavily occluded during surgery by the heads or hands of doctors or nurses. We use a recording system which multiple cameras embedded in the surgical lamp, assuming that at least one camera is recording the target without occlusion. In this paper, we propose Conditional-BARF (C-BARF) to generate occlusion-free images by synthesizing novel view images from the camera, aiming to generate videos with smooth camera pose transitions. To the best of our knowledge, this is the first work to tackle the problem of synthesizing a novel view image from multiple images for the surgery scene. We conduct experiments using an original dataset of three different types of surgeries. Our experiments show that we can successfully synthesize novel views from the images recorded by the multiple cameras embedded in the surgical lamp.

This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less objects from a monocular image by using edge information. The deep learning-based pose estimation method needs a large dataset containing pairs of an image and ground truth pose of objects. To alleviate the cost of collecting a dataset, we focus on the method using a dataset made by computer graphics (CG). This simulation-based method prepares a thousand images by rendering the computer-aided design (CAD) data of the object and trains a deep-learning model. As an inference stage, a monocular RGB image is entered into the model, and the object’s pose is estimated. The representative simulation-based method, Pose Interpreter Networks, uses silhouette images as the input, thereby enabling common feature (contour) extraction from RGB and CG images. However, estimating rotation parameters is less accurate. To overcome this problem, we propose a method to use edge information extracted from the object’s ridgelines for training the deep learning model. Since edge distribution changes largely according to the pose, the estimation of rotation parameters becomes more robust. Through an experiment with simulation data, we quantitatively proved the accuracy improvement compared to the previous method (error rate decreases at a certain condition are translation 22.9% and rotation: 43.4%). Moreover, through an experiment with physical data, we clarified the issues of this method and proposed an effective solution by fine-tuning (error rate decrease at a certain condition are translation 20.1% and rotation 57.7%).

As data analysis becomes increasingly significant, streamlining data collection is essential, particularly in sports. Among various aspects, player performance analysis is crucial for both professional and amateur levels. In computer vision, Multi-Object Tracking (MOT) and Multi-Camera Multi-Object Tracking (MCMOT) enable the analysis of people’s trajectories captured in videos. However, many existing methods do not account for crowded scenarios, such as sports matches, and require complex preprocessing of datasets, making them impractical for real-world applications. To address these challenges, we introduce a simple yet novel extension of MOT methods by utilizing the Gaussian distribution to associate tracking information obtained from multiple cameras. By integrating tracking data from all cameras, our methodology effectively mitigates identity switches and expands the tracking range. Experimental results demonstrate our tracking system’s effectiveness, showing a reduction in ID switches and false negatives compared to results achieved with a standard multi-object tracker.

The process of video recording open surgical procedures is a critical aspect of medical practice, as it enables physicians to retrospectively analyze and assess the surgery. The synthesis of a novel view of surgical procedures from recorded footage is a vital task, facilitating the review of these procedures from multiple angles. Although employing neural radiance fields (NeRF) may seem a straightforward approach, the application of standard NeRF models to medical environments often results in suboptimal rendering quality. This issue predominantly arises due to the significant occlusions present in open surgical videos and the constraints imposed by the limited number of cameras available, further exacerbated by the unique environmental restrictions of the operating room. Through the adoption of a learning framework for novel view rendering, derived from both natural and open surgical scenes using a generalizable NeRF representation, our methodology offers the capacity to render new perspectives from a constrained set of source images. Furthermore, we implement an automatic occlusion mask estimation to exclude obscured pixels during the training phase. The efficacy of our proposed approach has been validated through comparative analyses with other robust training configurations, utilizing real-world video data recorded at Keio University School of Medicine. Additionally, our findings demonstrate the practical effectiveness of the proposed occlusion mask.

Monocular depth estimation is a challenging task in computer vision. Although many approaches using Convolutional neural networks (CNNs) have been proposed, most of them are trained on large-scale datasets mainly composed of gravity-aligned images. Therefore, conventional approaches fail to predict reliable depth for tilted images containing large pitch and roll camera rotations. To tackle this problem, we propose a novel refining method based on the distribution of gravity directions in the training sets. We designed a gravity rectifier that is learned to transform the gravity direction of a tilted image into a rectified one that matches the gravity-aligned training data distribution. For the evaluation, we employed public datasets and also created our own dataset composed of large pitch and roll camera movements. Our experiments showed that our approach successfully rectified the camera rotation and outperformed our baselines, which achieved 29% improvement in abs rel over the vanilla model. Additionally, our method had competitive accuracy comparable to state-of-the-art monocular depth prediction approaches considering camera rotation.

Multi-camera multi-person (MCMP) tracking and re-identification (ReID) are essential tasks in safety, pedestrian analysis, and so on; however, most research focuses on outdoor scenarios because they are much more complicated to deal with occlusions and misidentification in a crowded room with obstacles. Moreover, it is challenging to complete the two tasks in one framework. We present a trajectory-based method, integrating tracking and ReID tasks. First, the poses of all surgical members captured by each camera are detected frame-by-frame; then, the detected poses are exploited to track the trajectories of all members for each camera; finally, these trajectories of different cameras are clustered to re-identify the members in the operating room across all cameras. Compared to other MCMP tracking and ReID methods, the proposed one mainly exploits trajectories, taking texture features that are less distinguishable in the operating room scenario as auxiliary cues. We also integrate temporal information during ReID, which is more reliable than the state-of-the-art framework where ReID is conducted frame-by-frame. In addition, our framework requires no training before deployment in new scenarios. We also created an annotated MCMP dataset with actual operating room videos. Our experiments prove the effectiveness of the proposed trajectory-based ReID algorithm. The proposed framework achieves 85.44% accuracy in the ReID task, outperforming the state-of-the-art framework in our operating room dataset.