Published papers

Ball detection in sports, particularly in fast-paced games like volleyball, where the ball is constantly in high motion, presents a significant challenge for game analysis and automated sports broadcasting. Conventional camera-based ball detection faces issues, such as motion blur, in high-speed ball movement scenes. To address these challenges, we propose a deep learning-based method for detecting balls using event cameras. Event cameras, also known as dynamic vision sensors, operate differently from traditional cameras. Instead of capturing frames at fixed intervals, they record individual pixel-level luminance changes, referred to as events. This unique feature enables event cameras to provide precise temporal information with low latency. Our proposed method transforms sparse events into an image format, enabling the use of current deep-learning architectures for object detection. Given the limited amount of events available for training an object detector, we generate synthetic events from RGB frames. This approach reduces the need for extensive annotation and ensures sufficient data availability. Experimental results confirm that our proposed method can detect balls that are undetectable in RGB frames and outperform existing methods that utilize event-based ball detection. Moreover, we conducted tests to verify our method's ability to detect balls in real events, not just synthetic ones. These results demonstrate that our proposed method opens up new possibilities in sports ball detection.

Training and education have become increasingly crucial in obtaining new skills in a variety of fields, especially in assembly and disassembly operations. The main issue in mechanical engineering, particularly in the assembly department, was that automobile engine components assembly was found to be complicated and challenging to assemble using an existing method, where they only rely on a video-based method. The purpose of this paper is to create interactive Virtual Reality (VR) and Augmented Reality (AR) applications that allow users to efficiently assist and complete the assembly tasks. In this work, the authors designed and developed a fully immersive VR application using an HTC Vive headset and two AR applications (marker-less AR application and marker-based AR application) using EPSON MOVERIO BT-300 (AR Smart Glasses). Fourteen engineering students from Universiti Putra Malaysia were selected for the experiment. They were divided into four groups: video-based group, VR-based group, marker-less AR group, and marker-based AR group. They are required to complete all four experiments (video-based experiment, VR-based experiment, marker-less AR experiment, and marker-based AR experiment). The results showed that the marker-less AR application is the best impressive method (37% better), the VR application is the second impressive method (23% better) followed by the marker-based AR application is the third impressive method (3% better) compared to the existing video-based guideline. Therefore, the students favored AR and VR applications rather than the existing method to be used in automobile engine assembly tasks.

Only virtual hands and feet move synchronously with an observer's movement, inducing body ownership of an invisible body between them. However, it is unclear whether body ownership is also induced when the size of the invisible body is changed. In this study, we investigated whether body ownership is induced in large or small invisible bodies and whether size perception changes with the size of the invisible body. The results showed that body ownership was induced even if the size of the invisible body was changed, but the size perception did not change.

Egocentric open-surgery videos capture rich, fine-grained details essential for accurately modelling surgical procedures and human behaviour in the operating room. A detailed, pixel-level understanding of hands and surgical tools is crucial for interpreting a surgeon action and intention. We introduce EgoSurgery-HTS, a new dataset with pixel-wise annotations and a benchmark suite for segmenting surgical tools, hands, and interacting tools in egocentric open-surgery videos. Specifically, we provide a labelled dataset for (1) tool instance segmentation of 14 distinct surgical tools, (2) hand instance segmentation, and (3) hand–tool segmentation to label main operating hands and the tools they manipulate. Using EgoSurgery-HTS, we conduct extensive evaluations of state-of-the-art segmentation methods and demonstrate significant improvements in the accuracy of hand and hand–tool segmentation in egocentric open-surgery videos compared to existing datasets. The dataset will be released upon acceptance.

High-speed phenomena, which are important in fields such as sports and fluid dynamics, are often challenging for the human visual system to perceive precisely. To better understand these high-speed phenomena, it is important to present users the information intuitively and precisely without time delays. Existing visualization techniques, such as high-speed cameras and stroboscopes, are effective for capturing high-speed events but are not well-suited for interactive visualization purposes.In this study, we propose an interactive visualization system of high-speed events using an event camera. The system precisely captures high-speed phenomena and allows users to replay important scenes instantly with a single key press. Through a user study involving 38 participants, we confirmed that the event camera system enhances human visual perception. Furthermore, a case study demonstrated that the interactive design enables users to interpret high-speed phenomena more intuitively. These findings suggest that our system can effectively augment users' visual perception.

We propose a method for dense depth estimation from an event stream generated when sweeping the focal plane of the driving lens attached to an event camera. In this method, a depth map is inferred from an 'event focal stack' composed of the event stream using a convolutional neural network trained with synthesized event focal stacks. The synthesized event stream is created from a focal stack generated by Blender for any arbitrary 3D scene. This allows for training on scenes with diverse structures. Additionally, we explored methods to eliminate the domain gap between real event streams and synthetic event streams. Our method demonstrates superior performance over a depth-from-defocus method in the image domain on synthetic and real datasets.

Most robots implicitly assume that the road surface on which they move is flat, without differences in level. Detecting differences in level on roads contributes to robots moving safely without stacking and falling. Although there are some studies on detecting differences in level in RGB or RGB-D images, directly finding only differences in level on roads is difficult due to the abundance and complexity of the types of differences in level on roads. This paper presents a new method for detecting differences in level from RGB-D images obtained by a modern smartphone equipped with a high-performance depth camera. First, we extract a part of differences in level on roads by finding the change of the normal vector in the contour of the detected plane. Then, a deep learning model trained on the dataset created by using the extracted image patches is used for detecting all the differences in level in outdoor images. To evaluate the effectiveness of the proposed method, quantitative and qualitative comparisons with existing methods were conducted. Further, the results from various inputs were qualitatively and quantitatively evaluated. As a result, we verified that the proposed method was able to detect all differences in level in an image, even in complex scenes where existing methods cannot detect.

Although recent deep learning-based inpainting techniques achieve excellent restoration quality, their stringent requirement for computational resources such as GPU/VRAM render them difficult to use in settings where high-end hardware is not available. We address this gap between research and real-world applications in our submission to the DREAMING challenge, which explores such a use case - diminished reality - where run time and GPU hardware are limited. Specifically, this paper proposes a method that divides a video optimally into sections of variable lengths for the downstream inpainting model. By maximizing the number of frames, i.e., spatio-temporal information, in each subvideo, inference using a SOTA model can often be performed without significant degradation to output quality. In addition, to expedite inference, we examine techniques that intelligently reduce the amount of input pixels, e.g., downsampling and cropping, while maintaining decent inpainting quality.

Event cameras, known for their high dynamic range, absence of motion blur, and low energy usage, have recently found a wide range of applications thanks to these attributes. In the past few years, the field of event-based 3D reconstruction saw remarkable progress, with the Neural Radiance Field (NeRF) based approach demonstrating photorealistic view synthesis results. However, the volume rendering paradigm of NeRF necessitates extensive training and rendering times. In this paper, we introduce Event Enhanced Gaussian Splatting (E2GS), a novel method that incorporates event data into Gaussian Splatting, which has recently made significant advances in the field of novel view synthesis. Our E2GS effectively utilizes both blurry images and event data, significantly improving image deblurring and producing high-quality novel view synthesis. Our comprehensive experiments on both synthetic and real-world datasets demonstrate our E2GS can generate visually appealing renderings while offering faster training and rendering speed (140 FPS). Our code is available at https://github.com/deguchihiroyuki/E2GS.

A recent singleshot multiplane image (MPI) generation enables to copy an observed reality within a camera frame into other reality domains via view synthesis. While the scene scale is unknown due to the nature of singleshot MPI processing, camera tracking algorithms can estimate depth within the application world coordinate system. Given such depth information, we propose to adjust the scale of singleshot MPI to that of the currently observed scene. We find the individual scales of the MPI layers by minimizing the differences between the depth of MPI rendering and that of camera tracking. We eventually found that many layers fall within a close depth. Therefore, we merge such layers into one to compact the MPI representation. We compared our method with baselines using real and synthetic datasets with dense and sparse depth inputs. Our results demonstrate that our algorithm achieves higher scores in image metrics and reduces MPI data amount by up to 78%.

In equestrian competitions, the precision of horse movements significantly impacts scoring. Accurate recognition and analysis of these movements can enhance the scoring process. Existing action recognition methods for sports are primarily designed for humans, presenting challenges when applied to equestrian sports where horses are the main subjects. Additionally, research on animal behavior has been largely confined to behavior recognition. This paper evaluates and compares the advantages and limitations of different modalities by constructing a dataset of three distinct modalities specific to equestrian sports and applying existing human action recognition techniques. Our findings indicate that optical flow is the most practical modality. Furthermore, combining the three modalities achieved a motion recognition accuracy of 0.97. By removing error-prone data—such as instances where the horse is facing forward or backward, or when obstacles are present around the horse’s limbs—the action recognition accuracy increased to 0.99. The dataset is available at https://github.com/kotokamatsunaga/horsedataset.git.

A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However, the past crowd density maps are often incomplete due to the miss-detection of pedestrians, and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion frame-work for crowd density forecasting (CrowdMAC), which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e., forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e., imputing past maps with miss-detection). Additionally, we propose Temporal-Density-aware Masking (TDM), which non-uniformly masks tokens in the observed crowd density map, considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover, we introduce multitask masking to enhance training efficiency. In the experiments, CrowdMAC achieves state-of-the-art performance on seven large-scale datasets, including SDD, ETH-UCY, inD, JRDB, VSCrowd, FDST, and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at project page.

There has been a substantial amount of research involving computer methods and technology for the detection and recognition of diabetic foot ulcers (DFUs), but there is a lack of systematic comparisons of state-of-the-art deep learning object detection frameworks applied to this problem. DFUC2020 provided participants with a comprehensive dataset consisting of 2,000 images for training and 2,000 images for testing. This paper summarizes the results of DFUC2020 by comparing the deep learning-based algorithms proposed by the winning teams: Faster R–CNN, three variants of Faster R–CNN and an ensemble method; YOLOv3; YOLOv5; EfficientDet; and a new Cascade Attention Network. For each deep learning method, we provide a detailed description of model architecture, parameter settings for training and additional stages including pre-processing, data augmentation and post-processing. We provide a comprehensive evaluation for each method. All the methods required a data augmentation stage to increase the number of images available for training and a post-processing stage to remove false positives. The best performance was obtained from Deformable Convolution, a variant of Faster R–CNN, with a mean average precision (mAP) of 0.6940 and an F1-Score of 0.7434. Finally, we demonstrate that the ensemble method based on different deep learning methods can enhance the F1-Score but not the mAP.

Recording surgery is an important technique for education and the evaluation of medical treatments. However, capturing targets such as the surgical field, surgical tools, and the surgeon's hands, is almost impossible since these targets are heavily occluded by the surgeon's head and body during a surgery. We used a recording system in which multiple cameras are installed in the surgical lump, supposing at least one camera would capture the target without occlusion. As this system records multiple video sequences, we address the task to select a best view camera automatically. Recently, learning-based approaches in a fully supervised manner have been proposed for this task, but these previous approaches completely rely on manual annotation of the training data. In this paper, we focus on the eye tracker mounted on the surgeon's head, which can capture the recording targets without occlusion. Employing this first-person-view video synchronized with multiple videos of the surgical lump, we propose a novel camera selection approach using a self-supervised learning framework. In experiments, we created a dataset composed of four different breast surgery. Our extended experiments showed that our approach successfully switched to the best camera view without manual annotation and achieved competitive accuracy compared with conventional supervised methods. Also, our approach yielded effective visual representations comparable to state-of-the-art self-supervised learning frameworks.

Study Design.Cross-sectional study.Objective.To develop a binary classification model for cervical myelopathy (CM) screening based on a machine learning algorithm using Leap Motion (Leap Motion, San Francisco, CA), a novel noncontact sensor device.Summary of Background Data.Progress of CM symptoms are gradual and cannot be easily identified by the patients themselves. Therefore, screening methods should be developed for patients of CM before deterioration of myelopathy. Although some studies have been conducted to objectively evaluate hand movements specific to myelopathy using cameras or wearable sensors, their methods are unsuitable for simple screening outside hospitals because of the difficulty in obtaining and installing their equipment and the long examination time.Methods.In total, 50 and 28 participants in the CM and control groups were recruited, respectively. The diagnosis of CM was made by spine surgeons. We developed a desktop system using Leap Motion that recorded 35 parameters of fingertip movements while participants gripped and released their fingers as rapidly as possible. A support vector machine was used to develop the binary classification model, and a multiple linear regression analysis was performed to create regression models to estimate the total Japanese Orthopaedic Association (JOA) score and the JOA score of the motor function of the upper extremity (MU-JOA score).Results.The binary classification model indexes were as follows: sensitivity, 84.0%; specificity, 60.7%; accuracy, 75.6%; area under the curve, 0.85. The Spearman rank correlation coefficient between the estimated score and the total JOA score was 0.44 and that between the estimated score and the MU-JOA score was 0.51.Conclusion.Our binary classification model using a machine learning algorithm and Leap Motion could classify CM with high sensitivity and would be useful for CM screening in daily life before consulting doctors and telemedicine.Level of Evidence: 3.

Metaverse has been attracting more and more attention because of its potential for various use cases. In metaverse applications, the seamless integration of digital and physical worlds is vital for synchronizing information from one world to another. One way to achieve this is to reconstruct 3D environmental maps every time, which is not feasible due to computational complexity. A cheaper alternative is to detect what objects have changed and update only the changed objects. To build the foundation of the change detection algorithm for that, in this paper, we propose a change detection method combined with object classification. Despite its simplicity, the experiment showed promising results with an object detector fine-tuned with data from the target environment. Furthermore, with our clustering-based post-processing, false positives produced by the frame-wise change detection were observed to be successfully suppressed.

This paper addresses the task of class-imbalanced medical image classification using complication labels. Medical image classification plays an essential role in improving patient care and reducing the burden on healthcare systems, along with other computer aided diagnosis tasks. However, there are two common obstacles with medical image image classification. One is that there is a high cost of acquiring annotations from doctors/experts. Two is that in many cases there is a differences in the number of incidents that appear per symptom which lead to class-imbalance. The above factors make medical image classification difficult to deal with. Additionally, there are cases where multiple symptoms appear in a mixed manner (complications), such as diabetic complications where both symptoms of infection and ischaemia could appear at the same time, which also could influence the classification result negatively. In this paper, a method that jointly tackles the obstacles mentioned above is presented. First, we conduct extensive experiment with various class-imbalanced learning methods introduced in previous works and also propose a method that improves the baseline class-imbalanced approaches by utilizing complication labels in pre-training. Second, we validate the proposed method with the diabetic foot ulcer dataset introduced in Diabetic Foot Ulcer Challenge 2021.

Object handover is a fundamental and essential capability for robots interacting with humans in many applications such as household chores. In this challenge, we estimate the physical properties of a variety of containers with different fillings such as container capacity and the type and percentage of the content to achieve collaborative physical handover between humans and robots. We introduce multi-modal prediction models using audio-visual-datasets of people interacting with containers distributed by CORSMAL.

The ability to recognize and identify terrain characteristics is an essential function required for many autonomous ground robots such as social robots, assistive robots, autonomous vehicles, and ground exploration robots. Recognizing and identifying terrain characteristics is challenging because similar terrains may have very different appearances (e.g., carpet comes in many colors), while terrains with very similar appearance may have very different physical properties (e.g., mulch versus dirt). In order to address the inherent ambiguity in vision-based terrain recognition and identification, we propose a multi-modal self-supervised learning technique that switches between audio features extracted from a microphone attached to the underside of a mobile platform and image features extracted by a camera on the platform to cluster terrain types. The terrain cluster labels are then used to train an image-based real-time CNN (Convolutional Neural Network) to predict terrain types changes. Through experiments, we demonstrate that the proposed self-supervised terrain type recognition method achieves over 80% accuracy, which greatly outperforms several baselines and suggests strong potential for assistive applications.

This position paper introduces the concept of a real-time auditory feedback system designed to promote physical activity and improve well-being through visual skill development in ball sports. The system uses event cameras to translate ball trajectories into intuitive sound cues and complements visual perception with high-temporal-resolution auditory feedback, enabling precise timing and enhanced engagement during practice. Unlike conventional setups requiring wearables or complex hardware, this approach ensures seamless interaction while preserving natural movement. This multimodal approach can potentially offer a scalable and immersive solution for various sports and real-world environments.