Published papers

Ball spin estimation in sports is important for analyzing the game. Since spin is generally too fast to be captured by a conventional camera, a high-speed camera is often used to capture images of the ball and estimate its spin. However, since a high-speed camera is not robust to changes in the lighting conditions, it is difficult to estimate spin in some environments. To solve these problems, this paper proposes a new method for ball spin estimation using an event camera. An event camera is a sensor inspired by the visual system of animals, which outputs the brightness changes in a scene. Event cameras have advantages such as high temporal resolution and high dynamic range, and can accurately capture the motion of a fast-spinning ball in various lighting conditions. Experimental results in a synthesized dataset showed that the proposed method can stably estimate spin up to 500 rps. It is also confirmed that the proposed method can estimate spin in the data obtained from actual sports games.

Motion blur can significantly reduce the quality of images, and researchers have developed various algorithms to address this issue. One common approach to deblurring is to use deconvolution to cancel out the blur effect, but this method is limited by the difficulty of accurately estimating blur kernels from blurred images. This is because the motion causing the blur is often complex and nonlinear. In this paper, a new method for estimating blur kernels is proposed. This method uses an event camera, which captures high-temporal-resolution data on pixel luminance changes, along with a conventional camera to capture the input blurred image. By analyzing the event data stream, the proposed method estimates the 2D motion of the blurred image at short intervals during the exposure time, and integrates this information to estimate a variety of complex blur motions. With the estimated blur kernel, the input blurred image can be deblurred using deconvolution. The proposed method does not rely on machine learning and therefore can restore blurry images without depending on the quality and quantity of training data. Experimental results show that the proposed method can estimate blur kernels even for images blurred by complex camera motions, outperforming conventional methods. Overall, this paper presents a promising approach to motion deblurring that could have practical applications in a range of fields.

Event cameras, known for their high dynamic range, absence of motion blur, and low energy usage, have recently found a wide range of applications thanks to these attributes. In the past few years, the field of event-based 3D reconstruction saw remarkable progress, with the Neural Radiance Field (NeRF) based approach demonstrating photorealistic view synthesis results. However, the volume rendering paradigm of NeRF necessitates extensive training and rendering times. In this paper, we introduce Event Enhanced Gaussian Splatting (E2GS), a novel method that incorporates event data into Gaussian Splatting, which has recently made significant advances in the field of novel view synthesis. Our E2GS effectively utilizes both blurry images and event data, significantly improving image deblurring and producing high-quality novel view synthesis. Our comprehensive experiments on both synthetic and real-world datasets demonstrate our E2GS can generate visually appealing renderings while offering faster training and rendering speed (140 FPS). Our code is available at https://github.com/deguchihiroyuki/E2GS.

Training and education have become increasingly crucial in obtaining new skills in a variety of fields, especially in assembly and disassembly operations. The main issue in mechanical engineering, particularly in the assembly department, was that automobile engine components assembly was found to be complicated and challenging to assemble using an existing method, where they only rely on a video-based method. The purpose of this paper is to create interactive Virtual Reality (VR) and Augmented Reality (AR) applications that allow users to efficiently assist and complete the assembly tasks. In this work, the authors designed and developed a fully immersive VR application using an HTC Vive headset and two AR applications (marker-less AR application and marker-based AR application) using EPSON MOVERIO BT-300 (AR Smart Glasses). Fourteen engineering students from Universiti Putra Malaysia were selected for the experiment. They were divided into four groups: video-based group, VR-based group, marker-less AR group, and marker-based AR group. They are required to complete all four experiments (video-based experiment, VR-based experiment, marker-less AR experiment, and marker-based AR experiment). The results showed that the marker-less AR application is the best impressive method (37% better), the VR application is the second impressive method (23% better) followed by the marker-based AR application is the third impressive method (3% better) compared to the existing video-based guideline. Therefore, the students favored AR and VR applications rather than the existing method to be used in automobile engine assembly tasks.

Only virtual hands and feet move synchronously with an observer's movement, inducing body ownership of an invisible body between them. However, it is unclear whether body ownership is also induced when the size of the invisible body is changed. In this study, we investigated whether body ownership is induced in large or small invisible bodies and whether size perception changes with the size of the invisible body. The results showed that body ownership was induced even if the size of the invisible body was changed, but the size perception did not change.

Egocentric open-surgery videos capture rich, fine-grained details essential for accurately modelling surgical procedures and human behaviour in the operating room. A detailed, pixel-level understanding of hands and surgical tools is crucial for interpreting a surgeon action and intention. We introduce EgoSurgery-HTS, a new dataset with pixel-wise annotations and a benchmark suite for segmenting surgical tools, hands, and interacting tools in egocentric open-surgery videos. Specifically, we provide a labelled dataset for (1) tool instance segmentation of 14 distinct surgical tools, (2) hand instance segmentation, and (3) hand–tool segmentation to label main operating hands and the tools they manipulate. Using EgoSurgery-HTS, we conduct extensive evaluations of state-of-the-art segmentation methods and demonstrate significant improvements in the accuracy of hand and hand–tool segmentation in egocentric open-surgery videos compared to existing datasets. The dataset will be released upon acceptance.

A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However, the past crowd density maps are often incomplete due to the miss-detection of pedestrians, and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion frame-work for crowd density forecasting (CrowdMAC), which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e., forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e., imputing past maps with miss-detection). Additionally, we propose Temporal-Density-aware Masking (TDM), which non-uniformly masks tokens in the observed crowd density map, considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover, we introduce multitask masking to enhance training efficiency. In the experiments, CrowdMAC achieves state-of-the-art performance on seven large-scale datasets, including SDD, ETH-UCY, inD, JRDB, VSCrowd, FDST, and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at project page.

There has been a substantial amount of research involving computer methods and technology for the detection and recognition of diabetic foot ulcers (DFUs), but there is a lack of systematic comparisons of state-of-the-art deep learning object detection frameworks applied to this problem. DFUC2020 provided participants with a comprehensive dataset consisting of 2,000 images for training and 2,000 images for testing. This paper summarizes the results of DFUC2020 by comparing the deep learning-based algorithms proposed by the winning teams: Faster R–CNN, three variants of Faster R–CNN and an ensemble method; YOLOv3; YOLOv5; EfficientDet; and a new Cascade Attention Network. For each deep learning method, we provide a detailed description of model architecture, parameter settings for training and additional stages including pre-processing, data augmentation and post-processing. We provide a comprehensive evaluation for each method. All the methods required a data augmentation stage to increase the number of images available for training and a post-processing stage to remove false positives. The best performance was obtained from Deformable Convolution, a variant of Faster R–CNN, with a mean average precision (mAP) of 0.6940 and an F1-Score of 0.7434. Finally, we demonstrate that the ensemble method based on different deep learning methods can enhance the F1-Score but not the mAP.

We propose a method for dense depth estimation from an event stream generated when sweeping the focal plane of the driving lens attached to an event camera. In this method, a depth map is inferred from an 'event focal stack' composed of the event stream using a convolutional neural network trained with synthesized event focal stacks. The synthesized event stream is created from a focal stack generated by Blender for any arbitrary 3D scene. This allows for training on scenes with diverse structures. Additionally, we explored methods to eliminate the domain gap between real event streams and synthetic event streams. Our method demonstrates superior performance over a depth-from-defocus method in the image domain on synthetic and real datasets.

Most robots implicitly assume that the road surface on which they move is flat, without differences in level. Detecting differences in level on roads contributes to robots moving safely without stacking and falling. Although there are some studies on detecting differences in level in RGB or RGB-D images, directly finding only differences in level on roads is difficult due to the abundance and complexity of the types of differences in level on roads. This paper presents a new method for detecting differences in level from RGB-D images obtained by a modern smartphone equipped with a high-performance depth camera. First, we extract a part of differences in level on roads by finding the change of the normal vector in the contour of the detected plane. Then, a deep learning model trained on the dataset created by using the extracted image patches is used for detecting all the differences in level in outdoor images. To evaluate the effectiveness of the proposed method, quantitative and qualitative comparisons with existing methods were conducted. Further, the results from various inputs were qualitatively and quantitatively evaluated. As a result, we verified that the proposed method was able to detect all differences in level in an image, even in complex scenes where existing methods cannot detect.

Although recent deep learning-based inpainting techniques achieve excellent restoration quality, their stringent requirement for computational resources such as GPU/VRAM render them difficult to use in settings where high-end hardware is not available. We address this gap between research and real-world applications in our submission to the DREAMING challenge, which explores such a use case - diminished reality - where run time and GPU hardware are limited. Specifically, this paper proposes a method that divides a video optimally into sections of variable lengths for the downstream inpainting model. By maximizing the number of frames, i.e., spatio-temporal information, in each subvideo, inference using a SOTA model can often be performed without significant degradation to output quality. In addition, to expedite inference, we examine techniques that intelligently reduce the amount of input pixels, e.g., downsampling and cropping, while maintaining decent inpainting quality.

A recent singleshot multiplane image (MPI) generation enables to copy an observed reality within a camera frame into other reality domains via view synthesis. While the scene scale is unknown due to the nature of singleshot MPI processing, camera tracking algorithms can estimate depth within the application world coordinate system. Given such depth information, we propose to adjust the scale of singleshot MPI to that of the currently observed scene. We find the individual scales of the MPI layers by minimizing the differences between the depth of MPI rendering and that of camera tracking. We eventually found that many layers fall within a close depth. Therefore, we merge such layers into one to compact the MPI representation. We compared our method with baselines using real and synthetic datasets with dense and sparse depth inputs. Our results demonstrate that our algorithm achieves higher scores in image metrics and reduces MPI data amount by up to 78%.

In equestrian competitions, the precision of horse movements significantly impacts scoring. Accurate recognition and analysis of these movements can enhance the scoring process. Existing action recognition methods for sports are primarily designed for humans, presenting challenges when applied to equestrian sports where horses are the main subjects. Additionally, research on animal behavior has been largely confined to behavior recognition. This paper evaluates and compares the advantages and limitations of different modalities by constructing a dataset of three distinct modalities specific to equestrian sports and applying existing human action recognition techniques. Our findings indicate that optical flow is the most practical modality. Furthermore, combining the three modalities achieved a motion recognition accuracy of 0.97. By removing error-prone data—such as instances where the horse is facing forward or backward, or when obstacles are present around the horse’s limbs—the action recognition accuracy increased to 0.99. The dataset is available at https://github.com/kotokamatsunaga/horsedataset.git.

Machine vision systems are crucial for quality control in manufacturing, ensuring that products meet standards through automatic in-line visual inspections. While reference images are typically used as benchmarks for comparison, a significant challenge arises when objects arrive at inspection points in misaligned orientations. This misalignment can lead to erroneous decisions by automated systems, resulting in false positives or negatives that increase waste and slow production. To address this issue, we propose a visual inspection pipeline that leverages recent machine-learning-based approaches. Our pipeline virtually reorients a 3D reference to the inspection target's orientation for comparison, effectively addressing the misalignment issue. Specifically, it integrates 3D Gaussian Splatting and MASt3R, enabling a robust 3D-based defect detection system that uses only a single camera sensor, thereby moving beyond traditional 2D image-based methods. We validated our approach in real-world scenarios by testing it on 20 synthetic objects and real industrial parts. The proposed pipeline maintains accuracy and processing speed comparable to those of existing methods.

In recent years, abnormal human activity detection has become an important research topic. However, most existing methods focus on detecting abnormal activities of pedestrians in surveillance videos; even those methods using egocentric videos deal with the activities of pedestrians around the camera wearer. In this paper, the authors present an unsupervised auto-encoder-based network trained by one-class learning that inputs RGB image sequences recorded by egocentric cameras to detect abnormal activities of the camera wearers themselves. To improve the performance of network, the authors introduce a ‘re-encoding’ architecture and a regularisation loss function term, minimising the KL divergence between the distributions of features extracted by the first and second encoders. Unlike the common use of KL divergence loss to obtain a feature distribution close to an already-known distribution, the aim is to encourage the features extracted by the second encoder to have a close distribution to those extracted from the first encoder. The authors evaluate the proposed method on the Epic-Kitchens-55 dataset and conduct an ablation study to analyse the functions of different components. Experimental results demonstrate that the method outperforms the comparison methods in all cases and demonstrate the effectiveness of the proposed re-encoding architecture and the regularisation term.

Recording surgery is an important technique for education and the evaluation of medical treatments. However, capturing targets such as the surgical field, surgical tools, and the surgeon's hands, is almost impossible since these targets are heavily occluded by the surgeon's head and body during a surgery. We used a recording system in which multiple cameras are installed in the surgical lump, supposing at least one camera would capture the target without occlusion. As this system records multiple video sequences, we address the task to select a best view camera automatically. Recently, learning-based approaches in a fully supervised manner have been proposed for this task, but these previous approaches completely rely on manual annotation of the training data. In this paper, we focus on the eye tracker mounted on the surgeon's head, which can capture the recording targets without occlusion. Employing this first-person-view video synchronized with multiple videos of the surgical lump, we propose a novel camera selection approach using a self-supervised learning framework. In experiments, we created a dataset composed of four different breast surgery. Our extended experiments showed that our approach successfully switched to the best camera view without manual annotation and achieved competitive accuracy compared with conventional supervised methods. Also, our approach yielded effective visual representations comparable to state-of-the-art self-supervised learning frameworks.

Study Design.Cross-sectional study.Objective.To develop a binary classification model for cervical myelopathy (CM) screening based on a machine learning algorithm using Leap Motion (Leap Motion, San Francisco, CA), a novel noncontact sensor device.Summary of Background Data.Progress of CM symptoms are gradual and cannot be easily identified by the patients themselves. Therefore, screening methods should be developed for patients of CM before deterioration of myelopathy. Although some studies have been conducted to objectively evaluate hand movements specific to myelopathy using cameras or wearable sensors, their methods are unsuitable for simple screening outside hospitals because of the difficulty in obtaining and installing their equipment and the long examination time.Methods.In total, 50 and 28 participants in the CM and control groups were recruited, respectively. The diagnosis of CM was made by spine surgeons. We developed a desktop system using Leap Motion that recorded 35 parameters of fingertip movements while participants gripped and released their fingers as rapidly as possible. A support vector machine was used to develop the binary classification model, and a multiple linear regression analysis was performed to create regression models to estimate the total Japanese Orthopaedic Association (JOA) score and the JOA score of the motor function of the upper extremity (MU-JOA score).Results.The binary classification model indexes were as follows: sensitivity, 84.0%; specificity, 60.7%; accuracy, 75.6%; area under the curve, 0.85. The Spearman rank correlation coefficient between the estimated score and the total JOA score was 0.44 and that between the estimated score and the MU-JOA score was 0.51.Conclusion.Our binary classification model using a machine learning algorithm and Leap Motion could classify CM with high sensitivity and would be useful for CM screening in daily life before consulting doctors and telemedicine.Level of Evidence: 3.

Metaverse has been attracting more and more attention because of its potential for various use cases. In metaverse applications, the seamless integration of digital and physical worlds is vital for synchronizing information from one world to another. One way to achieve this is to reconstruct 3D environmental maps every time, which is not feasible due to computational complexity. A cheaper alternative is to detect what objects have changed and update only the changed objects. To build the foundation of the change detection algorithm for that, in this paper, we propose a change detection method combined with object classification. Despite its simplicity, the experiment showed promising results with an object detector fine-tuned with data from the target environment. Furthermore, with our clustering-based post-processing, false positives produced by the frame-wise change detection were observed to be successfully suppressed.

This paper addresses the task of class-imbalanced medical image classification using complication labels. Medical image classification plays an essential role in improving patient care and reducing the burden on healthcare systems, along with other computer aided diagnosis tasks. However, there are two common obstacles with medical image image classification. One is that there is a high cost of acquiring annotations from doctors/experts. Two is that in many cases there is a differences in the number of incidents that appear per symptom which lead to class-imbalance. The above factors make medical image classification difficult to deal with. Additionally, there are cases where multiple symptoms appear in a mixed manner (complications), such as diabetic complications where both symptoms of infection and ischaemia could appear at the same time, which also could influence the classification result negatively. In this paper, a method that jointly tackles the obstacles mentioned above is presented. First, we conduct extensive experiment with various class-imbalanced learning methods introduced in previous works and also propose a method that improves the baseline class-imbalanced approaches by utilizing complication labels in pre-training. Second, we validate the proposed method with the diabetic foot ulcer dataset introduced in Diabetic Foot Ulcer Challenge 2021.

In the visual inspection of industrial products using a microscope with a shallow depth of field, it is difficult to capture an all-in-focus image. This paper presents a method to generate an all-in-focus image from a large focal stack, which can be applied to visual inspection. The proposed method adds two improvements to a deep learning method [M. Maximov et al., CVPR, (2020) 1068] that leverages defocus cues in depth estimation. First, it interpolates an index map (a collection of in-focus image indices at all pixels) after the forward pass. It generates an all-in-focus image using all images in focal stacks while reducing the number of images input to the network. Second, the network is trained with synthetic datasets to which a random texture with high-frequency components is added. It helps the network to learn the degree of defocus blur. Experiments using synthetic images and real microscope images show that interpolating the index map and adding texture improves the accuracy of all-in-focus image generation.