Published papers

This study addresses a robotic scooping task for deformable plastic objects like wet soil, pastes, or plasticines. In this task, a robot arm holding a spoon scoops the object with an arbitrary target weight. Scooping deformable plastic objects presents a challenge in robot control. The robot needs to consider the object's invisible physical properties, which can be varied due to extrinsic factors. For this reason, the same scooping action on the same objects can exhibit different scooping outcomes. We tackle this problem with tactile sensors. If a robot perceives the reactive force during the scooping action through the sensors, it will lead to more accurate weighing. We realize this concept with a scooped weight prediction neural network. We first train the network to predict scooped weights, where input is tactile and motion parameters. Then, we identify the optimal motion parameters from the tactile and target weights. To verify the proposed method, we performed experiments by comparing two approaches: our proposed method with tactile sensors and the baseline without them. We trained the scoop weight predictive model with 80 scooping motions by randomizing one of the parameters. Our method had 1.40, 1.40, 1.30, 1.75, and 1.64 g average errors in scooped weights with 3, 5, 7, 9, and 11 g target weights, respectively, demonstrating smaller errors for most target weights than the baseline. The proposed method is easy to implement and potentially available for various applications such as food industries, laboratory automation, and assistive feeding in everyday life.

Multi-Object Tracking (MOT) plays a critical role in analyzing player behavior from videos, enabling performance evaluation. Current MOT methods are often evaluated using publicly available datasets. However, most of these focus on everyday scenarios such as pedestrian tracking or are tailored to specific sports, including soccer and basketball. Despite the inherent challenges of tracking players in American football, such as frequent occlusion and physical contact, no standardized dataset has been publicly available, making fair comparisons between methods difficult. To address this gap, we constructed the first dedicated detection and tracking dataset for the American football players and conducted a comparative evaluation of various detection and tracking methods. Our results demonstrate that accurate detection and tracking can be achieved even in crowded scenarios. Fine-tuning detection models improved performance over pre-trained models. Furthermore, when these fine-tuned detectors and re-identification models were integrated into tracking systems, we observed notable improvements in tracking accuracy compared to existing approaches. This work thus enables robust detection and tracking of American football players in challenging, high-density scenarios previously underserved by conventional methods.

Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods.

Occlusion-free video generation is challenging due to surgeons’ obstructions in the camera field of view. Prior work has addressed this issue by installing multiple cameras on a surgical light, hoping some cameras will observe the surgical field with less occlusion. However, this special camera setup poses a new imaging challenge since camera configurations can change every time surgeons move the light, and manual image alignment is required. This paper proposes an algorithm to automate this alignment task. The proposed method detects frames where the lighting system moves, realigns them, and selects the camera with the least occlusion. This algorithm results in a stabilized video with less occlusion. Quantitative results show that our method outperforms conventional approaches. A user study involving medical doctors also confirmed the superiority of our method.

Machine vision systems play a pivotal role in streamlining manufacturing processes, notably in quality control through automatic in-line visual inspections. A common practice for inspecting parts, components, and final products is to use a master part benchmark for quality comparison. However, challenges arise when objects enter inspection points in unintended orientations. This misalignment potentially leads to erroneous decisions by automated systems, resulting in additional checkpoints or wastage affecting the production rate. To tackle this issue, we propose a visual inspection pipeline that leverages recent machine learning-based approaches to compare the inspection target and a master part virtually oriented to the same perspective. Specifically, we suggest combining 3D Gaussian Splatting and DUSt3R as a practical solution. Our approach demonstrates its efficacy in real-world scenarios through testing on three mock parts and a real industrial component.

Video analysis has become an essential aspect of net sports, such as badminton. Accurately predicting the future trajectory of balls and shuttlecocks can significantly benefit players by enhancing their performance and enabling them to devise effective game strategies. This paper aims to analyze data to provide players with an advantage in the fast-paced rallies of badminton matches. The paper delves into the innovative task of predicting future shuttlecock trajectories in badminton match videos and presents a method that takes into account both the shuttlecock position and the positions and postures of the players. In the experiments, players were extracted from the match video, their postures were analyzed, and a time-series model was trained. The results indicate that the proposed method improved accuracy by 13% compared to methods that solely used shuttlecock position information as input, and by 8.4% compared to methods that employed both shuttlecock and player position information as input.

How much can we infer about human shape using an event camera that only detects the pixel position where the luminance changed and its timestamp? This neuromorphic vision technology captures changes in pixel values at ultra-high speeds, regardless of the variations in environmental lighting brightness. Existing methods for human mesh recovery (HMR) from event data need to utilize intensity images captured with a generic frame-based camera, rendering them vulnerable to low-light conditions, energy/memory constraints, and privacy issues. In contrast, we explore the potential of solely utilizing event data to alleviate these issues and ascertain whether it offers adequate cues for HMR, as illustrated in Fig. 1. This is a quite challenging task due to the substantially limited information ensuing from the absence of intensity images. To this end, we propose EventPointMesh, a framework which treats event data as a three-dimensional (3D) spatio-temporal point cloud for reconstructing the human mesh. By employing a coarse-to-fine pose feature extraction strategy, we extract both global features and local features. The local features are derived by processing the spatio-temporally dispersed event points into groups associated with individual body segments. This combination of global and local features allows the framework to achieve a more accurate HMR, capturing subtle differences in human movements. Experiments demonstrate that our method with only sparse event data outperforms baseline methods.

When a camera travels across a 3D world, only a fraction of pixel value changes; an event-based camera observes the change as sparse events. How can we utilize sparse events for efficient recovery of the camera pose? We show that we can recover the camera pose by minimizing the error between sparse events and the temporal gradient of the scene represented as a neural radiance field (NeRF). To enable the computation of the temporal gradient of the scene, we augment NeRF's camera pose as a time function. When the input pose to the NeRF coincides with the actual pose, the output of the temporal gradient of NeRF equals the observed intensity changes on the event's points. Using this principle, we propose an event-based camera pose tracking framework called TeGRA which realizes the pose update by using the sparse event's observation. To the best of our knowledge, this is the first camera pose estimation algorithm using the scene's implicit representation and the sparse intensity change from events.

Ball detection in sports, particularly in fast-paced games like volleyball, where the ball is constantly in high motion, presents a significant challenge for game analysis and automated sports broadcasting. Conventional camera-based ball detection faces issues, such as motion blur, in high-speed ball movement scenes. To address these challenges, we propose a deep learning-based method for detecting balls using event cameras. Event cameras, also known as dynamic vision sensors, operate differently from traditional cameras. Instead of capturing frames at fixed intervals, they record individual pixel-level luminance changes, referred to as events. This unique feature enables event cameras to provide precise temporal information with low latency. Our proposed method transforms sparse events into an image format, enabling the use of current deep-learning architectures for object detection. Given the limited amount of events available for training an object detector, we generate synthetic events from RGB frames. This approach reduces the need for extensive annotation and ensures sufficient data availability. Experimental results confirm that our proposed method can detect balls that are undetectable in RGB frames and outperform existing methods that utilize event-based ball detection. Moreover, we conducted tests to verify our method's ability to detect balls in real events, not just synthetic ones. These results demonstrate that our proposed method opens up new possibilities in sports ball detection.

High-speed phenomena, which are important in fields such as sports and fluid dynamics, are often challenging for the human visual system to perceive precisely. To better understand these high-speed phenomena, it is important to present users the information intuitively and precisely without time delays. Existing visualization techniques, such as high-speed cameras and stroboscopes, are effective for capturing high-speed events but are not well-suited for interactive visualization purposes.In this study, we propose an interactive visualization system of high-speed events using an event camera. The system precisely captures high-speed phenomena and allows users to replay important scenes instantly with a single key press. Through a user study involving 38 participants, we confirmed that the event camera system enhances human visual perception. Furthermore, a case study demonstrated that the interactive design enables users to interpret high-speed phenomena more intuitively. These findings suggest that our system can effectively augment users' visual perception.

Ball spin estimation in sports is important for analyzing the game. Since spin is generally too fast to be captured by a conventional camera, a high-speed camera is often used to capture images of the ball and estimate its spin. However, since a high-speed camera is not robust to changes in the lighting conditions, it is difficult to estimate spin in some environments. To solve these problems, this paper proposes a new method for ball spin estimation using an event camera. An event camera is a sensor inspired by the visual system of animals, which outputs the brightness changes in a scene. Event cameras have advantages such as high temporal resolution and high dynamic range, and can accurately capture the motion of a fast-spinning ball in various lighting conditions. Experimental results in a synthesized dataset showed that the proposed method can stably estimate spin up to 500 rps. It is also confirmed that the proposed method can estimate spin in the data obtained from actual sports games.

Machine vision systems are crucial for quality control in manufacturing, ensuring that products meet standards through automatic in-line visual inspections. While reference images are typically used as benchmarks for comparison, a significant challenge arises when objects arrive at inspection points in misaligned orientations. This misalignment can lead to erroneous decisions by automated systems, resulting in false positives or negatives that increase waste and slow production. To address this issue, we propose a visual inspection pipeline that leverages recent machine-learning-based approaches. Our pipeline virtually reorients a 3D reference to the inspection target's orientation for comparison, effectively addressing the misalignment issue. Specifically, it integrates 3D Gaussian Splatting and MASt3R, enabling a robust 3D-based defect detection system that uses only a single camera sensor, thereby moving beyond traditional 2D image-based methods. We validated our approach in real-world scenarios by testing it on 20 synthetic objects and real industrial parts. The proposed pipeline maintains accuracy and processing speed comparable to those of existing methods.

Motion blur can significantly reduce the quality of images, and researchers have developed various algorithms to address this issue. One common approach to deblurring is to use deconvolution to cancel out the blur effect, but this method is limited by the difficulty of accurately estimating blur kernels from blurred images. This is because the motion causing the blur is often complex and nonlinear. In this paper, a new method for estimating blur kernels is proposed. This method uses an event camera, which captures high-temporal-resolution data on pixel luminance changes, along with a conventional camera to capture the input blurred image. By analyzing the event data stream, the proposed method estimates the 2D motion of the blurred image at short intervals during the exposure time, and integrates this information to estimate a variety of complex blur motions. With the estimated blur kernel, the input blurred image can be deblurred using deconvolution. The proposed method does not rely on machine learning and therefore can restore blurry images without depending on the quality and quantity of training data. Experimental results show that the proposed method can estimate blur kernels even for images blurred by complex camera motions, outperforming conventional methods. Overall, this paper presents a promising approach to motion deblurring that could have practical applications in a range of fields.

Only virtual hands and feet move synchronously with an observer's movement, inducing body ownership of an invisible body between them. However, it is unclear whether body ownership is also induced when the size of the invisible body is changed. In this study, we investigated whether body ownership is induced in large or small invisible bodies and whether size perception changes with the size of the invisible body. The results showed that body ownership was induced even if the size of the invisible body was changed, but the size perception did not change.

Egocentric open-surgery videos capture rich, fine-grained details essential for accurately modelling surgical procedures and human behaviour in the operating room. A detailed, pixel-level understanding of hands and surgical tools is crucial for interpreting a surgeon action and intention. We introduce EgoSurgery-HTS, a new dataset with pixel-wise annotations and a benchmark suite for segmenting surgical tools, hands, and interacting tools in egocentric open-surgery videos. Specifically, we provide a labelled dataset for (1) tool instance segmentation of 14 distinct surgical tools, (2) hand instance segmentation, and (3) hand–tool segmentation to label main operating hands and the tools they manipulate. Using EgoSurgery-HTS, we conduct extensive evaluations of state-of-the-art segmentation methods and demonstrate significant improvements in the accuracy of hand and hand–tool segmentation in egocentric open-surgery videos compared to existing datasets. The dataset will be released upon acceptance.

Event cameras, known for their high dynamic range, absence of motion blur, and low energy usage, have recently found a wide range of applications thanks to these attributes. In the past few years, the field of event-based 3D reconstruction saw remarkable progress, with the Neural Radiance Field (NeRF) based approach demonstrating photorealistic view synthesis results. However, the volume rendering paradigm of NeRF necessitates extensive training and rendering times. In this paper, we introduce Event Enhanced Gaussian Splatting (E2GS), a novel method that incorporates event data into Gaussian Splatting, which has recently made significant advances in the field of novel view synthesis. Our E2GS effectively utilizes both blurry images and event data, significantly improving image deblurring and producing high-quality novel view synthesis. Our comprehensive experiments on both synthetic and real-world datasets demonstrate our E2GS can generate visually appealing renderings while offering faster training and rendering speed (140 FPS). Our code is available at https://github.com/deguchihiroyuki/E2GS.

Training and education have become increasingly crucial in obtaining new skills in a variety of fields, especially in assembly and disassembly operations. The main issue in mechanical engineering, particularly in the assembly department, was that automobile engine components assembly was found to be complicated and challenging to assemble using an existing method, where they only rely on a video-based method. The purpose of this paper is to create interactive Virtual Reality (VR) and Augmented Reality (AR) applications that allow users to efficiently assist and complete the assembly tasks. In this work, the authors designed and developed a fully immersive VR application using an HTC Vive headset and two AR applications (marker-less AR application and marker-based AR application) using EPSON MOVERIO BT-300 (AR Smart Glasses). Fourteen engineering students from Universiti Putra Malaysia were selected for the experiment. They were divided into four groups: video-based group, VR-based group, marker-less AR group, and marker-based AR group. They are required to complete all four experiments (video-based experiment, VR-based experiment, marker-less AR experiment, and marker-based AR experiment). The results showed that the marker-less AR application is the best impressive method (37% better), the VR application is the second impressive method (23% better) followed by the marker-based AR application is the third impressive method (3% better) compared to the existing video-based guideline. Therefore, the students favored AR and VR applications rather than the existing method to be used in automobile engine assembly tasks.

Most robots implicitly assume that the road surface on which they move is flat, without differences in level. Detecting differences in level on roads contributes to robots moving safely without stacking and falling. Although there are some studies on detecting differences in level in RGB or RGB-D images, directly finding only differences in level on roads is difficult due to the abundance and complexity of the types of differences in level on roads. This paper presents a new method for detecting differences in level from RGB-D images obtained by a modern smartphone equipped with a high-performance depth camera. First, we extract a part of differences in level on roads by finding the change of the normal vector in the contour of the detected plane. Then, a deep learning model trained on the dataset created by using the extracted image patches is used for detecting all the differences in level in outdoor images. To evaluate the effectiveness of the proposed method, quantitative and qualitative comparisons with existing methods were conducted. Further, the results from various inputs were qualitatively and quantitatively evaluated. As a result, we verified that the proposed method was able to detect all differences in level in an image, even in complex scenes where existing methods cannot detect.

Although recent deep learning-based inpainting techniques achieve excellent restoration quality, their stringent requirement for computational resources such as GPU/VRAM render them difficult to use in settings where high-end hardware is not available. We address this gap between research and real-world applications in our submission to the DREAMING challenge, which explores such a use case - diminished reality - where run time and GPU hardware are limited. Specifically, this paper proposes a method that divides a video optimally into sections of variable lengths for the downstream inpainting model. By maximizing the number of frames, i.e., spatio-temporal information, in each subvideo, inference using a SOTA model can often be performed without significant degradation to output quality. In addition, to expedite inference, we examine techniques that intelligently reduce the amount of input pixels, e.g., downsampling and cropping, while maintaining decent inpainting quality.

A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However, the past crowd density maps are often incomplete due to the miss-detection of pedestrians, and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion frame-work for crowd density forecasting (CrowdMAC), which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e., forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e., imputing past maps with miss-detection). Additionally, we propose Temporal-Density-aware Masking (TDM), which non-uniformly masks tokens in the observed crowd density map, considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover, we introduce multitask masking to enhance training efficiency. In the experiments, CrowdMAC achieves state-of-the-art performance on seven large-scale datasets, including SDD, ETH-UCY, inD, JRDB, VSCrowd, FDST, and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at project page.