Published papers

Multi-layer images are a powerful scene representation for high-performance rendering in virtual/augmented reality (VR/AR). The major approach to generate such images is to use a deep neural network trained to encode colors and alpha values of depth certainty on each layer using registered multi-view images. A typical network is aimed at using a limited number of nearest views. Therefore, local noises in input images from a user-navigated camera deteriorate the final rendering quality and interfere with coherency over view transitions. We propose to use a focal stack composed of multi-view inputs to diminish such noises. We also provide theoretical analysis for ideal focal stacks to generate multi-layer images. Our results demonstrate the advantages of using focal stacks in coherent rendering, memory footprint, and AR-supported data capturing. We also show three applications of imaging for VR.

Mounting multi-view cameras within a surgical light is a practical choice since somecameras are expected to observe surgery with few occlusions. Such multi-view videos must bereassembled for easy reference. A typical way is to reconstruct the surgery in 3D. However, thegeometrical relationship among cameras is changed because each camera independently moves everytime the lighting is reconfigured (i.e., every time surgeons touch the surgical light). Moreover, featurematching between surgical images is potentially challenging because of missing rich features. Toaddress the challenge, we propose a feature-matching strategy that enables robust calibration ofthe multi-view camera system by collecting a set of a small number of matches over time whilethe cameras stay stationary. Our approach would enable conversion from multi-view videos to a3D video. However, surgical videos are long and, thus, the cost of the conversion rapidly grows.Therefore, we implement a video player where only selected frames are converted to minimizetime and data until playbacks. We demonstrate that sufficient calibration quality with real surgicalvideos can lead to a promising 3D mesh and a recently emerged 3D multi-layer representation. Wereviewed comments from surgeons to discuss the differences between those 3D representations on anautostereoscopic display with respect to medical usage.

Due to the recent technological advances in inertial measurement units (IMUs), many applications for the measurement of human motion using multiple body-worn IMUs have been developed. In these applications, each IMU has to be attached to a predefined body segment. A technique to identify the body segment on which each IMU is mounted allows users to attach inertial sensors to arbitrary body segments, which avoids having to remeasure due to incorrect attachment of the sensors. We address this IMU-to-segment assignment problem and propose a novel end-to-end learning model that incorporates a global feature generation module and an attention-based mechanism. The former extracts the feature representing the motion of all attached IMUs, and the latter enables the model to learn the dependency relationships between the IMUs. The proposed model thus identifies the IMU placement based on the features from global motion and relevant IMUs. We quantitatively evaluated the proposed method using synthetic and real public datasets with three sensor configurations, including a full-body configuration mounting 15 sensors. The results demonstrated that our approach significantly outperformed the conventional and baseline methods for all datasets and sensor configurations.

The fourth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'21) is part of the ACM International Conference on Multimedia 2021 (ACM Multimedia 2021). Exceptionally, due to the corona pandemic, the workshop is held virtually. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding and visualizing the multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, statistical analysis and evaluation, and sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

The fifth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'22) is part of the ACM International Conference on Multimedia 2022 (ACM Multimedia 2022). After two years of pure virtual MMSports workshops due to COVID-19, MMSports'22 is held on-site again. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, for statistical analysis and evaluation, and for sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports. Related Workshop Proceedings are available in the ACM DL at: https://dl.acm.org/doi/proceedings/10.1145/3552437.

The sixth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'23) is part of the ACM International Conference on Multimedia 2023 (ACM Multimedia 2023). The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, for statistical analysis and evaluation, and for sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

In this paper, we propose a method for absolute 3D human pose estimation (HPE) with an uncalibrated monocular camera. In the case of analyzing workers’ movement with an existing uncalibrated camera, the previous method cannot estimate absolute human pose due to the lack of information on camera parameters. Our proposed method overcomes this limitation by determining the position and scale of humans based on the pose of surrounding objects. Specifically, we predict the intrinsic and extrinsic parameters of the camera through user-guided manual manipulation. Subsequently, the estimated human pose is transformed from local coordinates to global coordinates for each frame. This absolute coordinate representation allows for real-time prediction of human movements relative to objects. To assess the efficacy of our method, we conducted three kinds of experiments. A user study revealed that the proposed user-guided method archives accurate estimation of camera parameters. Quantitative evaluation using a public dataset demonstrated that our method can predict human pose with practical accuracy, providing a benchmark for future enhancements. Qualitative evaluation with a unique dataset showed that our method could easily generate digital twin representations across diverse environments and camera positions.

Occlusion-free video generation is challenging due to surgeons’ obstructions in the camera field of view. Prior work has addressed this issue by installing multiple cameras on a surgical light, hoping some cameras will observe the surgical field with less occlusion. However, this special camera setup poses a new imaging challenge since camera configurations can change every time surgeons move the light, and manual image alignment is required. This paper proposes an algorithm to automate this alignment task. The proposed method detects frames where the lighting system moves, realigns them, and selects the camera with the least occlusion. This algorithm results in a stabilized video with less occlusion. Quantitative results show that our method outperforms conventional approaches. A user study involving medical doctors also confirmed the superiority of our method.

Identifying individual humpback whales by photographs of their tails is valuable for understanding the ecology of wild whales. We have about 10,000 photos of 1,850 identified whales taken in the sea area around Okinawa over a 30-year period. The identification process on this large scale of numbers is difficult not only for the human eye but also for machine vision, as the numbers of photographs per individual whale are very low. About 30% of the whales have only a single photograph, and 80% have fewer than five. In addition, the shapes of the tails and the black and white patterns on them are vague, and these change readily with the whale’s slightest movement and changing photo-shooting conditions. We propose a practical method for identifying a humpback whale by accurate segmentation of the fluke region using a combination of deep neural networking and GrabCut. Then useful features for identifying each individual whale are extracted by both histograms of image features and wavelet transform of the trailing edge. The test results for 323 photos show the correct individuals are ranked within the top 30 for 89% of the photos, and at the same time for 76% of photos ranked at the top.

Novel view synthesis from images, for example, with 3D Gaussian splatting, has made great progress. Rendering fidelity and speed are now ready even for demanding virtual reality applications. However, the problem of assisting humans in collecting the input images for these rendering algorithms has received much less attention. High-quality view synthesis requires uniform and dense view sampling. Unfortunately, these requirements are not easily addressed by human camera operators, who are in a hurry, impatient, or lack understanding of the scene structure and the photographic process. Existing approaches to guide humans during image acquisition concentrate on single objects or neglect view-dependent material characteristics. We propose a novel situated visualization technique for scanning at multiple scales. During the scanning of a scene, our method identifies important objects that need extended image coverage to properly represent view-dependent appearance. To this end, we leverage semantic segmentation and category identification, ranked by a vision-language model. Spherical proxies are generated around highly ranked objects to guide the user during scanning. Our results show superior performance in real scenes compared to conventional view sampling strategies.

Food picking is trivial for humans but not for robots, as foods are fragile. Presetting foods' physical properties does not help robots much due to the objects' inter- and intra-category diversity. A recent study proved that learning-based fracture anticipation with tactile sensors could overcome this problem; however, the method trains the model for each food to deal with intra-category differences, and tuning robots for each food leads to an undesirable amount of food consumption. This study proposes a novel framework for learning food-picking tasks without consuming foods. The key idea is to leverage the object-breaking experiences of several reusable fragile objects instead of consuming real foods while making the picking ability object-invariant with domain generalization (DG). In real-robot experiments, we trained a model with reusable objects (toy blocks, ping-pong balls, and jellies), selected based on the three common fracture types (crack, rupture, and crush). We then tested the model with four real food objects (tofu, bananas, potato chips, and tomatoes). The results showed that the proposed combination of reusable objects' breaking experiences and DG is effective for the food-picking task.

This study addresses a robotic scooping task for deformable plastic objects like wet soil, pastes, or plasticines. In this task, a robot arm holding a spoon scoops the object with an arbitrary target weight. Scooping deformable plastic objects presents a challenge in robot control. The robot needs to consider the object's invisible physical properties, which can be varied due to extrinsic factors. For this reason, the same scooping action on the same objects can exhibit different scooping outcomes. We tackle this problem with tactile sensors. If a robot perceives the reactive force during the scooping action through the sensors, it will lead to more accurate weighing. We realize this concept with a scooped weight prediction neural network. We first train the network to predict scooped weights, where input is tactile and motion parameters. Then, we identify the optimal motion parameters from the tactile and target weights. To verify the proposed method, we performed experiments by comparing two approaches: our proposed method with tactile sensors and the baseline without them. We trained the scoop weight predictive model with 80 scooping motions by randomizing one of the parameters. Our method had 1.40, 1.40, 1.30, 1.75, and 1.64 g average errors in scooped weights with 3, 5, 7, 9, and 11 g target weights, respectively, demonstrating smaller errors for most target weights than the baseline. The proposed method is easy to implement and potentially available for various applications such as food industries, laboratory automation, and assistive feeding in everyday life.

Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods.

How much can we infer about human shape using an event camera that only detects the pixel position where the luminance changed and its timestamp? This neuromorphic vision technology captures changes in pixel values at ultra-high speeds, regardless of the variations in environmental lighting brightness. Existing methods for human mesh recovery (HMR) from event data need to utilize intensity images captured with a generic frame-based camera, rendering them vulnerable to low-light conditions, energy/memory constraints, and privacy issues. In contrast, we explore the potential of solely utilizing event data to alleviate these issues and ascertain whether it offers adequate cues for HMR, as illustrated in Fig. 1. This is a quite challenging task due to the substantially limited information ensuing from the absence of intensity images. To this end, we propose EventPointMesh, a framework which treats event data as a three-dimensional (3D) spatio-temporal point cloud for reconstructing the human mesh. By employing a coarse-to-fine pose feature extraction strategy, we extract both global features and local features. The local features are derived by processing the spatio-temporally dispersed event points into groups associated with individual body segments. This combination of global and local features allows the framework to achieve a more accurate HMR, capturing subtle differences in human movements. Experiments demonstrate that our method with only sparse event data outperforms baseline methods.

Machine vision systems play a pivotal role in streamlining manufacturing processes, notably in quality control through automatic in-line visual inspections. A common practice for inspecting parts, components, and final products is to use a master part benchmark for quality comparison. However, challenges arise when objects enter inspection points in unintended orientations. This misalignment potentially leads to erroneous decisions by automated systems, resulting in additional checkpoints or wastage affecting the production rate. To tackle this issue, we propose a visual inspection pipeline that leverages recent machine learning-based approaches to compare the inspection target and a master part virtually oriented to the same perspective. Specifically, we suggest combining 3D Gaussian Splatting and DUSt3R as a practical solution. Our approach demonstrates its efficacy in real-world scenarios through testing on three mock parts and a real industrial component.

Video analysis has become an essential aspect of net sports, such as badminton. Accurately predicting the future trajectory of balls and shuttlecocks can significantly benefit players by enhancing their performance and enabling them to devise effective game strategies. This paper aims to analyze data to provide players with an advantage in the fast-paced rallies of badminton matches. The paper delves into the innovative task of predicting future shuttlecock trajectories in badminton match videos and presents a method that takes into account both the shuttlecock position and the positions and postures of the players. In the experiments, players were extracted from the match video, their postures were analyzed, and a time-series model was trained. The results indicate that the proposed method improved accuracy by 13% compared to methods that solely used shuttlecock position information as input, and by 8.4% compared to methods that employed both shuttlecock and player position information as input.

Multi-Object Tracking (MOT) plays a critical role in analyzing player behavior from videos, enabling performance evaluation. Current MOT methods are often evaluated using publicly available datasets. However, most of these focus on everyday scenarios such as pedestrian tracking or are tailored to specific sports, including soccer and basketball. Despite the inherent challenges of tracking players in American football, such as frequent occlusion and physical contact, no standardized dataset has been publicly available, making fair comparisons between methods difficult. To address this gap, we constructed the first dedicated detection and tracking dataset for the American football players and conducted a comparative evaluation of various detection and tracking methods. Our results demonstrate that accurate detection and tracking can be achieved even in crowded scenarios. Fine-tuning detection models improved performance over pre-trained models. Furthermore, when these fine-tuned detectors and re-identification models were integrated into tracking systems, we observed notable improvements in tracking accuracy compared to existing approaches. This work thus enables robust detection and tracking of American football players in challenging, high-density scenarios previously underserved by conventional methods.

Ball spin estimation in sports is important for analyzing the game. Since spin is generally too fast to be captured by a conventional camera, a high-speed camera is often used to capture images of the ball and estimate its spin. However, since a high-speed camera is not robust to changes in the lighting conditions, it is difficult to estimate spin in some environments. To solve these problems, this paper proposes a new method for ball spin estimation using an event camera. An event camera is a sensor inspired by the visual system of animals, which outputs the brightness changes in a scene. Event cameras have advantages such as high temporal resolution and high dynamic range, and can accurately capture the motion of a fast-spinning ball in various lighting conditions. Experimental results in a synthesized dataset showed that the proposed method can stably estimate spin up to 500 rps. It is also confirmed that the proposed method can estimate spin in the data obtained from actual sports games.

Motion blur can significantly reduce the quality of images, and researchers have developed various algorithms to address this issue. One common approach to deblurring is to use deconvolution to cancel out the blur effect, but this method is limited by the difficulty of accurately estimating blur kernels from blurred images. This is because the motion causing the blur is often complex and nonlinear. In this paper, a new method for estimating blur kernels is proposed. This method uses an event camera, which captures high-temporal-resolution data on pixel luminance changes, along with a conventional camera to capture the input blurred image. By analyzing the event data stream, the proposed method estimates the 2D motion of the blurred image at short intervals during the exposure time, and integrates this information to estimate a variety of complex blur motions. With the estimated blur kernel, the input blurred image can be deblurred using deconvolution. The proposed method does not rely on machine learning and therefore can restore blurry images without depending on the quality and quantity of training data. Experimental results show that the proposed method can estimate blur kernels even for images blurred by complex camera motions, outperforming conventional methods. Overall, this paper presents a promising approach to motion deblurring that could have practical applications in a range of fields.

When a camera travels across a 3D world, only a fraction of pixel value changes; an event-based camera observes the change as sparse events. How can we utilize sparse events for efficient recovery of the camera pose? We show that we can recover the camera pose by minimizing the error between sparse events and the temporal gradient of the scene represented as a neural radiance field (NeRF). To enable the computation of the temporal gradient of the scene, we augment NeRF's camera pose as a time function. When the input pose to the NeRF coincides with the actual pose, the output of the temporal gradient of NeRF equals the observed intensity changes on the event's points. Using this principle, we propose an event-based camera pose tracking framework called TeGRA which realizes the pose update by using the sparse event's observation. To the best of our knowledge, this is the first camera pose estimation algorithm using the scene's implicit representation and the sparse intensity change from events.