Published papers

Detecting anomalies on the road is crucial for generating hazard maps within factory premises and facilitating navigation for visually impaired individuals or robots. This paper proposes a method for anomaly detection on road surfaces using normal maps and a Long Short-Term Memory (LSTM). While existing research primarily focuses on detecting anomalies on the road based on variations in height or color information of images, our approach leverages anomaly detection to identify changes in the spatial structure of the walking scenario. The normal (non-anomaly) data consists of time series normal maps depicting previously traversed roads, which are utilized to predict the upcoming road conditions. Subsequently, an anomaly score is computed by comparing the predicted normal map with the normal map at t+1. If the anomaly score exceeds a dynamically set threshold, it indicates the presence of anomalies on the road. The proposed method employs unsupervised learning for anomaly detection. To assess the effectiveness of the proposed method, we conducted accuracy assessments using a custom dataset, taking into account a qualitative comparison with the results of existing methods. The results confirm that the proposed method effectively detects anomalies on road surfaces through anomaly detection.

Object handover is a fundamental and essential capability for robots interacting with humans in many applications such as household chores. In this challenge, we estimate the physical properties of a variety of containers with different fillings such as container capacity and the type and percentage of the content to achieve collaborative physical handover between humans and robots. We introduce multi-modal prediction models using audio-visual-datasets of people interacting with containers distributed by CORSMAL.

The ability to recognize and identify terrain characteristics is an essential function required for many autonomous ground robots such as social robots, assistive robots, autonomous vehicles, and ground exploration robots. Recognizing and identifying terrain characteristics is challenging because similar terrains may have very different appearances (e.g., carpet comes in many colors), while terrains with very similar appearance may have very different physical properties (e.g., mulch versus dirt). In order to address the inherent ambiguity in vision-based terrain recognition and identification, we propose a multi-modal self-supervised learning technique that switches between audio features extracted from a microphone attached to the underside of a mobile platform and image features extracted by a camera on the platform to cluster terrain types. The terrain cluster labels are then used to train an image-based real-time CNN (Convolutional Neural Network) to predict terrain types changes. Through experiments, we demonstrate that the proposed self-supervised terrain type recognition method achieves over 80% accuracy, which greatly outperforms several baselines and suggests strong potential for assistive applications.

This position paper introduces the concept of a real-time auditory feedback system designed to promote physical activity and improve well-being through visual skill development in ball sports. The system uses event cameras to translate ball trajectories into intuitive sound cues and complements visual perception with high-temporal-resolution auditory feedback, enabling precise timing and enhanced engagement during practice. Unlike conventional setups requiring wearables or complex hardware, this approach ensures seamless interaction while preserving natural movement. This multimodal approach can potentially offer a scalable and immersive solution for various sports and real-world environments.

Event cameras are bio-inspired sensors that have a high dynamic range and temporal resolution. This property enables motion estimation from textures with repeating patterns, which is difficult to achieve with RGB cameras. Therefore, motion estimation of an event camera is expected to be applied to vehicle position estimation. An existing method, called contrast maximization, is one of the methods that can be used for event camera motion estimation by capturing road surfaces. However, contrast maximization tends to fall into a local solution when estimating three-dimensional motion, which makes correct estimation difficult. To solve this problem, we propose a method for motion estimation by optimizing contrast in the bird’s-eye view space. Instead of performing three-dimensional motion estimation, we reduced the dimensionality to two-dimensional motion estimation by transforming the event data to a bird’s-eye view using homography calculated from the event camera position. This transformation mitigates the problem of the loss function becoming non-convex, which occurs in conventional methods. As a quantitative experiment, we created event data by using a car simulator and evaluated our motion estimation method, showing an improvement in accuracy and speed. In addition, we conducted estimation from real event data and evaluated the results qualitatively, showing an improvement in accuracy.

This paper proposes an application to statistically predict the accuracy of single-image geometric camera calibration that uses given 2D-3D correspondences. Deriving both camera intrinsics and extrinsics from correspondences between a single image and a 3D shape, is important for the scene analysis when the optical system of the camera is lost, such as in the analyses of traffic accidents. It is unclear whether the single-image calibration will be successful in practice, particularly when the number of 2D-3D correspondences is small, even if we could assign accurate correspondences by manual labor. To this end, we perform a systematic evaluation of the camera parameter accuracy using synthetic environments. Based on the statistics observed during the experiments, our application predicts the calibration accuracy from simple variables (e.g., the area that correspondences could be given). Since the prediction process does not rely on 3D shapes, it provides an estimate of the success of the calibration before time-consuming processes, i.e., 3D scanning and 2D-3D correspondence mapping.

Data analysis in sports is becoming increasingly important, and one of the sports in which sports analysts play an active role is volleyball. Volleyball analysts have the task of annotating match videos, a time-consuming and technically challenging task that makes use of data difficult. In this paper, we propose a method for recognizing players' actions from volleyball game videos using time-series heat maps of joint positions to automate the analysis of volleyball match videos. In experiments to verify the effectiveness of the proposed method, we confirmed that the use of time-series heat maps of joint positions improves both the accuracy and F1 score compared to the baseline method using only RGB images as an input. We also confirmed the effectiveness of the proposed method in recognizing players' actions from volleyball match videos, which were not included in the dataset.

We present a workflow for applying the SOTA in multi-view reconstruction over realistic images in the domain of sketches and drawn-like images. With this aim, we leverage NVDiffRec to study its performance over the non-realistic domain through custom use cases. When doing so, we will expose the challenges of using the system over a different domain and present our solutions. Finally, we will detail the obtained results and conclusions on the viability of NVDiffRec as a possible tool for fictional 3D content generation from concept art.

The 8th ACM International Workshop on Multimedia Content Analysis in Sports is held in Dublin, Ireland on October 28th, 2025. It is co-located with ACM Multimedia 2025. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing the multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation as well as understanding, statistical analysis, and evaluation in amateur and professional sports. There is a lack of research communities focusing on the fusion of multiple modalities. Thus, this workshop series on multimedia content analysis in sports aims to contribute to the closure of this research gap by bringing together the breadth and depth of these diverse approaches to stimulate each other with new ideas and foster research progress.

We are developing a service platform to achieve use cases such as finding people in trouble and sharing them with AR applications to ask for help from the people around them, finding information about a sale at a store and sharing it with a map service, or alerting the police when a crime occurs, as well as quickly alerting the people around them. Our system detects “objects” and “events” in the real world using edge devices that can share the detected information in cyberspace so that the users can remotely obtain real-world information from distant locations. We believe our system can make life more safe and convenient. In this paper, we report on the findings from our experiments on searching for lost dogs by mapping object recognition results, on mapping bird habitat areas by environmental sound recognition results, and on mapping public facility announcement voices by speech recognition results.

We are developing a system that detects "objects"and "events"in the real-world using edge devices that can share the detected information among them in the cyber-space (virtual world) so that the users can remotely obtain real-world information from distant locations. We aim for the system can serve as a foundation for services that construct and utilize a digital twin of the real world. In this paper, we report our findings by implementing "objects"and "things"detection in edge devices (smartphones) and making experiments for sharing the detected "objects"and "things"registered on a digital map.

Trajectory prediction has gained great attention and significant progress has been made in recent years. However, most works rely on a key assumption that each video is successfully preprocessed by detection and tracking algorithms and the complete observed trajectory is always available. However, in complex real-world environments, we often encounter miss-detection of target agents (e.g., pedestrian, vehicles) caused by the bad image conditions, such as the occlusion by other agents. In this paper, we address the problem of trajectory prediction from incomplete observed trajectory due to miss-detection, where the observed trajectory includes several missing data points. We introduce a two-block RNN model that approximates the inference steps of the Bayesian filtering framework and seeks the optimal estimation of the hidden state when miss-detection occurs. The model uses two RNNs depending on the detection result. One RNN approximates the inference step of the Bayesian filter with the new measurement when the detection succeeds, while the other does the approximation when the detection fails. Our experiments show that the proposed model improves the prediction accuracy compared to the three baseline imputation methods on publicly available datasets: ETH and UCY (9% and 7% improvement on the ADE and FDE metrics). We also show that our proposed method can achieve better prediction compared to the baselines when there is no miss-detection.

Machine vision systems are crucial for quality control in manufacturing, ensuring that products meet standards through automatic in-line visual inspections. While reference images are typically used as benchmarks for comparison, a significant challenge arises when objects arrive at inspection points in misaligned orientations. This misalignment can lead to erroneous decisions by automated systems, resulting in false positives or negatives that increase waste and slow production. To address this issue, we propose a visual inspection pipeline that leverages recent machine-learning-based approaches. Our pipeline virtually reorients a 3D reference to the inspection target's orientation for comparison, effectively addressing the misalignment issue. Specifically, it integrates 3D Gaussian Splatting and MASt3R, enabling a robust 3D-based defect detection system that uses only a single camera sensor, thereby moving beyond traditional 2D image-based methods. We validated our approach in real-world scenarios by testing it on 20 synthetic objects and real industrial parts. The proposed pipeline maintains accuracy and processing speed comparable to those of existing methods.

Electrical cables consist of numerous wires. The 3D shape of individual wires in the cables significantly affects their characteristics. With the increasing diversity of wire structures, understanding the 3D shape of each wire in an electrical cable is crucial for analyzing characteristics such as bending stiffness. To accurately estimate the bending stiffness of actual electrical cables, a detailed 3D representation of each wire is required. Therefore, it is important to obtain the 3D shape of actual electrical cable wires using non-destructive inspection. In this study, we propose a new method to associate wires using particle tracking techniques. The proposed method performs wire tracking in two steps, linking detected wire positions between adjacent frames and connecting segmented wires across frames. The effectiveness of the proposed method was evaluated quantitatively and qualitatively. In the quantitative evaluation, the correctness of the tracking was evaluated using the data with artificial noise added to the real data detection results. The proposed method achieves highly accurate wire tracking when there is little undetected noise, and shows robustness to over-detection.

We present a method for synchronizing three-dimensional (3D) point cloud from 3D scene with estimation using a 3D Lidar and an RGB camera. These 3D points sensed by the 3D Lidar are not captured at the same time, which makes it difficult to measure the correct shape of the object in a dynamic scene. In our method, we generate synchronized 3D points at arbitrary times using linear interpolation in four-dimensional space, time and space. For interpolating the 3D point, we obtain corresponding 3D point matching with the pixel value captured by the RGB camera in a continuous frame. The experimental results demonstrate the effectiveness of the presented method by depicting a synchronized 3D point cloud that is correctly shaped.

We present a novel 3D reconstruction system that can generate a stable triangle mesh using data from multiple RGB-D sensors in real time for dynamic scenes. The first part of the system uses moving least squares (MLS) point set surfaces to smooth and filter point clouds acquired from RGB-D sensors. The second part of the system generates triangle meshes from point clouds. The whole pipeline is executed on the GPU and is tailored to scale linearly with the size of the input data. Our contributions include changes to the MLS method for improving meshing, a fast triangle mesh generation method and GPU implementations of all parts of the pipeline.

© 2018 IEEE. We propose an efficient and scalable method for incrementally building a dense, semantically annotated 3D map in real-time. The proposed method assigns class probabilities to each region, not each element (e.g., surfel and voxel), of the 3D map which is built up through a robust SLAM framework and incrementally segmented with a geometric-based segmentation method. Differently from all other approaches, our method has a capability of running at over 30Hz while performing all processing components, including SLAM, segmentation, 2D recognition, and updating class probabilities of each segmentation label at every incoming frame, thanks to the high efficiency that characterizes the computationally intensive stages of our framework. By utilizing a specifically designed CNN to improve the frame-wise segmentation result, we can also achieve high accuracy. We validate our method on the NYUv2 dataset by comparing with the state of the art in terms of accuracy and computational efficiency, and by means of an analysis in terms of time and space complexity.

© 2018, The Author(s). Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.

© 2019 Elsevier B.V. Background: An individual's gait is a key factor for consideration in evaluating their overall health. Several medical studies have demonstrated the correlation between gait and incidence rate of diseases, mortality, and risk of fall. However, gait is only occasionally evaluated during medical visits, which may delay the detection of health problems. Methods: In this paper, we propose a gait measurement system that is suitable for use at home. Our method requires only a single RGB camera, whereas other visionary sensor-based methods require depth sensors or multiple RGB cameras. In addition, the setup for the measurement is easy. What the user has to do is only putting a single camera in a room and choosing four location known points on the floor. Our method can measure step positions and step timings, and therefore, other important parameters such as stride length, step width, walking speed, and cadence may also be captured. The individual's gait is captured by the camera, and therefore the user is not required to wear any devices. Results: In the experiment described herein, we demonstrate our method's accuracy by comparing it with the motion capture system. The results indicate that our method can measure walking speed with an error of 3.62 cm/s from the side view, and which is too small a change to be clinically meaningful.

© Springer Nature Switzerland AG 2020. Real-time camera pose estimation is one of the indispensable technologies for Augmented Reality (AR). While a large body of work in Visual Odometry (VO) has been proposed for AR, practical challenges such as scale ambiguities and accumulative errors still remain especially when we apply VO to large-scale scenes due to limited hardware and resources. We propose a camera pose registration method, where a local VO is consecutively optimized with respect to a large-scale scene map on the fly. This framework enables the scale estimation between a VO map and a scene map and reduces accumulative errors by finding corresponding locations in the map to the current frame and by on-the-fly pose graph optimization. The results using public datasets demonstrated that our approach reduces the accumulative errors of naïve VO.