Published papers

In recent years, abnormal human activity detection has become an important research topic. However, most existing methods focus on detecting abnormal activities of pedestrians in surveillance videos; even those methods using egocentric videos deal with the activities of pedestrians around the camera wearer. In this paper, the authors present an unsupervised auto-encoder-based network trained by one-class learning that inputs RGB image sequences recorded by egocentric cameras to detect abnormal activities of the camera wearers themselves. To improve the performance of network, the authors introduce a ‘re-encoding’ architecture and a regularisation loss function term, minimising the KL divergence between the distributions of features extracted by the first and second encoders. Unlike the common use of KL divergence loss to obtain a feature distribution close to an already-known distribution, the aim is to encourage the features extracted by the second encoder to have a close distribution to those extracted from the first encoder. The authors evaluate the proposed method on the Epic-Kitchens-55 dataset and conduct an ablation study to analyse the functions of different components. Experimental results demonstrate that the method outperforms the comparison methods in all cases and demonstrate the effectiveness of the proposed re-encoding architecture and the regularisation term.

This paper proposes an application to statistically predict the accuracy of single-image geometric camera calibration that uses given 2D-3D correspondences. Deriving both camera intrinsics and extrinsics from correspondences between a single image and a 3D shape, is important for the scene analysis when the optical system of the camera is lost, such as in the analyses of traffic accidents. It is unclear whether the single-image calibration will be successful in practice, particularly when the number of 2D-3D correspondences is small, even if we could assign accurate correspondences by manual labor. To this end, we perform a systematic evaluation of the camera parameter accuracy using synthetic environments. Based on the statistics observed during the experiments, our application predicts the calibration accuracy from simple variables (e.g., the area that correspondences could be given). Since the prediction process does not rely on 3D shapes, it provides an estimate of the success of the calibration before time-consuming processes, i.e., 3D scanning and 2D-3D correspondence mapping.

Data analysis in sports is becoming increasingly important, and one of the sports in which sports analysts play an active role is volleyball. Volleyball analysts have the task of annotating match videos, a time-consuming and technically challenging task that makes use of data difficult. In this paper, we propose a method for recognizing players' actions from volleyball game videos using time-series heat maps of joint positions to automate the analysis of volleyball match videos. In experiments to verify the effectiveness of the proposed method, we confirmed that the use of time-series heat maps of joint positions improves both the accuracy and F1 score compared to the baseline method using only RGB images as an input. We also confirmed the effectiveness of the proposed method in recognizing players' actions from volleyball match videos, which were not included in the dataset.

In the visual inspection of industrial products using a microscope with a shallow depth of field, it is difficult to capture an all-in-focus image. This paper presents a method to generate an all-in-focus image from a large focal stack, which can be applied to visual inspection. The proposed method adds two improvements to a deep learning method [M. Maximov et al., CVPR, (2020) 1068] that leverages defocus cues in depth estimation. First, it interpolates an index map (a collection of in-focus image indices at all pixels) after the forward pass. It generates an all-in-focus image using all images in focal stacks while reducing the number of images input to the network. Second, the network is trained with synthetic datasets to which a random texture with high-frequency components is added. It helps the network to learn the degree of defocus blur. Experiments using synthetic images and real microscope images show that interpolating the index map and adding texture improves the accuracy of all-in-focus image generation.

Detecting anomalies on the road is crucial for generating hazard maps within factory premises and facilitating navigation for visually impaired individuals or robots. This paper proposes a method for anomaly detection on road surfaces using normal maps and a Long Short-Term Memory (LSTM). While existing research primarily focuses on detecting anomalies on the road based on variations in height or color information of images, our approach leverages anomaly detection to identify changes in the spatial structure of the walking scenario. The normal (non-anomaly) data consists of time series normal maps depicting previously traversed roads, which are utilized to predict the upcoming road conditions. Subsequently, an anomaly score is computed by comparing the predicted normal map with the normal map at t+1. If the anomaly score exceeds a dynamically set threshold, it indicates the presence of anomalies on the road. The proposed method employs unsupervised learning for anomaly detection. To assess the effectiveness of the proposed method, we conducted accuracy assessments using a custom dataset, taking into account a qualitative comparison with the results of existing methods. The results confirm that the proposed method effectively detects anomalies on road surfaces through anomaly detection.

We are developing a service platform to achieve use cases such as finding people in trouble and sharing them with AR applications to ask for help from the people around them, finding information about a sale at a store and sharing it with a map service, or alerting the police when a crime occurs, as well as quickly alerting the people around them. Our system detects “objects” and “events” in the real world using edge devices that can share the detected information in cyberspace so that the users can remotely obtain real-world information from distant locations. We believe our system can make life more safe and convenient. In this paper, we report on the findings from our experiments on searching for lost dogs by mapping object recognition results, on mapping bird habitat areas by environmental sound recognition results, and on mapping public facility announcement voices by speech recognition results.

We are developing a system that detects "objects"and "events"in the real-world using edge devices that can share the detected information among them in the cyber-space (virtual world) so that the users can remotely obtain real-world information from distant locations. We aim for the system can serve as a foundation for services that construct and utilize a digital twin of the real world. In this paper, we report our findings by implementing "objects"and "things"detection in edge devices (smartphones) and making experiments for sharing the detected "objects"and "things"registered on a digital map.

Trajectory prediction has gained great attention and significant progress has been made in recent years. However, most works rely on a key assumption that each video is successfully preprocessed by detection and tracking algorithms and the complete observed trajectory is always available. However, in complex real-world environments, we often encounter miss-detection of target agents (e.g., pedestrian, vehicles) caused by the bad image conditions, such as the occlusion by other agents. In this paper, we address the problem of trajectory prediction from incomplete observed trajectory due to miss-detection, where the observed trajectory includes several missing data points. We introduce a two-block RNN model that approximates the inference steps of the Bayesian filtering framework and seeks the optimal estimation of the hidden state when miss-detection occurs. The model uses two RNNs depending on the detection result. One RNN approximates the inference step of the Bayesian filter with the new measurement when the detection succeeds, while the other does the approximation when the detection fails. Our experiments show that the proposed model improves the prediction accuracy compared to the three baseline imputation methods on publicly available datasets: ETH and UCY (9% and 7% improvement on the ADE and FDE metrics). We also show that our proposed method can achieve better prediction compared to the baselines when there is no miss-detection.

Event cameras are bio-inspired sensors that have a high dynamic range and temporal resolution. This property enables motion estimation from textures with repeating patterns, which is difficult to achieve with RGB cameras. Therefore, motion estimation of an event camera is expected to be applied to vehicle position estimation. An existing method, called contrast maximization, is one of the methods that can be used for event camera motion estimation by capturing road surfaces. However, contrast maximization tends to fall into a local solution when estimating three-dimensional motion, which makes correct estimation difficult. To solve this problem, we propose a method for motion estimation by optimizing contrast in the bird’s-eye view space. Instead of performing three-dimensional motion estimation, we reduced the dimensionality to two-dimensional motion estimation by transforming the event data to a bird’s-eye view using homography calculated from the event camera position. This transformation mitigates the problem of the loss function becoming non-convex, which occurs in conventional methods. As a quantitative experiment, we created event data by using a car simulator and evaluated our motion estimation method, showing an improvement in accuracy and speed. In addition, we conducted estimation from real event data and evaluated the results qualitatively, showing an improvement in accuracy.

Electrical cables consist of numerous wires. The 3D shape of individual wires in the cables significantly affects their characteristics. With the increasing diversity of wire structures, understanding the 3D shape of each wire in an electrical cable is crucial for analyzing characteristics such as bending stiffness. To accurately estimate the bending stiffness of actual electrical cables, a detailed 3D representation of each wire is required. Therefore, it is important to obtain the 3D shape of actual electrical cable wires using non-destructive inspection. In this study, we propose a new method to associate wires using particle tracking techniques. The proposed method performs wire tracking in two steps, linking detected wire positions between adjacent frames and connecting segmented wires across frames. The effectiveness of the proposed method was evaluated quantitatively and qualitatively. In the quantitative evaluation, the correctness of the tracking was evaluated using the data with artificial noise added to the real data detection results. The proposed method achieves highly accurate wire tracking when there is little undetected noise, and shows robustness to over-detection.

We present a workflow for applying the SOTA in multi-view reconstruction over realistic images in the domain of sketches and drawn-like images. With this aim, we leverage NVDiffRec to study its performance over the non-realistic domain through custom use cases. When doing so, we will expose the challenges of using the system over a different domain and present our solutions. Finally, we will detail the obtained results and conclusions on the viability of NVDiffRec as a possible tool for fictional 3D content generation from concept art.

The 8th ACM International Workshop on Multimedia Content Analysis in Sports is held in Dublin, Ireland on October 28th, 2025. It is co-located with ACM Multimedia 2025. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing the multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation as well as understanding, statistical analysis, and evaluation in amateur and professional sports. There is a lack of research communities focusing on the fusion of multiple modalities. Thus, this workshop series on multimedia content analysis in sports aims to contribute to the closure of this research gap by bringing together the breadth and depth of these diverse approaches to stimulate each other with new ideas and foster research progress.

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.

Surgical tool detection is essential for analyzing and evaluating minimally invasive surgery videos. Current approaches are mostly based on supervised methods that require large, fully instance-level labels (i.e., bounding boxes). However, large image datasets with instance-level labels are often limited because of the burden of annotation. Thus, surgical tool detection is important when providing image-level labels instead of instance-level labels since image-level annotations are considerably more time-efficient than instance-level annotations. In this work, we propose to strike a balance between the extremely costly annotation burden and detection performance. We further propose a co-occurrence loss, which considers a characteristic that some tool pairs often co-occur together in an image to leverage image-level labels. Encapsulating the knowledge of co-occurrence using the co-occurrence loss helps to overcome the difficulty in classification that originates from the fact that some tools have similar shapes and textures. Extensive experiments conducted on the Endovis2018 dataset in various data settings show the effectiveness of our method.

Reliably and accurately forecasting future trajectories of pedestrians is necessary for systems like autonomous vehicles or visual assistive devices to function correctly. While previous state-of-the-art methods relied on modeling social interactions with LSTMs, with videos captured with a static camera from a bird’s-eye view, our paper presents a new method that leverages the Transformers architecture and offers a reliable way to model future trajectories in first-person videos captured by a body-mounted camera, without having to model any social interactions. Accurately forecasting future trajectories is a challenging task, mainly due to how unpredictably humans move. We tackle this issue by using information about target persons’ previous locations, scales, and dynamic poses, as well as information about the camera wearer’s ego-motion. The model we propose predicts future trajectories in a simple way, modeling each target’s trajectory separately, without the use of complex social interactions between humans or interactions between targets and the scene. Experimental results show that our method overall outperforms previous state-of-the-art methods, and yields better results in challenging situations where previous state-of-the-art methods fail.

Automatic fall detection is a crucial task in healthcare as falls pose a significant risk to the health of elderly individuals. This paper presents a lightweight acceleration-based fall detection method that can be implemented on edge devices. The proposed method uses Autoencoders, a type of unsupervised learning, within the framework of anomaly detection, allowing for network training without requiring extensive labeled fall data. One of the challenges in fall detection is the difficulty in collecting fall data. However, our proposed method can overcome this limitation by training the neural network without fall data, using the anomaly detection framework of Autoencoders. Additionally, this method employs an extremely lightweight Autoencoder that can run independently on an edge device, eliminating the need to transmit data to a server and minimizing privacy concerns. We conducted experiments comparing the performance of our proposed method with that of a baseline method using a unique fall detection dataset. Our results confirm that our method outperforms the baseline method in detecting falls with higher accuracy.

Augmented reality (AR) provides ways to visualize missing view samples for novel view synthesis. Existing approaches present 3D annotations for new view samples and task users with taking images by aligning the AR display. This data collection task is known to be mentally demanding and limits capture areas to pre-defined small areas due to the ideal but restrictive underlying sampling theory. To free users from 3D annotations and limited scene exploration, we propose using locally reconstructed light fields and visualizing errors to be removed by inserting new views. Our results show that the error-peaking visualization is less invasive, reduces disappointment in final results, and is satisfactory with fewer view samples in our mobile view synthesis system. We also show that our approach can contribute to recent radiance field reconstruction for larger scenes, such as 3D Gaussian splatting.

Cervical myelopathy (CM) is a pathology of the spinal cord that causes upper limb disorders. CM is often screened by conducting the 10-s grip and release (G&R) test, which mainly focuses on hand dysfunction caused by CM. This test has patients repeat gripping and releasing their hands as quickly as possible. Spine surgeons observe the quickness of this repetition to screen for CM. We propose an automatic screening method of CM that involves patients’ hands recorded as videos when they are performing the G&R test. The videos are used to estimate feature values, i.e., the positions of each part of the hand, which are obtained through image processing. A support vector machine classifier classifies CM patients and controls with these feature values after pre-processing. We validated our method with 10-fold cross-validation and the videos of 20 CM patients and 15 controls. The results indicate that sensitivity, specificity, and area under the receiver operating characteristic curve were 90.0 %, 93.3 %, and 0.947, respectively.

Electrical cables consist of numerous wires, the three-dimensional (3D) shape of which significantly impacts the cables' overall properties, such as bending stiffness. Although X-ray computed tomography (CT) provides a non-destructive method to assess these properties, accurately determining the 3D shape of individual wires from CT images is challenging due to the large number of wires, low image resolution, and indistinguishable appearance of the wires. Previous research lacked quantitative evaluation for wire tracking, and its overall accuracy heavily relied on the accuracy of wire detection. In this study, we present a long short-term memory-based approach for wire tracking that improves robustness against detection errors. The proposed method predicts wire positions in subsequent frames based on previous frames. We evaluate the performance of the proposed method using both actual annotated cables and artificially noised annotations. Our method exhibits greater tracking accuracy and robustness to detection errors compared with the previous method.

Ball tracking based on sports videos captured by high-speed cameras has had limited applicability due to the need for sufficient lighting and the necessity of abundant computational resources to process the large amount of captured data. To overcome these problems, this paper addresses ball tracking using an event camera. An event camera outputs luminance changes in a scene as "events" and is characterized by high temporal resolution, high dynamic range, and data efficiency. The proposed method selects the ball position from the bounding boxes output by the object detector by taking into account the ball motion estimated from the events and the past ball positions using Contrast Maximization (CMax). Considering past ball positions enables us to estimate the ball trajectory consistently over time and detect the ball position more accurately. Additionally, CMax is applied to the events within the tracked ball bounding box, allowing for the estimation of the ball spin. Experimental results in this paper show that the proposed method is quantitatively and qualitatively more accurate than conventional ball tracking methods and demonstrates the potential ability to estimate ball spin.