Published papers

There has been a substantial amount of research involving computer methods and technology for the detection and recognition of diabetic foot ulcers (DFUs), but there is a lack of systematic comparisons of state-of-the-art deep learning object detection frameworks applied to this problem. DFUC2020 provided participants with a comprehensive dataset consisting of 2,000 images for training and 2,000 images for testing. This paper summarizes the results of DFUC2020 by comparing the deep learning-based algorithms proposed by the winning teams: Faster R–CNN, three variants of Faster R–CNN and an ensemble method; YOLOv3; YOLOv5; EfficientDet; and a new Cascade Attention Network. For each deep learning method, we provide a detailed description of model architecture, parameter settings for training and additional stages including pre-processing, data augmentation and post-processing. We provide a comprehensive evaluation for each method. All the methods required a data augmentation stage to increase the number of images available for training and a post-processing stage to remove false positives. The best performance was obtained from Deformable Convolution, a variant of Faster R–CNN, with a mean average precision (mAP) of 0.6940 and an F1-Score of 0.7434. Finally, we demonstrate that the ensemble method based on different deep learning methods can enhance the F1-Score but not the mAP.

We propose a method for dense depth estimation from an event stream generated when sweeping the focal plane of the driving lens attached to an event camera. In this method, a depth map is inferred from an 'event focal stack' composed of the event stream using a convolutional neural network trained with synthesized event focal stacks. The synthesized event stream is created from a focal stack generated by Blender for any arbitrary 3D scene. This allows for training on scenes with diverse structures. Additionally, we explored methods to eliminate the domain gap between real event streams and synthetic event streams. Our method demonstrates superior performance over a depth-from-defocus method in the image domain on synthetic and real datasets.

This paper addresses the task of class-imbalanced medical image classification using complication labels. Medical image classification plays an essential role in improving patient care and reducing the burden on healthcare systems, along with other computer aided diagnosis tasks. However, there are two common obstacles with medical image image classification. One is that there is a high cost of acquiring annotations from doctors/experts. Two is that in many cases there is a differences in the number of incidents that appear per symptom which lead to class-imbalance. The above factors make medical image classification difficult to deal with. Additionally, there are cases where multiple symptoms appear in a mixed manner (complications), such as diabetic complications where both symptoms of infection and ischaemia could appear at the same time, which also could influence the classification result negatively. In this paper, a method that jointly tackles the obstacles mentioned above is presented. First, we conduct extensive experiment with various class-imbalanced learning methods introduced in previous works and also propose a method that improves the baseline class-imbalanced approaches by utilizing complication labels in pre-training. Second, we validate the proposed method with the diabetic foot ulcer dataset introduced in Diabetic Foot Ulcer Challenge 2021.

A recent singleshot multiplane image (MPI) generation enables to copy an observed reality within a camera frame into other reality domains via view synthesis. While the scene scale is unknown due to the nature of singleshot MPI processing, camera tracking algorithms can estimate depth within the application world coordinate system. Given such depth information, we propose to adjust the scale of singleshot MPI to that of the currently observed scene. We find the individual scales of the MPI layers by minimizing the differences between the depth of MPI rendering and that of camera tracking. We eventually found that many layers fall within a close depth. Therefore, we merge such layers into one to compact the MPI representation. We compared our method with baselines using real and synthetic datasets with dense and sparse depth inputs. Our results demonstrate that our algorithm achieves higher scores in image metrics and reduces MPI data amount by up to 78%.

In equestrian competitions, the precision of horse movements significantly impacts scoring. Accurate recognition and analysis of these movements can enhance the scoring process. Existing action recognition methods for sports are primarily designed for humans, presenting challenges when applied to equestrian sports where horses are the main subjects. Additionally, research on animal behavior has been largely confined to behavior recognition. This paper evaluates and compares the advantages and limitations of different modalities by constructing a dataset of three distinct modalities specific to equestrian sports and applying existing human action recognition techniques. Our findings indicate that optical flow is the most practical modality. Furthermore, combining the three modalities achieved a motion recognition accuracy of 0.97. By removing error-prone data—such as instances where the horse is facing forward or backward, or when obstacles are present around the horse’s limbs—the action recognition accuracy increased to 0.99. The dataset is available at https://github.com/kotokamatsunaga/horsedataset.git.

In recent years, abnormal human activity detection has become an important research topic. However, most existing methods focus on detecting abnormal activities of pedestrians in surveillance videos; even those methods using egocentric videos deal with the activities of pedestrians around the camera wearer. In this paper, the authors present an unsupervised auto-encoder-based network trained by one-class learning that inputs RGB image sequences recorded by egocentric cameras to detect abnormal activities of the camera wearers themselves. To improve the performance of network, the authors introduce a ‘re-encoding’ architecture and a regularisation loss function term, minimising the KL divergence between the distributions of features extracted by the first and second encoders. Unlike the common use of KL divergence loss to obtain a feature distribution close to an already-known distribution, the aim is to encourage the features extracted by the second encoder to have a close distribution to those extracted from the first encoder. The authors evaluate the proposed method on the Epic-Kitchens-55 dataset and conduct an ablation study to analyse the functions of different components. Experimental results demonstrate that the method outperforms the comparison methods in all cases and demonstrate the effectiveness of the proposed re-encoding architecture and the regularisation term.

Recording surgery is an important technique for education and the evaluation of medical treatments. However, capturing targets such as the surgical field, surgical tools, and the surgeon's hands, is almost impossible since these targets are heavily occluded by the surgeon's head and body during a surgery. We used a recording system in which multiple cameras are installed in the surgical lump, supposing at least one camera would capture the target without occlusion. As this system records multiple video sequences, we address the task to select a best view camera automatically. Recently, learning-based approaches in a fully supervised manner have been proposed for this task, but these previous approaches completely rely on manual annotation of the training data. In this paper, we focus on the eye tracker mounted on the surgeon's head, which can capture the recording targets without occlusion. Employing this first-person-view video synchronized with multiple videos of the surgical lump, we propose a novel camera selection approach using a self-supervised learning framework. In experiments, we created a dataset composed of four different breast surgery. Our extended experiments showed that our approach successfully switched to the best camera view without manual annotation and achieved competitive accuracy compared with conventional supervised methods. Also, our approach yielded effective visual representations comparable to state-of-the-art self-supervised learning frameworks.

Study Design.Cross-sectional study.Objective.To develop a binary classification model for cervical myelopathy (CM) screening based on a machine learning algorithm using Leap Motion (Leap Motion, San Francisco, CA), a novel noncontact sensor device.Summary of Background Data.Progress of CM symptoms are gradual and cannot be easily identified by the patients themselves. Therefore, screening methods should be developed for patients of CM before deterioration of myelopathy. Although some studies have been conducted to objectively evaluate hand movements specific to myelopathy using cameras or wearable sensors, their methods are unsuitable for simple screening outside hospitals because of the difficulty in obtaining and installing their equipment and the long examination time.Methods.In total, 50 and 28 participants in the CM and control groups were recruited, respectively. The diagnosis of CM was made by spine surgeons. We developed a desktop system using Leap Motion that recorded 35 parameters of fingertip movements while participants gripped and released their fingers as rapidly as possible. A support vector machine was used to develop the binary classification model, and a multiple linear regression analysis was performed to create regression models to estimate the total Japanese Orthopaedic Association (JOA) score and the JOA score of the motor function of the upper extremity (MU-JOA score).Results.The binary classification model indexes were as follows: sensitivity, 84.0%; specificity, 60.7%; accuracy, 75.6%; area under the curve, 0.85. The Spearman rank correlation coefficient between the estimated score and the total JOA score was 0.44 and that between the estimated score and the MU-JOA score was 0.51.Conclusion.Our binary classification model using a machine learning algorithm and Leap Motion could classify CM with high sensitivity and would be useful for CM screening in daily life before consulting doctors and telemedicine.Level of Evidence: 3.

Metaverse has been attracting more and more attention because of its potential for various use cases. In metaverse applications, the seamless integration of digital and physical worlds is vital for synchronizing information from one world to another. One way to achieve this is to reconstruct 3D environmental maps every time, which is not feasible due to computational complexity. A cheaper alternative is to detect what objects have changed and update only the changed objects. To build the foundation of the change detection algorithm for that, in this paper, we propose a change detection method combined with object classification. Despite its simplicity, the experiment showed promising results with an object detector fine-tuned with data from the target environment. Furthermore, with our clustering-based post-processing, false positives produced by the frame-wise change detection were observed to be successfully suppressed.

Detecting anomalies on the road is crucial for generating hazard maps within factory premises and facilitating navigation for visually impaired individuals or robots. This paper proposes a method for anomaly detection on road surfaces using normal maps and a Long Short-Term Memory (LSTM). While existing research primarily focuses on detecting anomalies on the road based on variations in height or color information of images, our approach leverages anomaly detection to identify changes in the spatial structure of the walking scenario. The normal (non-anomaly) data consists of time series normal maps depicting previously traversed roads, which are utilized to predict the upcoming road conditions. Subsequently, an anomaly score is computed by comparing the predicted normal map with the normal map at t+1. If the anomaly score exceeds a dynamically set threshold, it indicates the presence of anomalies on the road. The proposed method employs unsupervised learning for anomaly detection. To assess the effectiveness of the proposed method, we conducted accuracy assessments using a custom dataset, taking into account a qualitative comparison with the results of existing methods. The results confirm that the proposed method effectively detects anomalies on road surfaces through anomaly detection.

Object handover is a fundamental and essential capability for robots interacting with humans in many applications such as household chores. In this challenge, we estimate the physical properties of a variety of containers with different fillings such as container capacity and the type and percentage of the content to achieve collaborative physical handover between humans and robots. We introduce multi-modal prediction models using audio-visual-datasets of people interacting with containers distributed by CORSMAL.

The ability to recognize and identify terrain characteristics is an essential function required for many autonomous ground robots such as social robots, assistive robots, autonomous vehicles, and ground exploration robots. Recognizing and identifying terrain characteristics is challenging because similar terrains may have very different appearances (e.g., carpet comes in many colors), while terrains with very similar appearance may have very different physical properties (e.g., mulch versus dirt). In order to address the inherent ambiguity in vision-based terrain recognition and identification, we propose a multi-modal self-supervised learning technique that switches between audio features extracted from a microphone attached to the underside of a mobile platform and image features extracted by a camera on the platform to cluster terrain types. The terrain cluster labels are then used to train an image-based real-time CNN (Convolutional Neural Network) to predict terrain types changes. Through experiments, we demonstrate that the proposed self-supervised terrain type recognition method achieves over 80% accuracy, which greatly outperforms several baselines and suggests strong potential for assistive applications.

This position paper introduces the concept of a real-time auditory feedback system designed to promote physical activity and improve well-being through visual skill development in ball sports. The system uses event cameras to translate ball trajectories into intuitive sound cues and complements visual perception with high-temporal-resolution auditory feedback, enabling precise timing and enhanced engagement during practice. Unlike conventional setups requiring wearables or complex hardware, this approach ensures seamless interaction while preserving natural movement. This multimodal approach can potentially offer a scalable and immersive solution for various sports and real-world environments.

Event cameras are bio-inspired sensors that have a high dynamic range and temporal resolution. This property enables motion estimation from textures with repeating patterns, which is difficult to achieve with RGB cameras. Therefore, motion estimation of an event camera is expected to be applied to vehicle position estimation. An existing method, called contrast maximization, is one of the methods that can be used for event camera motion estimation by capturing road surfaces. However, contrast maximization tends to fall into a local solution when estimating three-dimensional motion, which makes correct estimation difficult. To solve this problem, we propose a method for motion estimation by optimizing contrast in the bird’s-eye view space. Instead of performing three-dimensional motion estimation, we reduced the dimensionality to two-dimensional motion estimation by transforming the event data to a bird’s-eye view using homography calculated from the event camera position. This transformation mitigates the problem of the loss function becoming non-convex, which occurs in conventional methods. As a quantitative experiment, we created event data by using a car simulator and evaluated our motion estimation method, showing an improvement in accuracy and speed. In addition, we conducted estimation from real event data and evaluated the results qualitatively, showing an improvement in accuracy.

This paper proposes an application to statistically predict the accuracy of single-image geometric camera calibration that uses given 2D-3D correspondences. Deriving both camera intrinsics and extrinsics from correspondences between a single image and a 3D shape, is important for the scene analysis when the optical system of the camera is lost, such as in the analyses of traffic accidents. It is unclear whether the single-image calibration will be successful in practice, particularly when the number of 2D-3D correspondences is small, even if we could assign accurate correspondences by manual labor. To this end, we perform a systematic evaluation of the camera parameter accuracy using synthetic environments. Based on the statistics observed during the experiments, our application predicts the calibration accuracy from simple variables (e.g., the area that correspondences could be given). Since the prediction process does not rely on 3D shapes, it provides an estimate of the success of the calibration before time-consuming processes, i.e., 3D scanning and 2D-3D correspondence mapping.

Data analysis in sports is becoming increasingly important, and one of the sports in which sports analysts play an active role is volleyball. Volleyball analysts have the task of annotating match videos, a time-consuming and technically challenging task that makes use of data difficult. In this paper, we propose a method for recognizing players' actions from volleyball game videos using time-series heat maps of joint positions to automate the analysis of volleyball match videos. In experiments to verify the effectiveness of the proposed method, we confirmed that the use of time-series heat maps of joint positions improves both the accuracy and F1 score compared to the baseline method using only RGB images as an input. We also confirmed the effectiveness of the proposed method in recognizing players' actions from volleyball match videos, which were not included in the dataset.

In the visual inspection of industrial products using a microscope with a shallow depth of field, it is difficult to capture an all-in-focus image. This paper presents a method to generate an all-in-focus image from a large focal stack, which can be applied to visual inspection. The proposed method adds two improvements to a deep learning method [M. Maximov et al., CVPR, (2020) 1068] that leverages defocus cues in depth estimation. First, it interpolates an index map (a collection of in-focus image indices at all pixels) after the forward pass. It generates an all-in-focus image using all images in focal stacks while reducing the number of images input to the network. Second, the network is trained with synthetic datasets to which a random texture with high-frequency components is added. It helps the network to learn the degree of defocus blur. Experiments using synthetic images and real microscope images show that interpolating the index map and adding texture improves the accuracy of all-in-focus image generation.

The 8th ACM International Workshop on Multimedia Content Analysis in Sports is held in Dublin, Ireland on October 28th, 2025. It is co-located with ACM Multimedia 2025. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing the multimodal data in sports. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation as well as understanding, statistical analysis, and evaluation in amateur and professional sports. There is a lack of research communities focusing on the fusion of multiple modalities. Thus, this workshop series on multimedia content analysis in sports aims to contribute to the closure of this research gap by bringing together the breadth and depth of these diverse approaches to stimulate each other with new ideas and foster research progress.

We are developing a service platform to achieve use cases such as finding people in trouble and sharing them with AR applications to ask for help from the people around them, finding information about a sale at a store and sharing it with a map service, or alerting the police when a crime occurs, as well as quickly alerting the people around them. Our system detects “objects” and “events” in the real world using edge devices that can share the detected information in cyberspace so that the users can remotely obtain real-world information from distant locations. We believe our system can make life more safe and convenient. In this paper, we report on the findings from our experiments on searching for lost dogs by mapping object recognition results, on mapping bird habitat areas by environmental sound recognition results, and on mapping public facility announcement voices by speech recognition results.

We are developing a system that detects "objects"and "events"in the real-world using edge devices that can share the detected information among them in the cyber-space (virtual world) so that the users can remotely obtain real-world information from distant locations. We aim for the system can serve as a foundation for services that construct and utilize a digital twin of the real world. In this paper, we report our findings by implementing "objects"and "things"detection in edge devices (smartphones) and making experiments for sharing the detected "objects"and "things"registered on a digital map.