Published papers

In many sports events, replay footage is commonly used to enhance audience engagement. However, the process of selecting and editing these replays is typically manual, placing a significant operational burden on on-site staff. While recent advancements in computer vision have facilitated sports match analysis, most existing methods lack real-time capabilities, rendering them unsuitable for automatic replay generation. This paper proposes a real-time ball tracking and action classification method using an event camera for volleyball matches, addressing the gap in automatic replay generation. By leveraging the unique advantages of event cameras, our method avoids machine learning techniques, instead utilizing event-based data streams for lightweight computation in ball tracking and action classification. When ball identification fails, temporal and spatial patterns of key events in volleyball are used for localization. The tracked ball trajectory and classified actions are then employed to identify key moments for replay generation. Experimental results demonstrate the effectiveness of the proposed method, achieving real-time operation and successfully identifying moments suitable for replay generation.

This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less objects from a monocular image by using edge information. The deep learning-based pose estimation method needs a large dataset containing pairs of an image and ground truth pose of objects. To alleviate the cost of collecting a dataset, we focus on the method using a dataset made by computer graphics (CG). This simulation-based method prepares a thousand images by rendering the computer-aided design (CAD) data of the object and trains a deep-learning model. As an inference stage, a monocular RGB image is entered into the model, and the object’s pose is estimated. The representative simulation-based method, Pose Interpreter Networks, uses silhouette images as the input, thereby enabling common feature (contour) extraction from RGB and CG images. However, estimating rotation parameters is less accurate. To overcome this problem, we propose a method to use edge information extracted from the object’s ridgelines for training the deep learning model. Since edge distribution changes largely according to the pose, the estimation of rotation parameters becomes more robust. Through an experiment with simulation data, we quantitatively proved the accuracy improvement compared to the previous method (error rate decreases at a certain condition are translation 22.9% and rotation: 43.4%). Moreover, through an experiment with physical data, we clarified the issues of this method and proposed an effective solution by fine-tuning (error rate decrease at a certain condition are translation 20.1% and rotation 57.7%).

As data analysis becomes increasingly significant, streamlining data collection is essential, particularly in sports. Among various aspects, player performance analysis is crucial for both professional and amateur levels. In computer vision, Multi-Object Tracking (MOT) and Multi-Camera Multi-Object Tracking (MCMOT) enable the analysis of people’s trajectories captured in videos. However, many existing methods do not account for crowded scenarios, such as sports matches, and require complex preprocessing of datasets, making them impractical for real-world applications. To address these challenges, we introduce a simple yet novel extension of MOT methods by utilizing the Gaussian distribution to associate tracking information obtained from multiple cameras. By integrating tracking data from all cameras, our methodology effectively mitigates identity switches and expands the tracking range. Experimental results demonstrate our tracking system’s effectiveness, showing a reduction in ID switches and false negatives compared to results achieved with a standard multi-object tracker.

The process of video recording open surgical procedures is a critical aspect of medical practice, as it enables physicians to retrospectively analyze and assess the surgery. The synthesis of a novel view of surgical procedures from recorded footage is a vital task, facilitating the review of these procedures from multiple angles. Although employing neural radiance fields (NeRF) may seem a straightforward approach, the application of standard NeRF models to medical environments often results in suboptimal rendering quality. This issue predominantly arises due to the significant occlusions present in open surgical videos and the constraints imposed by the limited number of cameras available, further exacerbated by the unique environmental restrictions of the operating room. Through the adoption of a learning framework for novel view rendering, derived from both natural and open surgical scenes using a generalizable NeRF representation, our methodology offers the capacity to render new perspectives from a constrained set of source images. Furthermore, we implement an automatic occlusion mask estimation to exclude obscured pixels during the training phase. The efficacy of our proposed approach has been validated through comparative analyses with other robust training configurations, utilizing real-world video data recorded at Keio University School of Medicine. Additionally, our findings demonstrate the practical effectiveness of the proposed occlusion mask.

Difference-in-level detection in outdoor scenes has various possible applications, including walking assistance for blind people, robot walking assistance, and mapping the hazards of factory premises. It is difficult to detect all outdoor differences in level, such as RGB or RGB-D images, not only including road curbs, which are often targeted for detection in automated driving, but also differences in level on factory premises and sidewalks, because the pattern of outdoor differences in level is abundant and complex. This paper proposes a novel method for detecting differences in level from RGB-D images with segmented ground masks. First, image patches of differences in level were extracted from outdoor images to create the dataset. The change in the normal vector of the contour part on the detected plane is used to generate image patches of the difference in level, but this method strongly depends on the accuracy of planar detection, and it detects only some differences in level. Then, we created the dataset, consisting of image patches and including the extracted differences in level. The dataset is used for training a deep learning model for detecting differences in level in outdoor images without limitations. In addition, because the purpose of this paper is to detect differences in level in outdoor walking areas, regions in the image other than the target areas were excluded by the segmented ground mask. For the performance evaluation, we implemented our algorithm using a modern smartphone with a high-performance depth camera. To evaluate the effectiveness of the proposed method, the results from various inputs, such as RGB, depth, grayscale, normal, and combinations of them, were qualitatively and quantitatively evaluated, and Blender was used to generate synthetic test images for a quantitative evaluation of the difference in level. We confirm that the suggested method successfully detects various types of differences in level in outdoor images, even in complex scenes.

This study focuses on reconstructing game states in American football by generating minimap representations of plays from video footage. Our approach follows a five-step pipeline: detecting and tracking players and referees, identifying players via jersey numbers, transforming camera perspectives, stitching multiview images to create a panoramic image, and generating a minimap. Obtaining accurate player trajectories requires a sideline perspective wide view, providing a comprehensive field overview. Conversely, jersey number recognition demands an end-zone perspective tight view, where players face each other before playing. To integrate these perspectives, we apply image stitching to merge multiple camera views into a seamless scene. Camera transformation techniques ensure accurate spatial alignment across perspectives. Our pipeline achieved 59.9 mean localization error with 45.6% accuracy for team classification for players and referees. This work introduces an efficient pipeline for automated minimap generation, enabling effective game state reconstruction from video footage.

Multi-layer images are a powerful scene representation for high-performance rendering in virtual/augmented reality (VR/AR). The major approach to generate such images is to use a deep neural network trained to encode colors and alpha values of depth certainty on each layer using registered multi-view images. A typical network is aimed at using a limited number of nearest views. Therefore, local noises in input images from a user-navigated camera deteriorate the final rendering quality and interfere with coherency over view transitions. We propose to use a focal stack composed of multi-view inputs to diminish such noises. We also provide theoretical analysis for ideal focal stacks to generate multi-layer images. Our results demonstrate the advantages of using focal stacks in coherent rendering, memory footprint, and AR-supported data capturing. We also show three applications of imaging for VR.

Mounting multi-view cameras within a surgical light is a practical choice since somecameras are expected to observe surgery with few occlusions. Such multi-view videos must bereassembled for easy reference. A typical way is to reconstruct the surgery in 3D. However, thegeometrical relationship among cameras is changed because each camera independently moves everytime the lighting is reconfigured (i.e., every time surgeons touch the surgical light). Moreover, featurematching between surgical images is potentially challenging because of missing rich features. Toaddress the challenge, we propose a feature-matching strategy that enables robust calibration ofthe multi-view camera system by collecting a set of a small number of matches over time whilethe cameras stay stationary. Our approach would enable conversion from multi-view videos to a3D video. However, surgical videos are long and, thus, the cost of the conversion rapidly grows.Therefore, we implement a video player where only selected frames are converted to minimizetime and data until playbacks. We demonstrate that sufficient calibration quality with real surgicalvideos can lead to a promising 3D mesh and a recently emerged 3D multi-layer representation. Wereviewed comments from surgeons to discuss the differences between those 3D representations on anautostereoscopic display with respect to medical usage.

Human Activity Recognition (HAR) technology is pivotal in medical and surveillance systems. Traditional HAR methods using a single sensor are limited in the types of actions they can classify due to the inherent constraints of the sensor. This paper addresses multimodal HAR, which enables the classification of a broader range of activities by utilizing both a camera and an IMU, thereby compensating for each sensor's shortcomings. The proposed method employs images, acceleration, and angular velocity as inputs to a network that integrates an attention mechanism. This mechanism dynamically adjusts the importance of each sensor feature based on the input data. Furthermore, the proposed network is lightweight and high-performance, designed to operate on edge devices with limited computational resources. Experimental results demonstrate that our network can classify five actions with approximately 97% precision, significantly outperforming single-sensor baselines. For a robust evaluation, we conducted experiments using the UESTC-MMEA-CL dataset, confirming that our multimodal approach is more accurate than single-sensor methods.

We present a novel framework of motion tracking from event data using implicit expression. Our framework uses pre-trained event generation MLP called the implicit event generator (IEG) and carries out motion tracking by updating its state (position and velocity) based on the difference between the observed event and generated event from the current state estimation. The difference is computed implicitly by the IEG. Unlike the conventional explicit approach, which requires dense computation to evaluate the difference, our implicit approach realizes the update of the efficient state directly from sparse event data. Our sparse algorithm is especially suitable for mobile robotics applications in which computational resources and battery life are limited. To verify the effectiveness of our method on real-world data, we applied it to the AR marker tracking application. We have confirmed that our framework works well in real-world environments in the presence of noise and background clutter.

Recording surgery in operating rooms is one of the essential tasks for education and evaluation of medical treatment. However, recording the fields which depict the surgery is difficult because the targets are heavily occluded during surgery by the heads or hands of doctors or nurses. We use a recording system which multiple cameras embedded in the surgical lamp, assuming that at least one camera is recording the target without occlusion. In this paper, we propose Conditional-BARF (C-BARF) to generate occlusion-free images by synthesizing novel view images from the camera, aiming to generate videos with smooth camera pose transitions. To the best of our knowledge, this is the first work to tackle the problem of synthesizing a novel view image from multiple images for the surgery scene. We conduct experiments using an original dataset of three different types of surgeries. Our experiments show that we can successfully synthesize novel views from the images recorded by the multiple cameras embedded in the surgical lamp.

The sixth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'23) is part of the ACM International Conference on Multimedia 2023 (ACM Multimedia 2023). The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, for statistical analysis and evaluation, and for sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

In this paper, we propose a method for absolute 3D human pose estimation (HPE) with an uncalibrated monocular camera. In the case of analyzing workers’ movement with an existing uncalibrated camera, the previous method cannot estimate absolute human pose due to the lack of information on camera parameters. Our proposed method overcomes this limitation by determining the position and scale of humans based on the pose of surrounding objects. Specifically, we predict the intrinsic and extrinsic parameters of the camera through user-guided manual manipulation. Subsequently, the estimated human pose is transformed from local coordinates to global coordinates for each frame. This absolute coordinate representation allows for real-time prediction of human movements relative to objects. To assess the efficacy of our method, we conducted three kinds of experiments. A user study revealed that the proposed user-guided method archives accurate estimation of camera parameters. Quantitative evaluation using a public dataset demonstrated that our method can predict human pose with practical accuracy, providing a benchmark for future enhancements. Qualitative evaluation with a unique dataset showed that our method could easily generate digital twin representations across diverse environments and camera positions.

Monocular depth estimation is a challenging task in computer vision. Although many approaches using Convolutional neural networks (CNNs) have been proposed, most of them are trained on large-scale datasets mainly composed of gravity-aligned images. Therefore, conventional approaches fail to predict reliable depth for tilted images containing large pitch and roll camera rotations. To tackle this problem, we propose a novel refining method based on the distribution of gravity directions in the training sets. We designed a gravity rectifier that is learned to transform the gravity direction of a tilted image into a rectified one that matches the gravity-aligned training data distribution. For the evaluation, we employed public datasets and also created our own dataset composed of large pitch and roll camera movements. Our experiments showed that our approach successfully rectified the camera rotation and outperformed our baselines, which achieved 29% improvement in abs rel over the vanilla model. Additionally, our method had competitive accuracy comparable to state-of-the-art monocular depth prediction approaches considering camera rotation.

Multi-camera multi-person (MCMP) tracking and re-identification (ReID) are essential tasks in safety, pedestrian analysis, and so on; however, most research focuses on outdoor scenarios because they are much more complicated to deal with occlusions and misidentification in a crowded room with obstacles. Moreover, it is challenging to complete the two tasks in one framework. We present a trajectory-based method, integrating tracking and ReID tasks. First, the poses of all surgical members captured by each camera are detected frame-by-frame; then, the detected poses are exploited to track the trajectories of all members for each camera; finally, these trajectories of different cameras are clustered to re-identify the members in the operating room across all cameras. Compared to other MCMP tracking and ReID methods, the proposed one mainly exploits trajectories, taking texture features that are less distinguishable in the operating room scenario as auxiliary cues. We also integrate temporal information during ReID, which is more reliable than the state-of-the-art framework where ReID is conducted frame-by-frame. In addition, our framework requires no training before deployment in new scenarios. We also created an annotated MCMP dataset with actual operating room videos. Our experiments prove the effectiveness of the proposed trajectory-based ReID algorithm. The proposed framework achieves 85.44% accuracy in the ReID task, outperforming the state-of-the-art framework in our operating room dataset.

This study addresses a robotic scooping task for deformable plastic objects like wet soil, pastes, or plasticines. In this task, a robot arm holding a spoon scoops the object with an arbitrary target weight. Scooping deformable plastic objects presents a challenge in robot control. The robot needs to consider the object's invisible physical properties, which can be varied due to extrinsic factors. For this reason, the same scooping action on the same objects can exhibit different scooping outcomes. We tackle this problem with tactile sensors. If a robot perceives the reactive force during the scooping action through the sensors, it will lead to more accurate weighing. We realize this concept with a scooped weight prediction neural network. We first train the network to predict scooped weights, where input is tactile and motion parameters. Then, we identify the optimal motion parameters from the tactile and target weights. To verify the proposed method, we performed experiments by comparing two approaches: our proposed method with tactile sensors and the baseline without them. We trained the scoop weight predictive model with 80 scooping motions by randomizing one of the parameters. Our method had 1.40, 1.40, 1.30, 1.75, and 1.64 g average errors in scooped weights with 3, 5, 7, 9, and 11 g target weights, respectively, demonstrating smaller errors for most target weights than the baseline. The proposed method is easy to implement and potentially available for various applications such as food industries, laboratory automation, and assistive feeding in everyday life.

Due to the recent technological advances in inertial measurement units (IMUs), many applications for the measurement of human motion using multiple body-worn IMUs have been developed. In these applications, each IMU has to be attached to a predefined body segment. A technique to identify the body segment on which each IMU is mounted allows users to attach inertial sensors to arbitrary body segments, which avoids having to remeasure due to incorrect attachment of the sensors. We address this IMU-to-segment assignment problem and propose a novel end-to-end learning model that incorporates a global feature generation module and an attention-based mechanism. The former extracts the feature representing the motion of all attached IMUs, and the latter enables the model to learn the dependency relationships between the IMUs. The proposed model thus identifies the IMU placement based on the features from global motion and relevant IMUs. We quantitatively evaluated the proposed method using synthetic and real public datasets with three sensor configurations, including a full-body configuration mounting 15 sensors. The results demonstrated that our approach significantly outperformed the conventional and baseline methods for all datasets and sensor configurations.

The fourth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'21) is part of the ACM International Conference on Multimedia 2021 (ACM Multimedia 2021). Exceptionally, due to the corona pandemic, the workshop is held virtually. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding and visualizing the multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, statistical analysis and evaluation, and sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports.

The fifth ACM International Workshop on Multimedia Content Analysis in Sports (ACM MMSports'22) is part of the ACM International Conference on Multimedia 2022 (ACM Multimedia 2022). After two years of pure virtual MMSports workshops due to COVID-19, MMSports'22 is held on-site again. The goal of this workshop is to bring together researchers and practitioners from academia and industry to address challenges and report progress in mining, analyzing, understanding, and visualizing multimedia/multimodal data in sports, sports broadcasts, sports games and sports medicine. The combination of sports and modern technology offers a novel and intriguing field of research with promising approaches for visual broadcast augmentation and understanding, for statistical analysis and evaluation, and for sensor fusion during workouts as well as competitions. There is a lack of research communities focusing on the fusion of multiple modalities. We are helping to close this research gap with this workshop series on multimedia content analysis in sports. Related Workshop Proceedings are available in the ACM DL at: https://dl.acm.org/doi/proceedings/10.1145/3552437.

Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods.