Published papers

Analysis of the 3D structure of wires in electrical cables is an important technique for verifying various characteristics of cables. Nondestructive inspection using X-ray CT is available for this purpose. However, due to the limited resolution of X-ray CT images, each wire in the images is blurred, making it difficult to accurately extract the 3D structure of thousands of wires from thousands of CT images taken in the longitudinal direction of the cable. In this study, we propose a method for estimating the 3D structure of all wires in a cable from a 3D X-ray CT image of the cable. The proposed method first detects wires in each image using Faster R-CNN. It then tracks each wire in a set of cross-sectional images collected in the longitudinal direction of the cable to estimate the structure as a 3D trajectory of wires. The pretrained Faster R-CNN model is fine-tuned using our synthetic image dataset that simulates the wire region. It achieves wire detection without the need of annotating the training data. The results showed that wires can be detected with an error of about 1%, and 40% to 80% of the wires can be tracked continuously over 500 frames.

Strategic view sampling is fundamental for the quality view synthesis. However, no established evaluation approach exists to average individual view sampling patterns and compare them across different strategies with AR visual guidance. To address this, we propose a spatial and cross-participant evaluation method. Our method aligns independent 3D reconstructions from different participants into a common coordinate system using Coherent Point Drift. It evaluates view contribution by accumulating projected screen spaces onto the canonical mesh. Our method enables direct analysis of the spatial characteristics of view sampling that are not captured by conventional image quality metrics, providing insights for improving view sampling system.

Predicting future human behavior from egocentric videos is a challenging but critical task for human intention understanding. Existing methods for forecasting 2D hand positions rely on visual representations and mainly focus on hand-object interactions. In this paper, we investigate the hand forecasting task and tackle two significant issues that persist in the existing methods: (1) 2D hand positions in future frames are severely affected by ego-motions in egocentric videos; (2) prediction based on visual information tends to overfit to background or scene textures, posing a challenge for generalization on novel scenes or human behaviors. To solve the aforementioned problems, we propose EMAG, an ego-motion-aware and generalizable 2D hand forecasting method. In response to the first problem, we propose a method that considers ego-motion, represented by a sequence of homography matrices of two consecutive frames. We further leverage modalities such as optical flow, trajectories of hands and interacting objects, and ego-motions, thereby alleviating the second issue. Extensive experiments on two large-scale egocentric video datasets, Ego4D and EPIC-Kitchens 55, verify the effectiveness of the proposed method. In particular, our model outperforms prior methods by 1.7% and 7.0% on intra and cross-dataset evaluations, respectively. Project Page: https://masashi-hatano.github.io/EMAG/.

Gaitisafundamentalaspectofhumanmobility,anddisruptionsinnormalgaitcansignificantly reduce quality of life (QOL). Although recent advances in 3D gait analysis (3DGA) enable precise, quantitative assessments, these methods are typically confined to controlled laboratory environments and thus fail to accurately capture natural gait variability. Conversely, wearable IMU sensors offer cost-effective, portable solutions for capturing movements across diverse settings but face challenges such as invasiveness and sensor drift. In this study, we propose ‘‘Gait Inertial Poser (GIP),’’ a novel method estimating 3D full body pose during straight walking on flat ground, using only two shoe-embedded IMU sensors. GIP initially estimates personalized body shapes from user attributes (height, weight, age, gender) and then employs a Transformer-based module to infer gait motion parameters from IMU data. To ensure temporal continuity and smoothness of the estimated motion, we further introduce a smoothing module based on a Variational Autoencoder (VAE), whichfurther incorporates a specialized loss function that explicitly enforces kinematic constraints during foot-ground contact, thereby improving the overall estimation accuracy. Comprehensive experiments conducted on two public datasets quantitatively and qualitatively demonstrate that GIP achieves high accuracy in straight-line walking. This approach overcomes limitations of traditional laboratory based methods, opening new opportunities for real-time monitoring and remote rehabilitation in everyday environments.

Accurate human motion estimation is crucial for sports motion analysis. Traditional approaches often depend on motion capture systems or controlled environments, limiting their applicability to real-world sports scenarios. In this work, we propose a novel 3D human motion estimation method that leverages multiple dynamically moving cameras in an uncalibrated setting, eliminating the need for prior environmental knowledge. Our approach reconstructs human meshes from multiple viewpoints and adaptively assigns confidence-based weights to each view, ensuring robust motion estimation even under occlusions and varying camera positions. By integrating confidence-aware multiview fusion, our method significantly improves the accuracy and robustness of 3D human mesh reconstruction in unconstrained sports environments. This work advances the feasibility of motion analysis in real-world settings without requiring specialized motion capture systems or rigid spatial constraints.

Evaluating concept customization is challenging, as it requires a comprehensive assessment of fidelity to generative prompts and concept images. Moreover, evaluating multiple concepts is considerably more difficult than evaluating a single concept, as it demands detailed assessment not only for each individual concept but also for the interactions among concepts. While humans can intuitively assess generated images, existing metrics often provide either overly narrow or overly generalized evaluations, resulting in misalignment with human preference. To address this, we propose Decomposed GPT Score (D-GPTScore), a novel human-aligned evaluation method that decomposes evaluation criteria into finer aspects and incorporates aspect-wise assessments using Multimodal Large Language Model (MLLM). Additionally, we release Human Preference-Aligned Concept Customization Benchmark (CC-AlignBench), a benchmark dataset containing both single- and multi-concept tasks, enabling stage-wise evaluation across a wide difficulty range - from individual actions to multi-person interactions. Our method significantly outperforms existing approaches on this benchmark, exhibiting higher correlation with human preferences. This work establishes a new standard for evaluating concept customization and highlights key challenges for future research. The benchmark and associated materials are available at https://github.com/ReinaIshikawa/D-GPTScore

High-speed phenomena are difficult to perceive due to human vision limitations, yet their immediate understanding is crucial in fields like sports refereeing and neurosciences. Existing systems often rely on post-processing or require active user intervention, failing to provide intuitive, real-time insights. This study presents a system that instantly visualizes high-speed phenomena in slow-motion using an event camera. The system prepares a background image, captured in advance, as a spatial reference. It overlays detected high-speed phenomena on this image for visualization. The system automatically detects high-speed phenomena and transitions to slow-motion for detailed analysis, while maintaining a real-time display. Auditory notifications alert users to critical moments. Through two user studies, we confirmed that event cameras improved accurate perception of high-speed phenomena. Moreover, the integration of Event-Timeline enables users to perceive these phenomena intuitively, without experiencing cognitive overload. The experimental results demonstrated a significant usability improvement (SUS: 84.7, NASA-TLX: 16.3). Focus group discussions refined visualization techniques, resulting in the development of design implications. This system enables intuitive and immediate understanding of unpredictable high-speed occurrences, with applications in sports, animal behavior monitoring, and motion analysis.

We present a system for generating looping 3D videos from casual monocular footage captured by a single moving camera. Unlike existing looping 3D video methods that rely on a stationary multicamera system, our approach synthesizes multi-camera views to mimic a physical multi-camera setup. Our key technical contribution is the efficient construction of all-pairs long-term 2D tracking by reusing multi-step optical flow, which avoids the need for expensive pairwise tracking. Our approach is 122.8 times faster than the pairwise approach while maintaining 98.2,99.5, and 99.7% of the visual quality in PSNR, SSIM, and LPIPS, respectively.

We address a novel cross-domain few-shot learning task (CD-FSL) with multimodal input and unlabeled target data for egocentric action recognition. This paper simultaneously tackles two critical challenges associated with egocentric action recognition in CD-FSL settings: (1) the extreme domain gap in egocentric videos (e.g., daily life vs. industrial domain) and (2) the computational cost for real-world applications. We propose MM-CDFSL, a domain-adaptive and computationally efficient approach designed to enhance adaptability to the target domain and improve inference cost. To address the first challenge, we propose the incorporation of multimodal distillation into the student RGB model using teacher models. Each teacher model is trained independently on source and target data for its respective modality. Leveraging only unlabeled target data during multimodal distillation enhances the student model’s adaptability to the target domain. We further introduce ensemble masked inference, a technique that reduces the number of input tokens through masking. In this approach, ensemble prediction mitigates the performance degradation caused by masking, effectively addressing the second issue. Our approach outperformed the state-of-the-art CD-FSL approaches with a substantial margin on multiple egocentric datasets, improving by an average of 6.12/6.10 points for 1-shot/5-shot settings while achieving 2.2 times faster inference speed. Project page: https://masashi-hatano.github.io/MM-CDFSL/

Recording surgery in operating rooms is a crucial task for both medical education and evaluation of medical treatment. In this paper, we propose a method for visualizing surgical areas that are occluded by the heads or hands of medical professionals in various surgical scenes. To recover the occluded surgical areas, we utilize a surgery recording system equipped with multiple cameras embedded in the surgical lamp, with the aim of ensuring that at least one camera can capture the surgical area without occlusion. We propose the application of a transformer-based Neural Radiance Field (NeRF) model, originally proposed for normal scenes, to surgery scenes, and demonstrate through experimentation that it is feasible to generate occluded surgical areas. We believe this research has the potential to make our multi-camera recording system practical and useful for physicians.

Video recording of open surgery is in great demand for education and research purposes but is challenging due to the busy and dynamic environment. The state-of-the-art system uses multi-view cameras installed in shadowless lamps (McSL) and implements an automatic camera switching algorithm to avoid disturbances. However, this algorithm leads to missing pixels and distorted projection due to mathematical image warping and does not always provide the best perspective. We propose using 4D Gaussian Splatting (4DGS) to create editable 3D videos and remove Gaussians occluding surgical fields from a perspective. We enable occlusion-free 3D videos by addressing two occlusion removal approaches via (1) occlusion masking and (2) density-based Gaussian filtering. We create a real-surgery dataset and demonstrate that our method outperforms the state-of-the-art auto view-switching approach.

Open-skill sports require training methods that address perceptual-cognitive functions that influence action timing, in addition to movement technique. Sonification can support action timing by mapping continuous task dynamics to sound without visual overlays. We present a preliminary design exploration of sonification for real-time ball-position feedback in tennis. We implement an end-to-end system that estimates ball position using event-based vision and renders distance dynamics as continuous audio. We examine a 2×2 design space that crosses representation style (explicit tone vs. implicit wind) with frequency range (low: one octave vs. high: two octaves), reflecting a trade-off between change legibility and attentional demand. Pilot observations suggest that highly interpretable sonification can facilitate users' motor timing perception, whereas implicit designs may support attentional stability.

The growing interest in the Metaverse drives demand for digital twins that accurately replicate physical environments. While capturing real-time human dynamics in static environments is essential for immersion, it is limited by the scalability of conventional motion capture. Furthermore, multi-user spatial consistency is challenging when using only first-person view (FPV) streams. We propose a real-time system generating dynamic digital twins from multiple FPV videos. Our method achieves robust localization by integrating a pre-scanned 3D Gaussian Splatting map, ORB-SLAM3, and YOLO-based dynamic object removal. We integrate individual motion synthesis with spatial synchronization; leveraging Ego-Poser for full-body reconstruction, we unify disparate streams into a global coordinate system. Experimental results show our system operates at 30 to 60 FPS, enabling synchronized visualization of multiple 3D human models in a photorealistic environment.

We propose a method for estimating camera trajectories optimally aligned with prepared environment maps and its application. We implement Direct Sparse Odometry (DSO) as real-time Visual Odometry (VO) and Structure from Motion (SfM) to build environment maps. In this process, a pose graph is created and optimized using related camera poses to calculate the alignment of all frames from DSO towards the environment map with the scales matched. Therefore, the proposed method runs solely with a monocular color camera and does not require any training datasets. We demonstrate that the proposed method is able to be used as an object-based recognition system based on the difference between a map as a prior information and the current scene observations by the user. We tested this real-time application in a laboratory environment. In addition, we conducted an experiment to evaluate the accuracy of the proposed method using an existing dataset. The results showed that the proposed method improved the accuracy of pose estimation compared to the case using only DSO.

Surgical tool detection is essential for analyzing and evaluating minimally invasive surgery videos. Current approaches are mostly based on supervised methods that require large, fully instance-level labels (i.e., bounding boxes). However, large image datasets with instance-level labels are often limited because of the burden of annotation. Thus, surgical tool detection is important when providing image-level labels instead of instance-level labels since image-level annotations are considerably more time-efficient than instance-level annotations. In this work, we propose to strike a balance between the extremely costly annotation burden and detection performance. We further propose a co-occurrence loss, which considers a characteristic that some tool pairs often co-occur together in an image to leverage image-level labels. Encapsulating the knowledge of co-occurrence using the co-occurrence loss helps to overcome the difficulty in classification that originates from the fact that some tools have similar shapes and textures. Extensive experiments conducted on the Endovis2018 dataset in various data settings show the effectiveness of our method.

In recent years, much attention has been paid to the prediction of pedestrian trajectories, as they are one of the key factors for a better society, such as automatic driving, a guide for blind people, and social robots interacting with humans. To tackle this task, many methods have been proposed but few are from the first-person perspective because of the lack of a publicly available dataset. Therefore, we propose a method that uses egocentric vision, which does not need to be trained with a first-person video dataset. We made it possible to utilize existing methods, which predict from a bird’s-eye view. In addition, we propose a novel way to consider semantic information without changing the shape of the input to apply to all existing bird’s-eye methods that use only past trajectories. Therefore, there is no need to create a new dataset from egocentric vision. The experimental results demonstrate that the proposed method makes it possible to predict from an egocentric view via existing methods of bird’s-eye view. The proposed method qualitatively improves trajectory predictions without aggravating quantitative accracy, and the effectiveness of predicting the trajectories of multiple people simultaneously.

Reliably and accurately forecasting future trajectories of pedestrians is necessary for systems like autonomous vehicles or visual assistive devices to function correctly. While previous state-of-the-art methods relied on modeling social interactions with LSTMs, with videos captured with a static camera from a bird’s-eye view, our paper presents a new method that leverages the Transformers architecture and offers a reliable way to model future trajectories in first-person videos captured by a body-mounted camera, without having to model any social interactions. Accurately forecasting future trajectories is a challenging task, mainly due to how unpredictably humans move. We tackle this issue by using information about target persons’ previous locations, scales, and dynamic poses, as well as information about the camera wearer’s ego-motion. The model we propose predicts future trajectories in a simple way, modeling each target’s trajectory separately, without the use of complex social interactions between humans or interactions between targets and the scene. Experimental results show that our method overall outperforms previous state-of-the-art methods, and yields better results in challenging situations where previous state-of-the-art methods fail.

Automatic fall detection is a crucial task in healthcare as falls pose a significant risk to the health of elderly individuals. This paper presents a lightweight acceleration-based fall detection method that can be implemented on edge devices. The proposed method uses Autoencoders, a type of unsupervised learning, within the framework of anomaly detection, allowing for network training without requiring extensive labeled fall data. One of the challenges in fall detection is the difficulty in collecting fall data. However, our proposed method can overcome this limitation by training the neural network without fall data, using the anomaly detection framework of Autoencoders. Additionally, this method employs an extremely lightweight Autoencoder that can run independently on an edge device, eliminating the need to transmit data to a server and minimizing privacy concerns. We conducted experiments comparing the performance of our proposed method with that of a baseline method using a unique fall detection dataset. Our results confirm that our method outperforms the baseline method in detecting falls with higher accuracy.

Augmented reality (AR) provides ways to visualize missing view samples for novel view synthesis. Existing approaches present 3D annotations for new view samples and task users with taking images by aligning the AR display. This data collection task is known to be mentally demanding and limits capture areas to pre-defined small areas due to the ideal but restrictive underlying sampling theory. To free users from 3D annotations and limited scene exploration, we propose using locally reconstructed light fields and visualizing errors to be removed by inserting new views. Our results show that the error-peaking visualization is less invasive, reduces disappointment in final results, and is satisfactory with fewer view samples in our mobile view synthesis system. We also show that our approach can contribute to recent radiance field reconstruction for larger scenes, such as 3D Gaussian splatting.

Cervical myelopathy (CM) is a pathology of the spinal cord that causes upper limb disorders. CM is often screened by conducting the 10-s grip and release (G&R) test, which mainly focuses on hand dysfunction caused by CM. This test has patients repeat gripping and releasing their hands as quickly as possible. Spine surgeons observe the quickness of this repetition to screen for CM. We propose an automatic screening method of CM that involves patients’ hands recorded as videos when they are performing the G&R test. The videos are used to estimate feature values, i.e., the positions of each part of the hand, which are obtained through image processing. A support vector machine classifier classifies CM patients and controls with these feature values after pre-processing. We validated our method with 10-fold cross-validation and the videos of 20 CM patients and 15 controls. The results indicate that sensitivity, specificity, and area under the receiver operating characteristic curve were 90.0 %, 93.3 %, and 0.947, respectively.