Published papers

We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes natural music that causes minimal discomfort to listeners. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Our datasets and code will be made publicly available.

The development of Convolutional Neural Networks (CNN) has produced remarkable great contribution to a wide range of image processing fields. On the other hand,there are premised on the existence of a large amount of annotated data. In addition to it, when a model learned in a domain is applied to a different domain, even if in the same task, there is no guarantee of its accuracy. This is a very important issue when deep learning and machine learning are applied in the field. If re-annotation re-learning is performed for data with such a domain difference, the same accuracy can be expected to be maintained again. But, semantic segmentation needs fine annotation and its high labor cost makes its application difficult. Histopathological image segmentation which is expected to drug discovery and medical image analysis is expensive due to its annotation cost, a wide variety of specimen and the need for the skills of histopathological experts. Therefore, the purpose of this research is to reduce the re-annotation cost of data in the new environment using the idea of domain adaptation. In this study, we focus on domain adaptation focusing on the class imbalance problem. The output of the class with few labels tends to be unstable in the class imbalanced data We proposed new cost function which is focusing on the class of few labels with histopathological image. As you can see our result, we achived imporvement of this problem. This made it possible to create a model that suppressed the class imbalance of the dataset in domain adaptation for pathological image segmentation.

Event cameras are emerging vision sensors whose noise is challenging to characterize. Existing denoising methods for event cameras are often designed in isolation and thus consider other tasks, such as motion estimation, separately (i.e., sequentially after denoising). However, motion is an intrinsic part of event data, since scene edges cannot be sensed without motion. We propose, to the best of our knowledge, the first method that simultaneously estimates motion in its various forms (e.g., ego-motion, optical flow) and noise. The method is flexible, as it allows replacing the one-step motion estimation of the widely-used Contrast Maximization framework with any other motion estimator, such as deep neural networks. The experiments show that the proposed method achieves state-of-the-art results on the E-MLB denoising benchmark and competitive results on the DND21 benchmark, while demonstrating effectiveness across motion estimation and intensity reconstruction tasks. Our approach advances event-data denoising theory and expands practical denoising use-cases via open-source code. Project page: https://github.com/tub-rip/ESMD

In recent years, the collection of data for the purpose of improving playing techniques and tactical analysis has become a popular activity in various sports. These tasks are often handled manually by analysts, and there is a desire for a method to alleviate this burden. To meet this demand, this paper proposes a system that can automatically recognize game contexts. In this study, we focused on the most important event, the shot, and aimed to detect it with a temporal resolution of frame level. In order to do so, we apply conventional CNN and RNN-based action recognition methods and show that the proposed method can detect shots with an accuracy of more than 71% of F-score by processing and training the teacher data. In order to demonstrate that the proposed method works effectively in various environments (e.g., different courts), we created and verified a dataset of high-resolution videos taken by ourselves as well as videos of four major tournaments (Australian Open, French Open, Wimbledon, and US Open) obtained from YouTube with poor quality. This technology is expected to be applied not only to tennis but also to other sports that involve ball shots, such as table tennis, baseball, and volleyball.

In order to create a classification model for fungi, it is necessary to have robustness against out-of-distribution data from the viewpoint of practicality. Therefore, in this paper, we perform out-of-distribution detection on a fungi. Unlike the case of conventional out-of-distribution detection, the characteristics of in-distribution data and out-of-distribution data in this paper are very similar. Therefore, the problem in which conventional methods using out-of-distribution data for validation are not effective is mentioned. We also verify whether the accuracy of out-of-distribution detection can be improved using the attention branch network.

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flow.

This paper presents an algorithm to provide floor projection feedback according to the local distance and density of individuals. It is realized by a large-space floor projection system with a feedback function based on human tracking with laser ranging image sensors. The purpose is to support the cognition of spatial–temporal structures of groups of adolescents with neurodevelopmental disorders (NDs) that are conducting organized physical activity (PA). Observation and evaluation of behavioral changes in adolescents with NDs, when they were active with or without the floor projection based on the proposed algorithm, were conducted to validate its effectiveness. We observed that the proposed algorithm can be implemented in different organized PAs. It had the effect to help individuals in a behavior to keep a close distance to each other as a group rather than to keep the same distance apart from each other while walking.

In this paper, we propose a new method for audio-visual event localization 1) to find the corresponding segment between audio and visual event. While previous methods use Long Short-Term Memory (LSTM) networks to extract temporal features, recurrent neural networks like LSTM are not able to precisely learn long-term features. Thus, we propose a Temporal Cross-Modal Attention (TCMA) module, which extract temporal features more precisely from the two modalities. Inspired by the success of attention modules in capturing long-term features, we introduce TCMA, which incorporates self-attention. Finally, we were able to localize audio-visual event precisely and achieved a higher accuracy than the previous works.

In this paper, we propose preliminary action recognition which is an action anticipation method that can be used in real-time. Preliminary action recognition as a solution to action anticipation, which is the task of detecting the action of a person before the person performs the action. We define preliminary action class before action class that is anticipated by recognizing the preliminary actions specific to the action before it occurs, just as humans do. Conventional action anticipation methods have two problems for real-time use. First, it requires 10 to 20 seconds of video to anticipate 1 second ahead, and it can only anticipate every 10 seconds. Second, the video input in training always switches actions 1 second after the end of the video. This problem setting can be a leakage because there is no action break in real-time operation. Proposed method solves this problem and makes it possible to operate action prediction in real-time. In addition, since action classes and preliminary action classes tend to share a lot of similarities, it is difficult to distinguish between them. We propose a model that can discriminate the subtle differences in motion by increasing the latent distance between adjacent classes in terms of time.

Event cameras asynchronously report per-pixel intensity changes with high temporal resolution. Their sparse and temporally precise nature is well suited for fast visual flow estimation. The normal optical flow can be estimated by fitting a plane to spatio-temporal events. However, least-squares plane fitting suffers from outliers due to the significant noise of events. Existing methods involve 1) iterative outlier rejection or 2) goodness-of-fit rejection after single-shot planar fitting from greedily selected spatially neighboring events. In contrast to these methods, we propose a method of selecting the events supporting a plane before performing a fitting, using the inlier probability from a lightweight neural network that captures both global and local structures. During inference, single-shot planar fitting is performed from only events with a higher inlier probability. We model each event selection by a Bernoulli distribution with the inlier probability and train the network to maximize the inlier count while sampling in a self-supervised manner. We verify that our event selection improves the accuracy of optical flow estimation with publicly available real data.

In recent years, deep learning models have achieved great success in many fields, but their training relies on large labeled data. Annotation to obtain labeled samples is challenging, time-consuming, and expensive, and has been a major challenge in deep learning. In this study, we propose a method for unsupervised domain adaptation that can reduce annotation cost in terms of accuracy and time compared to conventional methods by using the uncertainty of the model in semantic segmentation. Unsupervised domain adaptation aims to adapt models trained on synthetic data to real-world data without the need for costly annotation of real images. Since the training data are images and correct labels created by the game engine, there is no need to manually annotate them. However, if the source (synthetic) data is trained using conventional supervised learning, the performance is significantly degraded because the domain is different from the target (real image) data.This study closes the domain gap between source and target by calculating the uncertainty in the target data from the model output on a pixel-by-pixel basis and minimizing this uncertainty as loss. Adding this uncertainty loss to conventional unsupervised domain adaptation methods results in a model that is more robust to the target domain and achieves state-of-the-art results.

In this work, we investigate the understudied effect of the training data used for image super-resolution (SR). Most commonly, novel SR methods are developed and benchmarked on common training datasets such as DIV2K and DF2K. However, we investigate and rethink the training data from the perspectives of diversity and quality, thereby addressing the question of “How important is SR training for SR models?”. To this end, we propose an automated image evaluation pipeline. With this, we stratify existing high-resolution image datasets and larger-scale image datasets such as ImageNet and PASS to compare their performances. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance. We hope that the proposed simple-yet-effective dataset curation pipeline will inform the construction of SR datasets in the future and yield overall better models. Code is available at: https://github.com/gohtanii/DiverSeg-dataset.

In this research, we propose a novel approach using unsupervised metric learning tailored to datasets characterized by complex similarities and connections, such as those found in paintings and makeup, which are challenging to express linguistically. These datasets often present the difficulty of adequately analyzing data points due to the intricate interplay of defining elements, a limitation of traditional labeling methods. Additionally, the high degree of specialization required makes annotation significantly costly. Unsupervised metric learning emerges as a powerful method for extracting more cost-effective features and for the comprehensive analysis of these datasets. Expanding upon previous research that utilized style transfer models, our study further explores feature design, specifically focusing on extracting detailed information about critical aspects of similarity assessment, such as color and shape. Our model adeptly incorporates visual information, unveiling the hidden abstract connections within datasets. We validated our approach using a dataset of Ukiyo-e, a genre of Japanese painting, and achieved accuracy comparable to supervised learning models. This research opens up new possibilities for the analysis of complex image datasets with abstract relational depth, fostering a deeper understanding of the data.

In this paper, we propose to pre-train audio encoders using synthetic patterns instead of real audio data. Our proposed framework consists of two key elements. The first one is Masked Autoencoder (MAE), a self-supervised learning framework that learns from reconstructing data from randomly masked counterparts. MAEs tend to focus on low-level information such as visual patterns and regularities within data. Therefore, it is unimportant what is portrayed in the input, whether it be images, audio mel-spectrograms, or even synthetic patterns. This leads to the second key element, which is synthetic data. Synthetic data, unlike real audio, is free from privacy and licensing infringement issues. By combining MAEs and synthetic patterns, our framework enables the model to learn generalized feature representations without real data, while addressing the issues related to real audio. To evaluate the efficacy of our framework, we conduct extensive experiments across a total of 13 audio tasks and 17 synthetic datasets. The experiments provide insights into which types of synthetic patterns are effective for audio. Our results demonstrate that our framework achieves performance comparable to models pre-trained on AudioSet-2M and partially outperforms image-based pre-training methods.

Training datasets for image super-resolution (SR) are often constructed from web images. However, these images are typically stored in JPEG format, introducing compression artifacts that degrade SR performance. To ensure data quality, conventional approaches discard severely compressed images. While this preserves image quality, it reduces data diversity, which in turn limits model performance. To address this issue, we propose RECA (Refinement of Compressed Artifacts), a novel pipeline that enables the reuse of JPEG images that would otherwise be excluded. RECA first detects heavily compressed images using a blockiness measure, then restores visual quality using a real-world diffusion-based SR model, and finally applies bicubic downsampling to suppress artifacts introduced during enhancement. This process allows the effective inclusion of compressed images in training without sacrificing data quality. Experimental results demonstrate that RECA consistently outperforms filtering-based baselines in terms of PSNR and SSIM. These findings highlight the effectiveness of RECA in expanding usable training data and improving SR model performance.

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.

In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

As part of efforts to address the issue of fairness in computer vision, image datasets are being reviewed, and inappropriate datasets are being temporarily suspended or withdrawn from public availability. What impact do pre-trained models using large-scale image datasets have on downstream tasks from the perspective of fairness? In this paper, we quantitatively evaluate how the pre-training methods, including manually supervised learning (MSL) and self-supervised learning (SSL) pre-training with ImageNet, affect downstream tasks in terms of fairness. We reached three findings from our experiments, summarized as follows: (i) SimCLRv2 and MoCov2 pre-trained models perform more fairly than MSL pre-trained models with ImageNet on IMDB-WIKI, FairFace, and CIFAR-10S. (ii) The MoCov2 pre-trained model achieves better performance in accuracy and fairness metrics than the SimCLRv2 pre-trained model. (iii) In MoCov2, color jitter tends to improve the fairness metrics for downstream tasks. By considering these findings, we demonstrate that SSL pre-trained models have the potential for fairer image recognition with similar accuracy compared to the MSL pre-trained model. By replacing human annotations with self-supervision, we can construct fair pre-trained models does not depend solely on human-annotated labels in the context of pre-training.

Event cameras provide rich signals that are suitable for motion estimation since they respond to changes in the scene. As any visual changes in the scene produce event data, it is paramount to classify the data into different motions (i.e., motion segmentation), which is useful for various tasks such as object detection and visual servoing. We pro-pose an iterative motion segmentation method, by classifying events into background (e.g., dominant motion hypoth-esis) and foreground (independent motion residuals), thus extending the Contrast Maximization framework. Experi-mental results demonstrate that the proposed method suc-cessfully classifies event clusters both for public and self-recorded datasets, producing sharp, motion-compensated edge-like images. The proposed method achieves state-of-the-art accuracy on moving object detection benchmarks with an improvement of over 30%, and demonstrates its possibility of applying to more complex and noisy real-world scenes. We hope this work broadens the sensitiv-ity of Contrast Maximization with respect to both motion parameters and input events, thus contributing to theoret-ical advancements in event-based motion segmentation es-timation. https://github.com/aoki-media-lab/event_based_segmentation.vcmax

In this paper, we propose Language-Guided Contrastive AudioVisual Masked Autoencoders (LG-CAV-MAE) to improve audio-visual representation learning. LG-CAV-MAE integrates a pretrained text encoder into contrastive audio-visual masked autoencoders, enabling the model to learn across audio, visual and text modalities. To train LG-CAV-MAE, we introduce an automatic method to generate audio-visual-text triplets from unlabeled videos. We first generate frame-level captions using an image captioning model and then apply CLAP-based filtering to ensure strong alignment between audio and captions. This approach yields high-quality audio-visual-text triplets without requiring manual annotations. We evaluate LG-CAV-MAE on audio-visual retrieval tasks, as well as an audio-visual classification task. Our method significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task.