Published papers

高齢化が進む社会において，自然言語指示文に基づき日常タスクを実行する生活支援ロボットの実用化は介助者不足の解決策の一つとなりうる．一方，自然言語指示文に基づき日常タスクを実行するためのマルチモーダル言語理解モデルは未だ性能が不十分であり，性能向上には高品質な自然言語指示文を含むデータセットを用いた訓練が必要である．よって生活支援ロボットの指示文理解性能向上において自然言語指示文生成は重要である．本研究では2枚の画像に基づき物体操作指示文を生成する手法を提案する．また，生成文により拡張したデータセットを用い，物体操作指示文理解モデルを訓練・評価した．結果として，理解性能が向上したことから，拡張データの有効性を確認した．また，実機実験においても同様に，拡張データが有効であることが検証された．

In this paper, we propose the Multimodal Diffusion Segmentation Model (MDSM), which generates a mask in the first stage and refines it in the second stage. We introduce a crossmodal parallel feature extraction mechanism and extend diffusion probabilistic models to handle crossmodal features. Our proposed MDSM surpasses that of the baseline method by a large margin of +10.13 mean IoU.

Visual scene understanding, such as image captioning, can be considered one of the essential topics in the artificial intelligence (AI) field. Image captioning with reading comprehension tasks as an extension of traditional image captioning is more challenging because the generated caption must be related to the text information in the image, and how to read and comprehend text in the context of an image needs to be studied. In this work, we propose multiple image-related attention blocks with multimodal Optical Character Recognition (OCR) information to model the relationship among the global image, multi-level recognized text, and the detected objects in the image. Our model is validated on the standard dataset TextCaps, and the results show that our model outperforms the baseline methods in terms of all evaluation matrices.

In the modern era where deep learning is applied across a wide range of fields, the explainability of models is of paramount importance. However, existing methods are not optimized for vision-language foundation models, leading to lower explanation quality for such models. Therefore, this study proposes the Alternative Adapter Model, an explanation generation model tailored to vision-language foundation models. By introducing a Side Branch Network connected to the vision-language foundation model, the proposed method extracts features suitable for explanation generation. Furthermore, by implementing the Alternative Epoch Architecture, which dynamically changes the outputs of modules and the layers to be frozen, we address the issue of overly narrow focus areas. To evaluate the proposed method, experiments were conducted using the CUB-200-2011 dataset. The results demonstrate that the proposed method surpasses existing methods in mean IoU, Insertion Score, Deletion Score, and Insertion-Deletion Score, which are standard metrics for visual explanation generation tasks.

In this paper, we focus on the task of visualizing important regions in an image as high-quality visual explanations of the model’s decisions with a clear theoretical background. We introduce a novel calculation method for Layer-wise Relevance Propagation (LRP) specifically tailored to models featuring skip connections such as ResNet. This method’s strength lies in its adaptability, as the backpropagation technique is distinctly defined for each layer, enhancing its extensibility. To validate our method, we conduct an experiment on the CUB-200-2011 dataset. The proposed method successfully generates appropriate explanations and, based on the Insertion-Deletion score, outperforms the baseline methods.

This study explores a task where a robot searches for images containing target objects based on user language queries from a large set of images captured in diverse indoor and outdoor environments. Both images with and without scene text are considered. For example, when searching with the query, "Pass me the red container of Sun-Maid raisins on the kitchen counter," the model ranks images containing a container labeled "Sun-Maid raisins" on the kitchen counter higher. However, linking visual semantics with scene text is challenging. Additionally, multimodal search requires large-scale, high-speed inference, making it impractical to rely solely on a multimodal large language model (MLLM). To address this, we introduce a Scene Text Visual Encoder, integrating an Aligned Representation with a narrative representation obtained using an MLLM based on Crosslingual Visual Prompting. Incorporating OCR results into the prompt further reduces hallucination. Experiments show that the proposed method outperforms multimodal foundation models across multiple benchmarks in standard evaluation metrics for ranking-based learning.

In the modern era, where deep learning is applied across a wide range of fields, the explainability of deep learning models is crucial, and the generation of explanations with high transparency is desirable. Layer-wise Relevance Propagation (LRP) is mentioned as a method for generating explanations by backpropagating relevance scores, offering high transparency. However, this method has been applied only to models without skip connections, as its application to models with skip connections does not satisfy the conservation property, resulting in poor quality explanations. Therefore, this paper proposes a method for calculating the backpropagation of relevance scores that satisfies the conservation property in models with skip connections. The proposed method outperformed existing methods in terms of Insertion-Deletion Score while satisfying the conservation property.

In this study, we focus on the learning-to-rank physical objects task, which involves retrieving target objects from open-vocabulary user instructions in a human-in-the-loop setting. We propose MultiRankIt, which introduces the Crossmodal Noun Phrase Encoder to model the relationship between referring expressions and target bounding box, and the Crossmodal Region Feature Encoder to model the relationship between the target object and its surrounding contextual environment. Our model outperforms the baseline method in terms of mean reciprocal rank and recall@K.

We focus on the task of retrieving nail design images based on dense intent descriptions, which represent long and multi-layered user intent for nail designs. This is challenging because such descriptions specify flexibly created paintings and pre-manufactured embellishments, as well as visual characteristics, spatial relationships, higher-order themes, and overall impressions. Existing vision-and-language foundation models often struggle to capture the interplay between paintings and embellishments, failing to incorporate multi-layered intent descriptions. To address this, we propose NaiLIA, a method that enables the retrieval of nail design images that comprehensively align with descriptions with dense user intent. Our approach estimates confidence scores for images that align with a given description and can be considered as positive examples but are not explicitly labeled (unlabeled positives), and incorporates this score into the loss function. To evaluate NaiLIA, we constructed a benchmark consisting of 10,625 images collected from people with diverse cultural backgrounds. The images were annotated with long and dense intent descriptions given by over 200 annotators. Experimental results demonstrate that the proposed method outperforms standard methods by 20.9 points in terms of recall@10.

Speech impairments from conditions like Amyotrophic Lateral Sclerosis and muscular dystrophy severely restrict patient communication, affecting daily and social life. Decoding technology based on Electrocorticography (ECoG) is essential for supporting these patients' communication. In this study, we propose a novel architecture combining a specialized convolutional layer for electrode feature extraction and a retentive network for ECoG signal classification of motor imagery, outperforming all baselines in accuracy.