Published papers

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. We conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that our method outperforms the baseline method in terms of language comprehension accuracy and the object grasping and placing actions are achieved with success rates of more than 90%.

Image captioning studies rely heavily on automatic evaluation metrics such as BLEU and METEOR, which are based on n-grams. However, these metrics have shown poor correlation with human evaluations, leading to the proposal of alternative metrics such as JaSPICE. JaSPICE has only been validated for a general image captioning task without an error analysis. In this paper, we analyze JaSPICE for a fetching instruction generation task and identify its errors for an image captioning task. We conducted experiments on STAIR Captions and PFN-PIC datasets and JaSPICE outperformed the baseline metrics on the correlation coefficient with human evaluation.

In the field of image captioning, constructing automatic evaluation metrics that align closely with human judgment is crucial for effective model development. A key challenge in this field is addressing hallucinations, which are instances where models generate words unrelated to the image, a frequent issue in image captioning. Existing metrics often fail to manage hallucinations, primarily due to their limited capability in contrasting candidate captions against a diverse range of reference captions. To overcome this, we propose DENEB, a novel metric for image captioning, specifically robust to hallucinations. DENEB incorporates the Sim-Vec Transformer, a mechanism capable of processing multiple references and extracting similarity vectors effectively. Additionally, to train DENEB, we have expanded the Polaris dataset to create Polaris2.0, significantly enhancing supervised automatic evaluation metrics. Our dataset comprises 32,978 images and 32,978 human judgments from 805 annotators. Our approach achieved state-of-the-art performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and the Polaris 2.0 dataset, thereby demonstrating its effectiveness and robustness to hallucinations.

For enhancing model performance in object manipulation tasks, high-performance prediction mechanisms for task success are crucial. However, existing methods are still insufficient in performance. Moreover, existing prediction mechanisms are designed to address only specific tasks, making it challenging to accommodate a diverse range of tasks. Therefore, our study aims to develop a task success prediction mechanism that can handle multiple object manipulation tasks. A key novelty of the proposed method is the introduction of the λ-Representation, which preserves all types of visual features: visual charactaristics such as colors and shapes; features aligned with natural language; features structured through natural language. For the experiments, we newly built datasets for task success prediction in object manipulation tasks based on the RT-1 dataset and VLMbench. The results show that the proposed method outperforms all baseline methods in accuracy.

In this paper, we address the task where domestic service robots perform object manipulation within a domestic environment following user instructions. Since objects are often moved in daily life, it is important to efficiently perform periodic searches to obtain the latest object positions. The more images collected, the more objects can be located, but this is time-consuming. Focusing on a small number of images can reduce the time required, however, the amount of information obtained is limited by the possibility of obtaining images of walls, corridors with few objects, and so on. Most existing methods do not properly account for the possibility that everyday objects may move to different positions at different points in time. In this paper, we propose an observation pose optimization method that takes advantage of object presence maps and submodularity. The proposed method outperformed the baseline method in terms of the ratio of everyday objects that can be observed in the observations.

In home environments, where the location of objects frequently changes, it is important for robots to quickly and accurately grasp the latest positions of these objects. Therefore, this study deals with the OSMI-3D task, a task that involves identifying target objects based on instructions given by users. We propose a method for efficiently manipulating objects in a home environment using reference expression segmentation based on 3D point cloud data, utilizing both a visual foundation model and a multimodal LLM. The main novelty of this study is the introduction of a Scene Narrative Module. This module combines the multimodal LLM with existing image feature extractors to extract structural features from images while meditating language. In experiments, our method demonstrated superior performance over traditional baseline methods in terms of mean IoU and precision at 0.5-0.9, confirming its effectiveness in the OSMI-3D task.

In this paper, we focus on the learning-to-rank physical objects task. In this task, images of objects within large-scale indoor environments are ranked based on open-vocabulary user instructions. We introduce the GREP module to construct visual features considering image, target object, relative positions, and pixel granularities. Additionally, we introduce the RCS module to efficiently learn from redundant images taken in the indoor environment. Our method outperformed baseline methods on the newly constructed YAGAMI dataset and an extended LTRRIE-subset, showing significant improvements in the standard metrics.

The practical use of domestic support robots that can naturally communicate with users to assist with daily tasks is one promising solution for support recipients. However, the ability of domestic support robots to understand natural language instructions and properly perform daily tasks is currently insufficient. In this paper, we aim to construct methods for robots to execute instructions given in natural language for fetch-and-carry tasks. Therefore, we propose a method that can be applied to tasks where multiple target objects and target areas are input and the best candidate is predicted by expanding existing methods. Additionally, by incorporating an instruction generation model, we make it possible to conduct simulations on-the-fly. In experiments, we achieved performance that exceeded existing methods in understanding instructions and confirmed that the task was executed properly.

We focus on the task of identifying target objects in domestic environments according to free-form natural language instructions. In this study, we propose a novel transfer learning approach for multimodal language understanding, Prototypical Contrastive Transfer Learning (PCTL) which uses a new contrastive loss, Dual ProtoNCE. Our experiment demonstrated that PCTL outperformed existing methods.

In this study, we develop a DSR that transports everyday objects to specified pieces of furniture based on open-vocabulary instructions. Our approach focuses on retrieving images of target objects and receptacles from pre-collected images of indoor environments. We propose a multimodal model, which learns diverse and robust representations from among positive, unlabeled positive, and negative samples. The experimental results demonstrate that our model outperformed existing baseline models across standard image retrieval metrics. Moreover, we performed physical experiments using a DSR to evaluate the performance of our approach in a zero-shot transfer setting. The experiments involved the DSR to carry objects to specific receptacles based on open-vocabulary instructions, achieving an overall success rate of 75%.

Task success prediction is important for open-vocabulary object manipulation tasks by manipulators, because it can enhance the reliability and efficiency of manipulations. In particular, it is highly convenient for efficient task execution if success prediction can be performed on-the-fly during the object manipulation. We propose a framework that extends Contrastive λ-Repformer, which determines task success based on images before and after open-vocabulary object manipulation and a instruction, to real-time task success prediction for open-vocabulary manipulation. Our framework detects the timing of successful object manipulation by focusing on minute changes between images, which is conducted by contrasting multi-level aligned representation for images in the initial state and images at any given time. Experimental results confirm that real-time success prediction could be performed using this framework.

In this study, we aim to develop a domestic service robot (DSR) that carries an everyday object to a piece of furniture by retrieving images of target objects and receptacles from collected images of an environment, based on an open-vocabulary instruction. We propose a multimodal model that retrieves both target objects and receptacles individually using a single model based on the switching mechanism via large language models. The experimental results show that our method outperformed baseline methods on the newly-built datasets in terms of the standard metrics. Furthermore, our method achieved task success rates of more than 80% in the physical experiments.

In this work, we propose a Trimodal Navigable Region Segmentation Model (TNRSM), which can handle three modalities: image, navigation instruction, and semantic segmentation masks. We validated our model on the Talk2Car-Regseg dataset. The results demonstrated that our method outperformed the baseline method in all evaluation metrics in the Referring Navigable Regions task.

ブレイン・コンピュータ・インターフェース（BCI）は，脳とコンピュータ間の直接的な通信を可能とするシステムの総称であり，その応用範囲は神経変性疾患の診断，脳機能マッピングの深化，ロボットの制御，没入型ゲーミングデバイスの開発など，多岐にわたる．特に，頭皮脳波および頭蓋内脳波を用いたBCIは，その優れた時間分解能および携帯性から実社会への広い普及が期待されている．そこで，本研究では，運動想起時および企図時に計測された頭蓋内脳波および頭皮脳波の分類において，深層状態空間モデルを拡張し，時間・空間・周波数の統合的な依存関係を捕捉可能な新しいアーキテクチャを提案する．本タスクを扱う3つのベンチマークにおいて，提案手法がベースライン手法を上回る結果を得た．さらに，提案手法の視覚的説明より，神経生理学的に有用な特徴を捉えられたことが確認された．

In this paper, we propose Co-Scale Cross-Attentional Transformer for Rearrangement Target Detection, where the model generates a change mask for objects that should be rearranged. We intorduce the Serial Encoder which consists of CoaT serial blocks and the Cross-Attentional Encoder which models the relationship between the goal and current states. We validated our method on the new dataset, and the results demonstrated that our method outperformed a baseline method on mean IoU and F1-score.