Published papers

Placing everyday objects in designated areas, such as placing a glass on a table, is a crucial task for Domestic service robots (DSRs). In this paper, we propose a physical reasoning method about collisions in placement tasks. The proposed method, Transformer PonNet, predicts the probability of a possible collision and visualizes areas involved in the collision. Unlike existing methods, Transformer PonNet can be applied to objects whose models are unavailable. We propose a novel Transformer Perception Branch that handles relationships among features more complex than simple self-attention. We built simulation and physical datasets using a DSR, and validated our method on the datasets. We obtained an accuracy of 82.5% for the physical dataset.

The purpose of this paper is to extend the dataset based on a cross-modal generative language generation model. We propose a Case Relation Transformer (CRT) that generates a fetching instruction sentence from an image, such as ``Move the blue flip-flop to the lower left box.'' Unlike existing methods, CRT uses Transformer to capture the visual and geometric features of objects in an image. The Case Relation Block allows the CRT to process the object. We conducted comparative experiments and human evaluations. Experimental results showed that CRT outperformed the baseline methods.

Currently, domestic service robots have an insufficient ability to interact naturally through language. This is because understanding human instructions is complicated by a variety of ambiguities and missing information. Existing methods are insufficient to model reference expressions that specify relationships between objects. In this paper, we propose Target-dependent UNITER, which learns directly the relationship between the target object and other objects by focusing on the relevant regions within an image, instead of the whole image. Our model is validated on two standard datasets, and the results show that Target-dependent UNITER outperforms the baseline method in terms of classification accuracy.

Asynchronous Advantage Actor-Critic (A3C) is a representative method of deep reinforcement learning and is possible to solve difficult tasks such as games and robot control. However, it is difficult for deep reinforcement learning including A3C to understand and to explain the reason of action selection. To address this problem, we propose a method called a Mask-Attention A3C, which performs mask processing on feature map of Policy branch using attention map. The propose method can obtain an attention map that is useful for a visual explanation of agent behavior. In the experiment with Atari2600, we compare the scores in each game and demonstrate that the attention map obtained from our method is useful for visual explanation. In addition, we evaluate the explainability of obtained attention map using the scores of each game by changing the attention region.

Domestic service robots (DSRs) are a promising solution to the shortage of home care workers. Nonetheless, one of the main limitations of DSRs is their inability to naturally interact through language. Recently, data-driven approaches have been shown to be effective for tackling this limitation, however, they often require large-scale datasets, which is costly. Based on this background, we aim to perform automatic sentence generation for fetching instructions, e.g., ``Bring me a green tea bottle on the table.'' This is particularly challenging because appropriate expressions depend on the target object, as well as its surroundings. In this paper, we propose a method that generates sentences from visual inputs. Unlike other approaches, the proposed method has multimodal attention branches that utilize subword-level attention and generate sentences based on subword embeddings. In the experiment, we compared the proposed method with a baseline method using four standard metrics in image captioning. Experimental results show that the proposed method outperformed the baseline in terms of these metrics.

We consider the problem of searching scientific data from vast heterogeneous scientific data repositories. This problem is challenging because scientific data contain relatively little text information compared to other search targets such as web pages. On the other hand, the metadata in scientific data contain other characteristic information such as spatio-temporal information. Although using this information make it possible to improve the search performance, many widely adopted scientific data search engines use this information exclusively for narrowing down search results. In this paper, we propose a novel query generation method using spatial, temporal, and text information based on pseudo relevance feedback. The proposed method generates new spatio-temporal queries from the initial search results. By using these queries, the search results are reranked such that more related results obtain higher rank. The experimental results show that the proposed method outperforms a baseline method when search targets do not have rich text information. (c) 2016 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.