Published papers

The prediction of solar flares is essential for reducing the potential damage to social infrastructures that are vital to society. However, predicting solar flares accurately is a very challenging task. In this paper, we propose a solar flare prediction model, Flare Transformer, which handles both images and physical features through the Magnetogram Module and the Sunspot Feature Module. We introduce the transformer attention mechanism to model the temporal relationships. We also introduce a new differentiable loss function to balance the two major metrics of the Gandin-Murphy-Gerrity score and Brier skill score. Comparative experiments using Gandin-Murphy-Gerrity score and true skill statistics as metrics showed that the proposed method achieves better performance than baseline methods and human experts.

Explanation generation for transformers enhances accountability for their predictions. However, there have been few studies on generating visual explanations for the transformers that use multidimensional context, such as LambdaNetworks. In this paper, we propose the Lambda Attention Branch Networks, which attend to important regions in detail and generate easily interpretable visual explanations. We also propose the Patch Insertion-Deletion score, an extension of the Insertion-Deletion score, as an effective evaluation metric for images with sparse important regions. Experimental results on two public datasets indicate that the proposed method successfully generates visual explanations.

In this paper, we focus on a vision-and-language task in which a robot is instructed to execute household tasks. We propose Moment-based Adversarial Training (MAT), which uses two types of moments for perturbation updates in adversarial training. We introduce MAT to the embedding spaces of the instruction, subgoals, and state representations to handle their varieties. We validated our method on the ALFRED benchmark, and the results demonstrated that our method outperformed the baseline method for all the metrics on the benchmark.

In this paper, our aim is to generate a caption about a future event. We propose the Relational Future Captioning Model (RFCM), a crossmodal language generation model for the future captioning task. The RFCM has the Relational Self-Attention Encoder to extract the relationships between events more effectively than the conventional self-attention in transformers. We conducted comparison experiments, and the results show the RFCM outperforms a baseline method on two datasets.

In this study, we develop a multimodal language comprehension model that allows domestic service robots to understand object fetching instructions. We propose a multimodal language understanding model, Funnel UNITER, which gradually reduces the dimensions of the query, key, and value in each transformer layer to reduce the computational cost of self-attention. We also built a new dataset for the multimodal language understanding for fetching instruction (MLU-FI) task called the ALFRED-fetch dataset. Our model outperformed the baseline method in both classification accuracy and training time.