Published papers

2017 IEEE Winter Applications of Computer Vision Workshops (WACVW)

In this paper, we propose a line-based SLAM from an image sequence captured by a vehicle in consideration with the directional distribution of line features that detected in an urban environments. The proposed SLAM is based on line segments detected from objects in an urban environment, for example, road markings and buildings, that are too conspicuous to be detected. We use additional constraints regarding the line segments so that we can improve the accuracy of the SLAM. We assume that the angle of the vector of the line segments to the vehicle’s direction of travel conform to four-component Gaussian mixture distribution. We define a new cost function considering the distribution and optimize the relative camera pose, position, and the 3D line segments by bundle adjustment. In addition, we make digital maps from the detected line segments. Our method increases the accuracy of localization and revises tilted lines in the digital maps. We implement our method to both the single-camera system and the multi-camera system. The accuracy of SLAM, which uses a single-camera system with our constraint, works just as well as a method that uses a multi-camera system without our constraint.

In this paper, we propose a method to generate a three-dimensional (3D) thermal map by overlaying thermal images onto a 3D surface reconstructed by a monocular RGB camera. In this method, we capture the target scene moving both an RGB camera and a thermal camera, which are mounted on the same zig. From the RGB image sequence, we reconstruct 3D structures of the scene by using Large-Scale Direct Monocular Simultaneous Localization and Mapping (LSD-SLAM), on which temperature distribution captured by the thermal camera is overlaid, thus generate a 3D thermal map. The geometrical relationship between those cameras is calibrated beforehand by using a calibration board that can be detected by both cameras. Since we do not use depth cameras such as Kinect, the depth of the target scene is not limited by the measurement range of the depth camera; any depth range can be captured. To demonstrating this technique, we show synthesized 3D thermal maps for both indoor and outdoor scenes.

It is known that humans recognize objects using combinations and positional relations of primitive shapes. The first step of such recognition is to recognize 3D primitive shapes. In this paper, we propose a method for primitive shape recognition using superquadric parameters with a metric learning method, large margin nearest neighbor (LMNN). Superquadrics can represent various types of primitive shapes using a single equation with few parameters. These parameters are used as the feature vector of classification. The real objects of primitive shapes are used in our experiment, and the results show the effectiveness of using LMNN for recognition based on superquadrics. Compared to the previous methods, which used k-nearest neighbors (76.5%) and Support Vector Machines (73.5%), our LMNN method has the best performance (79.5%).

Recent years have seen the introduction of service robots as waiters or waitresses in restaurants and cafes. In such venues, it is common for customers to visit in groups as well as for them to engage in conversation while eating and drinking. It is important for cyber serving staff to understand whether they are eating and drinking, or not, in order to wait on tables at appropriate times. In this paper, we present a method by which the robots can recognize eating and drinking actions performed by individuals in a group. Our approach uses the positions of joints in the human body as a feature and long short-term memory to achieve a recognition task on time-series data. We also used head directions in our method, as we assumed that it is effective for recognition in a group. The information garnered from head directions and joint positions is integrated via logistic regression and employed in recognition. The results show that this yielded the highest accuracy and effectiveness of the robots’ tasks.

Understanding the anatomy of the human body is vital for everyone working in the medical domain. Augmented reality (AR) systems for anatomy teaching, which display virtual information directly on top of a users' body, have proven to facilitate mental mapping compared to traditional teaching paradigms. In this paper, we explore the potential of diminished reality (DR) in the context of anatomy education. As a first necessary step to achieving a DR anatomy education system, parts of the human body have to be extracted and diminished from the video stream. Our system diminishes either the arm or head of the user by projecting a background image recovered using RGB-D cameras. Such a system, if combined with an accurate overlay of virtual counterparts, could potentially improve the learning effect by attracting the users' attention to the virtual information and improve visual perception by avoiding the well-known floating effect of AR.

Instructor's perspective videos are useful for presenting intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructor's arms often obstruct the trainee's view of the work area. In this article, we present a diminished reality method for visualizing the work area hidden by an instructor's arms by capturing the work area with multiple cameras. To achieve such diminished reality, we propose detour light field rendering (DLFR), in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from multiple viewpoint images. In DLFR, the camera blending field used in an existing freeviewpoint image generation method known as unstructured lumigraph is re-designed based on our use cases. In this re-design, lesser weights are given to light rays as they pass close to given penalty points. Experimental results demonstrate that using DLFR, the appearance of an undesirable object can be removed from an image in real time.

Camera pose estimation with respect to target scenes is an important technology for superimposing virtual information in augmented reality (AR). However, it is difficult to estimate the camera pose for all possible view angles because feature descriptors such as SIFT are not completely invariant from every perspective. We propose a novel method of robust camera pose estimation using multiple feature descriptor databases generated for each partitioned viewpoint, in which the feature descriptor of each keypoint is almost invariant. Our method estimates the viewpoint class for each input image using deep learning based on a set of training images prepared for each viewpoint class. We give two ways to prepare these images for deep learning and generating databases. In the first method, images are generated using a projection matrix to ensure robust learning in a range of environments with changing backgrounds. The second method uses real images to learn a given environment around a planar pattern. Our evaluation results confirm that our approach increases the number of correct matches and the accuracy of camera pose estimation compared to the conventional method.

Reconstruction of three-dimensional (3D) models from images is currently one of the most important areas of research in computer vision. In this paper, we propose a method to recover 3D models using the minimum number of line segments. By using structure-from-motion, the proposed method first recovers a 3D model of line segments detected from an input image sequence. We then detect overlapping 3D line segments that redundantly represent a single line structure so that the number of 3D line segments representing the target scene can be reduced without losing the detailed geometry of the structure. We apply matching and depth information to remove redundant line segments from the model while keeping the necessary segments. In experiments, we confirm that the proposed method can greatly reduce the number of line segments. We also demonstrate that the accuracy and computational time for camera pose estimation can be significantly improved with the 3D line segment model recovered by the proposed method. Moreover, we have applied the proposed method to see through occluded areas.