Published papers

For realizing autonomous vehicle driving and advanced safety systems, it is necessary to achieve accurate vehicle localization in cities. This paper proposes a method of accurately estimating vehicle position by matching a map and line segment features detected from images captured by a camera. Features such as white road lines, yellow road lines, road signs, and curb stones, which could be used as clues for vehicle localization, were expressed as line segment features on a two-dimensional road plane in an integrated manner. The detected line segments were subjected to bird’s-eye view transformation to transform them to the vehicle coordinate system so that they could be used for vehicle localization regardless of the camera configuration. Moreover, an extended Kalman filter was applied after a detailed study of the line observation errors for realizing real-time estimation. Vehicle localization was tested under city driving conditions, and the vehicle position was identified with sub-meter accuracy.

This paper presents a method for the geolocation of printed maps. It enables the registration of unprepared maps with a Geographical Information System (GIS) database, and can for example be used as a first step to augment an unknown map. We define and match local road pattern descriptors, which are similar to SIFT descriptors [6], but adapted to the case of simple textureless line segments. Using a processing pipeline commonly encountered in the feature-point based matching of texture images - composed of offline description and indexing, followed by an online description, matching and robust transformation estimation - we show that local descriptors can successfully register unprepared maps using only geographic features and no texture information. Our method is scale and rotation invariant, and circumvents the two hurdles that are the level-of-detail, and the changing colormaps and textures, allowing the processing of large classes of printed maps.

This paper proposes a diminished reality method for hiding a pedestrian from a hand-held camera sequence. In the proposed method, the pedestrian is extracted from each frame of input video sequence using HOG based people detector. The detected area is excluded from each frame by masking, and then two frames are stitched for generating a stitched background image. By using the stitched background image, the pedestrian's mask for hiding is made. It is used for overlaying the background pixel to the pedestrian in the original frame. The presented experimental result is demonstrate the effectiveness of the proposed method.

Diminished reality (DR) research aims to develop methods to visually hide objects from real scenes. One or more cameras capture a scene where some objects, called the occluding objects, are 'diminished' by an DR system. To do this in real time we need to use multiple cameras. We introduce a novel multi-camera DR system utilizing an RGB-D camera to hide arbitrary trackable objects from a scene. In our case, the scene background does not have to be planar for the system to work and we can handle scene changes in real time. Additonally, we introduce color correction technique to merge image content originating from different cameras and also demonstrate method to repair images with partially missing content.

We propose a method of ego-motion estimation for a self-driving vehicle using multiple cameras. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimizes the odometry. The result shows higher estimation accuracy when compared to other matching method.

We present a novel combination photometric stereo which can estimate surface normals precisely even for images including shadows and specular reflection. We can use photometric stereo if there are more than three input images. Therefore we can employ photometric stereo with nC3 combinations for n input images. We make 3D distribution of albedos and surface normals estimated from pixel intensities of nC3 pixel combinations. In the distribution, we define a novel value “compactness” to distinguish pixels which are included in neither shadows nor specular reflection from pixels which are included in shadows or specular reflection. Through experimental results, we demonstrate that the proposed method can estimate surface normals in the presence of shadows and specular reflection. Moreover the proposed method is superior to previous works in better accuracy.

The 3D trajectory of the ball is one of the most important performance indicators in ball games. However, it often cannot be measured because of the requirements: synchronized cameras and control points within the playing field. Thus, this paper proposes a method that reconstructs the 3D trajectory of a ball with unsynchronized cameras. The proposed method consists of ball detection, camera calibration, and trajectory reconstruction. At first, ball candidates are detected on the basis of their appearance. Balls are then extracted from the candidates on the basis of their motion. After the extraction, ball trajectories are reconstructed in image spaces. The corresponding points in two views are estimated on the basis of the ball trajectory in image spaces and the temporal offset between cameras, which is supposed to be recorded with a few erroneous frames. The matrix including the geometrical relationship between cameras, namely the fundamental matrix, is then computed from the corresponding points. The estimated one is inaccurate due to the error of the temporal offset. The key feature of this method is to optimize the temporal offset and fundamental matrix simultaneously. After the optimization, the geometrical relationship between cameras is computed from the matrix. Balls are extracted again by using the temporal offsets and the geometrical relationship between cameras. A ball trajectory is finally reconstructed as connected trajectories that are separated at collisions. It is experimentally demonstrated that the proposed method accurately calibrates cameras and successfully reconstructs the 3D trajectory of the ball.

We present a system that can accurately visualize changing (dynamic) areas captured by an RGB-D camera at an alternative viewpoint. Our system does not make assumptions about the scene 3D structure and works in real time.

In this paper, we propose a framework for end-to-end transmission system of free-viewpoint video. On soccer game taken multiple cameras, we adapt efficient multiple object tracking method to our system and generate free-viewpoint video represented billboard.

Sudden brake seems to be unavoidable in driving practice to prevent from accidence. The sudden brake may often cause falling risks to on-board passengers in public transport and even threaten the lives of the senior. Thus, this paper presents an omnidirectional vision system to assist bus drivers in ensuring the safety for passengers. To this end, we configure an auxiliary camera in the boarding gate to recognize the age of passengers and an omnidirectional camera on the ceiling of the bus to detect and track the passengers. Their trajectories and positions are mapped into a bus layout, together with their age information transferred from the auxiliary camera. This bus layout is shown on a small display in front of bus drivers. Instead of looking to the mirror to guess and gather information of passengers, the driver can collect such necessary information for safe driving. We set up experiments on a real bus and demonstrate the system by preliminary results.

We propose a digital content viewing system which visualizes the polymorphic relationships among digital content interactively and uses it as a visual interface. The relationships are expressed by associating of two content and grouping and shown by directed graphs. On the graphs, nodes mean the content files or the groups, and edges mean correspondences or group members. The graphs are archived as relation catalogues separately from content files. We visualize the relations of content on a directed graph. To visualize the graphs, we use a method of locating nodes by horizontal directions and height. Nodes linked with a target node will be located on a horizontal circle and their height is decided by their relation catalogue. After deciding the node location, nodes and links are drawn using 3D graphics and animation. The visualization result called 'polymorphic view' is used as a visual interface for selecting content. Selecting content on the visualized graph will expand new links and new content nodes additionally. We can show another content by tracing graph links. It becomes easy to understand the relations between digital content, because we can change the direction and size of the directed graph. We developed a content viewer system (MoSaIC-II) using the polymorphic view as a visual interface and we can see the usability of the polymorphic view.

In this paper, we propose a novel method for generating 3D line segment based model from an image sequence taken with a RGB-D camera. Constructing 3D geometrical representation by 3D model is essential for model based camera pose estimation that can be performed by corresponding 2D features in images with 3D features of the captured scene. While point features are mostly used as such features for conventional camera pose estimation, we aim to use line segment features for improving the performance of the camera pose estimation. In this method, using RGB images and depth images of two continuous frames, 2D line segments from the current frame and 3D line segments from the previous frame are corresponded. The 2D-3D line segment correspondences provide camera pose of the current frame. All of 2D line segments are finally back-projected to the world coordinate based on the estimated camera pose for generating 3D line segment based model of the target scene. In experiments, we confirmed that the proposed method can successfully generate line segment based models, while 3D models based on the point features often fail to successfully represent the target scene.

卓球においてボールの回転は戦術的に重要な要素と考えられている．しかし，従来の回転計測法は，表面全体にマークを追加したボールを高速度カメラを用いて撮影するものであり，撮影されるデータ量が膨大になる，あるいは試合と同じ条件下の回転による攻防を分析できない等の問題がある．ボールの運動モデルに基づき軌跡から回転を推定する方法も提案されているが，正確な回転計測は難しい．そこで本研究では，高速度カメラを用いることを前提とせず，且つ試合で用いられる卓球ボールを対象とできる回転計測法を提案する．提案手法ではまずボールのマークを検出し，検出されたマーク領域の重心から回転軸を算出する．高速度カメラを用いないことから，回転軸周りの回転方向及び隣接フレーム間の回転角速度は一意には求まらないが，回転角速度の全ての候補値に対応する軌跡を運動モデルに基づきシミュレートし，実測された軌跡に最も近い軌跡を探索することで最も真値に近い回転角速度を選択する．回転する卓球ボールを対象とした実画像実験により，本手法が実用的な精度で回転軸及び回転角速度を計測できることを示す．

本論文では，カメラ位置姿勢推定のための3次元線分モデルを，対象シーンをRGB-Dカメラで撮影するだけで簡易に生成する手法を提案する．従来の特徴点マッチングによるモデル生成は，特徴点が検出されない場合は失敗するという問題点があった．そのような場面でも線分は安定して検出されるため，3次元線分モデルの生成法が求められている．本提案手法では線分特徴記述子を用いた線分マッチングにより，簡易に，そして3次元情報が必要となるその場で3次元線分モデルを生成する．まず，入力されたRGB画像と距離画像から前フレームの3次元線分と現フレームの2次元線分の対応付けを行う．この2次元3次元線分対応から現フレームのカメラ位置姿勢を推定し，フレーム中の2次元線分を3次元空間上へ逆投影して3次元線分モデルとする．実験により，提案手法が特徴点マッチングではモデルが生成できないシーンでも正確なモデルを生成できることを確認した．

近年、温度カメラは、機械設備・建築物の検査や，人間や動物の体温の検査等，多くの目的に利用されるようになってきた．本論文では，RGB-Dカメラで取得した3次元環境モデルを利用して、このモデル上に温度分布をレンダリング表示することにより，対象環境の温度分布を自由な視点から観察可能なシステムを提案する．本システムでは，３Ｄ環境モデルを利用したカメラ位置姿勢のトラッキング手法に基づき，事前に取得しておいた温度分布に対して，現在の温度変化を表示することも可能である．提案システムの有効性を検証するために，幾つかのシーンに対して本システムを利用して表示した温度分布や温度変化分布を示す．

In this paper, we present a system for visualizing temperature changes in a scene using an RGB-D camera coupled with a thermal camera. This system has applications in the context of maintenance of power equipments. We propose a two-stage approach made of with an offline and an online phases. During the first stage, after the calibration, we generate a 3D reconstruction of the scene with the color and the thermal data. We then apply the Viewpoint Generative Learning (VGL) method on the colored 3D model for creating a database of descriptors obtained from features robust to strong viewpoint changes. During the second online phase we compare the descriptors extracted from the current view against the ones in the database for estimating the pose of the camera. In this situation, we can display the current thermal data and compare it with the data saved during the offline phase.

We propose a method of marker-less AR which uses line segment feature. Estimating camera poses is important part of AR system. In most of conventional marker-less AR system, feature point matching between a model and its image is required for the camera pose estimation. However, a sufficient number of correspondence points is not always detected to estimate accurate camera poses. To solve this problem, we propose the use of line segments feature that can possibly be detected even when only a few feature points can be detected. In this paper, we propose a marker-less AR system that uses line segment features for camera pose estimation. In this system, we propose a novel descriptor of the line segment feature for achieving fast computation of the camera pose estimation. We also construct a database which contains a k-d tree of line feature and 3D line segment position for nding 2D-3D line segment correspondences from input image and the database, so that we can estimate the camera pose and perform AR. We demonstrated that our proposed method can estimate the camera pose and provide robust marker-less AR in the situation where point matching method fails.

n this paper, we propose a relighting system combined with a n illumination estimation method using RGB- D camera. Relighting techniques can achieve the photometri c registration of composite images. They often need illumination environments of the scene which include a target object and the background scene. Some relighting methods obtain the illumination environments b eforehand. In this case, they cannot be used under the unknown dynamic illumination environment. Some on-lin e illumination estimation methods need light probes which can be invade the scene geometry. In our method, the illumination environment is estimated from pixel intensity, normal map and surface reflectance bas ed on inverse rendering in on-line processing. The normal map of the arbitrary object which is used in the ill umination estimation part and the relighting part is calculated from the denoised depth image on each frame. Re lighting is achieved by calculating the ratio for the estimated Illumination environment of the each scene. T hus our implementation can be used for dynamic illumination or a dynamic object.