Published papers

In retinal blood vessel extraction through background removal, the vessels in a fundus image which appear in a higher illumination variance area are often missing after the background is removed. This is because the intensity values of the vessel and the background are nearly the same. Thus, the estimated background should be robust to changes of the illumination intensity. This paper proposes retinal blood vessel extraction using background estimation. The estimated background is calculated by using a weight surface fitting method with a high degree polynomial. Bright pixels are defined as unwanted data and are set as zero in a weight matrix. To fit a retinal surface with a higher degree polynomial, fundus images are reduced in size by different scaling parameters in order to reduce the processing time and complexity in calculation. The estimated background is then removed from the original image. The candidate vessel pixels are extracted from the image by using the local threshold values. To identify the true vessel region, the candidate vessel pixels are dilated from the candidate. After that, the active contour without edge method is applied. The experimental results show that the efficiency of the proposed method is higher than the conventional low-pass filter and the conventional surface fitting method. Moreover, rescaling an image down using the scaling parameter at 0.25 before background estimation provides as good a result as a non-rescaled image does. The correlation value between the non-rescaled image and the rescaled image is 0.99. The results of the proposed method in the sensitivity, the specificity, the accuracy, the area under the receiver operating characteristic (ROC) curve (AUC) and the processing time per image are 0.7994, 0.9717, 0.9543, 0.9676 and 1.8320 seconds for the DRIVE database respectively.

近年登場した距離センサーを搭載したデバイスの中でもその価格と汎用性ゆえに最も多く利用されているのがMicrosoftのKinectに代表されるRGB-Dカメラである．しかし，RGB-Dカメラから得られる３次元点群は多くのノイズを含んでおり，精密な距離情報の取得が求められるアプリケーションでの利用は依然として困難である．そこで本論文ではRGB-Dカメラから得られた３次元点群の観測ノイズの低減と補間を実時間で行う手法を提案する．本手法はRGB-Dカメラから得られた３次元点群からSuperpixel Segmentaiotnとラベリングを併用して平面で構成される小領域群を検出し，領域毎に平面方程式を推定し３次元点群をフィッティングさせることにより，もとの距離画像に含まれるノイズの低減と補間を行うものである．本手法では検出された領域内の点群の法線分布を算出し，その領域が平面の場合平面フィッティングを適用し曲面の場合Joint Bilateral Filterを適用することで，曲面分布した点群を保持しながら平面分布した点群のみを平面フィッティングすることができる．また，領域分割と平面フィッティングはすべて局所領域ごとに並列化できる手法を用いているため，本手法の実装はすべてGPUで行い実時間での処理が可能である．従来の手法と比べて，本手法はRGB-Dカメラから取得した点群に対して，曲面領域を保持したまま平面領域を平滑化することでノイズ低減と補間を行うことができる．

This paper discusses the method for the automatically generating stroboscopic image from a video image using a handheld camera comprehensively. The stroboscopic image can be used in order to measure stride length and speed of the runner based on the authors' previously proposed method, in which it is shown that measuring stride length and speed of the runner from the stroboscopic image is theoretically possible. The proposed automatic stroboscopic image generating method has two steps. The first step is generating a stitched background image from input video image. In the second step, runner images extracted from arbitrary frames are overlapped onto the stitched background image. Thus, this method can generate a stroboscopic image including the runners in arbitrary frames without overwriting them in some frames. These processes can be executed automatically, and these are noteworthy features. In the experiments under various environment to show the effectiveness of this method, it was confirmed that could generate a stroboscopic images well in respectively environment. It is also demonstrated that stride length and speed of the runner captured into the generated stroboscopic image can be measured with satisfactory accuracy by manually selected landing points in the stroboscopic image.

Recent advances of ToF depth sensor devices enables us to easily retrieve scene depth data with high frame rates. However, the resolution of the depth map captured from these devices is much lower than that of color images and the depth data suffers from the optical noise effects. In this paper, we propose an efficient algorithm that upsamples depth map captured by ToF depth cameras and reduces noise. The upsampling is carried out by applying plane based interpolation to the groups of points similar to planar structures and depth variance based joint bilateral upsampling to curved or bumpy surface points. For dividing the depth map into piecewise planar areas, we apply superpixel segmentation and graph component labeling. In order to distinguish planar areas and curved areas, we evaluate the reliability of detected plane structures. Compared with other state-of-the-art algorithms, our method is observed to produce an upsampled depth map that is smoothed and closer to the ground truth depth map both visually and numerically. Since the algorithm is parallelizable, it can work in real-time by utilizing highly parallel processing capabilities of modern commodity GPUs.

This paper presents a method to detect and track a paper map and overlay virtual data over it. First, we take a picture of the map and identify the road colors using our road area extraction method. Using the color information we can then extract in real time the roads from a video and use the position of the intersections to identify the map from a database without any further user input. The main issue is to get reliable intersection positions in real time even though the map can contain a lot of overlapping data. Finally, using the location of the intersections we can track and augment the map with virtual information. Our method enables the real time matching, 3D tracking and augmentation of a paper map based on a database that can contain several thousands of intersections coordinates.

本論文では，線分特徴の事前分布を考慮した自動運転車のためのSLAM (Simultaneous Localization and Mapping) を提案する．提案手法は，道路標示や建物など，線分を検出しやすい物体が存在する都市環境を走行する車両に搭載されたカメラにより撮影される動画像を入力とすることを前提とし，この動画像から検出される線分特徴の方向に関する制約を加えることによりSLAM の精度を向上させるものである．線分特徴の方向に関する制約を加えるため，本手法では道路標示中の線分が車両進行方向に対して平行，もしくは垂直であるものが多いことに着目し，それら2 つの線分間の角度分布が混合ガウス分布に則ると仮定し，バンドル調整による最適化に導入する．これにより，カメラの位置姿勢と線分特徴の位置推定の精度を高めることが期待できる．本手法の有効性を確かめるため，カメラを1 つ用いたシングルカメラシステムと，2 つ用いたマルチカメラシステムの両方に提案手法を適用し精度検証を行った．その結果マルチカメラによる従来手法の精度と，シングルカメラによる本手法の精度が同等となった．また提案手法により生成した路面地図は，線分の方向に関する制約を導入したことにより補正され，精度向上を確認した．

Visual inspection is a vital step for maintaining the quality of industrial parts. In visual inspection, the relation between the three factors, camera pose, lights direction and normal vector of a part is very important to detect abnormalities on industrial part. We propose a visual inspection method for industrial parts using an image sequence captured while rotating an industrial part using a rotating table. By rotating a part, we can easily change the relation between three factors. We track the points on part's surfaces in an image sequence. We discriminate abnormal surface points such as scratches from the normal ones based on pixel value transitions in an image sequence. For accurately tracking the points on the part surface, we propose a novel camera calibration method of the rotating table and the camera with a telecentric lens and a pose estimation method for the rotating part. We use angular invariant feature vectors for improving robustness against types and shapes of industrial parts to be inspected. We presented experimental results using real data of industrial parts and verified that the proposed method could detect scratches on surfaces.

We propose a novel diminished reality method which is able to (i) automatically recognize the region to be diminished, (ii) work with a single RGB-D sensor, and (iii) work without pre-processing to generate a 3D model of the target scene by utilizing SLAM, segmentation, and recognition framework. Especially, regarding the recognition of the area to be diminished, our method is able to maintain high accuracy no matter how the camera moves by distributing the viewpoints for each object uniformly and aggregating recognition results from each distributed viewpoint as the same weight. These advantages are demonstrated on the UW RGB-D Dataset and Scenes.

We present a method for generating images in which people are hidden from image sequences taken with a hand-held camera. Our method is basically used for privacy protection of people whose images are unintentionally captured in image sequences. We hide people from images by reconstructing a 3D model of background and projecting it to 2D images. By detecting the area in which people are present beforehand, we can reconstruct a 3D model of the background without people. In the experiment, We compare our method with some conventional approaches for diminished reality.

Diminished reality (DR) enables us to see through real objects occluding some areas in our field of view. This interactive display has various applications, such as see-through vision to visualize invisible areas, work area visualization in surgery and landscape simulation. In this paper, we propose two underlying problems in see-through vision, in which hidden areas are observed in real time. First, see-through vision methods require a common area to calibrate every camera in the environment. However, the field of view is limited and many approaches rely on a time-consuming calibration, sensors, or fiducial markers. Second, see-through vision applications assume that the background is planar to ease image alignment. We therefore present a place-and-play see-through vision system using a wide field-of-view RGB-D camera. We validated the accuracy and the robustness of our system and showed results in various environments to show the applicability.

現実世界から実物体を消し去る隠消現実感 (Diminished Reality; DR) 技術は，見通しの悪い交差点におけるSee-Through Visionなど，さまざま応用例が考えられている．しかし，そこにはいくつかの課題が存在した．第一に隠背景を観測するカメラと，体験者のカメラの直接的なカメラの位置関係を取得することが困難であること．第二に，屋外において隠背景が平面で近似できる場面でしか応用できなかったことである．本稿では，魚眼カメラと3D-LIDARを組み合わせた，高画角のRGB-Dカメラを用いて，両者のカメラの共通の視野を獲得することにより上記の問題解決を図る．また，屋外実験を通して，提案手法によるSee-Through Visionの実現可能性について議論する．

本研究では，物体に対する人間の把持姿勢を3D復元するシステムを提案する．ペットボトルやビンといった筒状の物体の蓋の上面に魚眼レンズを装着したカメラを付与し，肌色抽出やセグメンテーションといった画像処理を加えて指を認識する．取得した指位置をディジタルハンドモデルと組合せることで手全体の位置姿勢を復元する．本システムは，日用品の利用状況の計測に基づく製品改良や，スポーツのトレーニングに活用できる．

Thermal imaging has become a valuable tool in various fields for remote sensing and can provide relevant information to perform object recognition or classification. In this paper, we present an automated method to obtain a 3D model fusing data from a visible and a thermal camera. The RGB and thermal point clouds are generated independently by structure from motion. The registration process includes a normalization of the point cloud scale, a global registration based on calibration data and the output of the structure from motion, and a fine registration employing a variant of the Iterative Closest Point optimization. Experimental results demonstrate the accuracy and robustness of the overall process.

Hand-held tools are indispensable for efficient manual working in fields ranging from woodworking to surgery. In this paper, we present a diminished reality (DR) method to visualize areas occluded by hands and tools in various hand-working scenarios. We propose a redesigned existing arbitrary viewpoint image generation method for DR applications as a core DR background rendering to recover views without undesirable objects. We conducted quantitative and qualitative experiments using real data to validate the performance of the proposed method. The experimental results showed that our method runs in real-time (40.1 fps) and surpasses conventional methods (including image inpainting-based and geometry-based approaches) in terms of image similarity measures.

Since conventional feature descriptor algorithms depend on brightness values, those algorithms cannot obtain point correspondences between aerial and map images where brightness values do not correspond but geometric structures do correspond. The research in this paper focused on obtaining point correspondences between those images. This study proposes a novel method by which curve features, edge shape, and road extraction by a CNN, independent on brightness value, produce robustness in image matching. Its effectiveness was confirmed through experiments on Massachusetts Roads Dataset.

This paper proposes a method for reconstructing 3D ball trajectories by using multiple temporally and geometrically uncalibrated cameras. To use cameras to measure the trajectory of a fast-moving object, such as a ball thrown by a pitcher, the cameras must be temporally synchronized and their position and orientation should be calibrated. In some cases, these conditions cannot be met, e.g., one cannot geometrically calibrate cameras when one cannot step into a baseball stadium. The basic idea of the proposed method is to use a ball captured by multiple cameras as a corresponding point. The method first detects a ball. Then, it estimates temporal difference between cameras. After that, the ball positions are used as corresponding points for geometrically calibrating the cameras. Experiments using actual pitching videos verify the effectiveness of our method.

本論文では，線分で構成された3次元形状の冗長な線分を除去する手法を提案する．実験では従来手法と提案手法の3次元形状を用いてカメラ位置姿勢推定を行い，その精度を保ちつつ処理時間を短縮したことを示す．また応用例の一つとして透視映像生成を行った．

In this paper, we review diminished reality (DR) studies that visually remove, hide, and see through real objects from the real world. We systematically analyze and classify publications and present a technology map as a reference for future research. We also discuss future directions, including multimodal diminished reality. We believe that this paper will be useful mainly for students who are interested in DR, beginning DR researchers, and teachers who introduce DR in their classes.

Live instructors perspective videos are useful to present intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructors hands often hide the work area. In this demo, we present a diminished hand for visualizing the work area hidden by hands by capturing the work area with multiple cameras. To achieve the diminished reality, we use a light field rendering technique, in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from the multiple viewpoint images.