Published papers

n this paper, we propose a system to achieve Augmented Reality (AR) on a smartphone. In this system, we assume a fixed RGB-D camera connected to a server is installed in the environment, and perform AR based on the 3D environment shape got by the RGB-D camera in real-time. On the smartphone side, it sends the image captured by its camera, and receives the output image processed to perform AR by the server. On the server side, it sends back the image that virtual objects are superimposed on. At this time, by constructing the 3D environment model in real-time using the RGB-D camera, it is possible to calculate the smartphone pose and the interaction between the virtual objects and the environment properly. It is also possible to change the lighting virtually. Our experiment revealed it is possible to superimpose virtual objects on the smartphone in approximately 8fps.

While HMAX features have been proved to have excellent performance in image categorization tasks, the computational cost of recognition is expensive. If we aim to apply the HMAX features to object localization tasks, in which the categorization tasks are repeatedly performed by sliding windows, their processing time increases enormously. In this paper, we propose a method for speed up in computation of object localization based on HMAX features. We found that the HMAX features cause specific redundancies in the sliding window approach. The speed up is achieved by eliminating the redundancies in our method. The results from experiments using the University of Illinois-Urbana-Champaign (UIUC) car dataset and the face detection dataset benchmark (FDDB) indicate that this modification improved processing speeds significantly with insignificant reductions in precision.

We have proposed a mixed reality based painting system. In this paper, we tackle a problem in our previous painting system that fully relies on magnetic sensor that is attached on the canvas, the users needed to detach and attach a sensor on the canvas during painting when they want to switch the canvas. Instead of that, in this paper, we aim to automatically detect the shape of the canvas for registration purpose. Using the shape or region detection method such as MSER (maximally stable extremal regions), we detect and track the shape on the canvas on the captured camera image. We then compute the camera pose for virtually overlay the painting result. Using the brush device, we can draw and paint freely on the tracked canvases. We show that using visual based tracking method, we can generate the equivalent result compared to the result of using the sensor.

We present the preliminary results of our proposal: a region-based detection and tracking method of arbitrary shapes. The method is designed to be robust against orientation and scale changes and also occlusions. In this work, we study the effectiveness of sequence of shape descriptors for matching purpose. We detect and track surfaces by matching the sequences of descriptor so called relevance measures with their correspondences in the database. First, we extract stable shapes as the detection target using Maximally Stable Extreme Region (MSER) method. The keypoints on the stable shapes are then extracted by simplifying the outline of the stable regions. The relevance measures that are composed by three keypoints are then computed and the sequences of them are composed as descriptors. During runtime, the sequences of relevance measures are extracted from the captured image and are matched with those in the database. When a particular region is matched with one in the database, the orientation of the region is then estimated and virtual annotations can be superimposed. We apply this approach in an interactive task support system that helps users for creating paper craft objects.

We propose a vehicle make and model recognition method for the smart phone, and implemented it onto them. Our method identifies the make and model from the variety of viewpoints while the conventional methods for VMMR work only for the frontal or rear view images. This method enables the users to take the pictures from a certain range of angles. Our method uses SIFT, that has scale and rotation invariance to solve the alignment issue. It creates the pseudo frontal view images by homography matrix and extracts the keypoints. Homography matrix is calculated with the position of the license plate. Our experimental result shows our method enables to recognize up to 60-degree angle.

We propose a vehicle make and model recognition method for the smart phone, and implemented it onto them. Our method identifies the make and model from the variety of viewpoints while the conventional methods for VMMR work only for the frontal or rear view images. This method enables the users to take the pictures from a certain range of angles. Our method uses SIFT, that has scale and rotation invariance to solve the alignment issue. It creates the pseudo frontal view images by homography matrix and extracts the keypoints. Homography matrix is calculated with the position of the license plate. Our experimental result shows our method enables to recognize up to 60-degree angle.

A method of reconstructing the 3D trajectories of a table tennis ball is introduced, which was developed to solve the problem with conventional analysis in table tennis. There are several methods of reconstructing 2D ball trajectories or 3D trajectories of balls heavier than those in table tennis. However, these methods cannot be adopted to reconstruct the 3D trajectories of table tennis balls, because there are problems that are attributed to the dimensions of the trajectories and weight of the balls. The method proposed in this paper could reconstruct the 3D trajectories of a table tennis ball. The key feature of the method is that it approximates that a ball is traveling on tilted planes. This approximation makes reconstruction robust against failure to measure 3D ball positions. A system using two RGB cameras was developed based on the new method. The system experimentally demonstrated that it could provide accurate information for match analysis. A system using an RGB-D camera was then developed to optimize usability for practitioners. We experimentally demonstrated that the system could provide accurate information for service analysis.

We propose a stable keypoint recognition method that is robust to viewpoint changes. Conventional local features such as SIFT, SURF, etc., have scale and rotation invariance but often fail in matching points when the camera pose significantly changes. In order to solve this problem, we adopt viewpoint generative learning. By generating various patterns as seen from different viewpoints and collecting local invariant features, our system can learn feature descriptors under various camera poses for each keypoint before actual matching. Ex- perimental results comparing usual local feature matching or patch classification method show both robustness and fastness of learning.

In this paper, we describe a method for overlaying a texture onto a T-shirt, for improving current virtual fitting system. In such systems, users can try on clothes virtually. In order to realize such a system, a depth camera has been used. These depth cameras can capture 3D data in real time and have been used by some industrial virtual cloth fitting systems. However, these systems roughly, or just do not, consider the shape of the clothes that user is wearing. So the appearance of these virtual fitting systems looks unnaturally. For a better fitting, we need to estimate 3D shape of cloth surface, and overlay a texture of the cloth that the user wants to see onto the surface. There are some methods that register a 3D deformable mesh onto captured depth data of a target surface. Although those registration methods are very accurate, most of them require large amount of processing time or either manually-set markers or special rectangles. The main contribution of our method is to overlay a texture onto a texture of T-shirt in real-time without modifying the surface.

In this paper, we aimto develop an AR system which supports spectators who are watching a sports game using smartphones in a spectators’ stand. The final goal of this system is that a spectator can watch information of players through a smartphone and share experiences with other spectators. For this goal, we propose a system which consists of smartphones and fixed cameras. Fixed cameras are set to cover the whole sports field and used to analyze players. Smartphones held by spectators are used to estimate positions where they are looking on the sports field. We built an AR system which makes annotation of players’ information onto a smartphone image. And we evaluated the accuracy and the processing time of our system and revealed its practicality.

A plane approximation-based approach for 3D reconstruction of ball trajectory is introduced, which was developed as a solution to the problem with conventional analysis in table tennis. There are methods of reconstructing 2D ball trajectories or 3D trajectories of balls heavier than those in table tennis. However, these methods cannot be adopted to reconstruct the 3D trajectories of table tennis balls, because there are problems that are attributed to the dimensions of the trajectories and weight of the balls. The method proposed in this chapter could reconstruct the 3D trajectories of a table tennis ball. The key feature of the method is that it approximates that a ball is traveling on tilted planes. This approximation makes reconstruction robust against failure to measure 3D ball positions. Two systems were developed based on the new method. A system using two RGB cameras experimentally demonstrated that it could provide accurate information for match analysis. A system using an RGB-D camera was experimentally demonstrated that the system could provide accurate information for service analysis.