Published papers

Beginners need a long time before being able to play correctly violin. Learning the bowing technique appears to be a difficult task and retains most of the attention of beginners. Besides this point, the finger placement is also an important part of the learning but often under estimated. One difficulty is that the fingerboard of the violin does not have frets. In this on-going work, we present a marker-less augmented reality system that advises the novice players about their fingering and bowing. We display in real-time the virtual frets by tracking the violin with a depth camera. We also capture and recognize the note currently played to direct the placement of the bow on the strings.

Optical consistency between the real world and the virtual objects is one of the important issues in Augmented Reality (AR). This paper proposes a method to estimate illuminations from an object shadow and incomplete object shape information captured by an RGB-D camera. The environmental illumination can be estimated without any prior knowledge of the object shape. The radiance of each light source is computed by solving linear equations derived from color and depth images. Since a priori knowledge of object shape is not necessary, we can flexibly use/combine any objects. Thus, we can use/combine multiple objects in the target scene to increase the accuracy and flexibility. Experimental results show the characteristics of our proposed method.

Spatial augmented reality extends augmented real- ity by projecting virtual data directly on a target sur- face, but requires to calibrate a projector-camera sys- tem. This paper introduces a free-calibration projector- camera system for spatial augmented reality with a pla- nar surface. A pattern is projected on the target surface that can be freely moved. The main difficulty is to make the projected image fitting the target surface. This is of- ten achieved by calibrating the projector according to the camera’s pose, but the calibration process is offline and restricts the configuration of the setup. Our pro- posed method allows to independently move the projec- tor, the camera or the surface while the projected image continuously fit the target surface. The experimental re- sults demonstrate the efficiency of our calibration-free spatial augmented reality approach when applied on a moving planar surface.

Violin is one of the most beautiful but also one of the most difficult musical instruments for a beginner. This paper presents an on-going work about a new augmented reality system for training how to play violin. We propose to help the players by virtually guiding the movement of the bow and the correct position of their fingers for pressing the strings. Our system also recognizes the musical note played and the correctness of its pitch. The main benefit of our system is that it does not require any specific marker since our real-time solu-tion is based on a depth camera.

Augmented Reality is an emerging research field, that aims for the composition of real and virtual imagery, by means of a camera and display device. Spatial augmented reality employs data projectors to augment the real world. In this setting, traditional tracking methods fall short due to the interference caused by the projector. Recent works assume a calibration process to model the projector and assume continuity in movement of the object being tracked. In this paper we present a tracking-by-detection system that does not require such a procedure and makes use of natural features represented by SIFT descriptors. We evaluate a set of photometric invariants that have previously been shown to improve the performance of object recognition, added to the descriptor to reduce the influence of the projector. We evaluate the descriptors based on precision-recall under projector distortion and the total system based on its tracking performance. Results show tracking is significantly more precise using one of the invariants.