Published papers

When a model learned in a domain is applied to a different domain, even if in the same task, there is no guarantee of accuracy. This is a very important issue when deep learning and machine learning are applied in the field. In medical applications, there is a wide variety of domain bias, making it very difficult to create a model appropriate for all domains. Furthermore, semantic segmentation needs fine annotation and its high labor cost makes its application difficult. Histopathological image segmentation enables drug discovery and medical image analysis, but it is expensive due to its annotation cost and the need for the skills of histopathological experts. In this paper, we focus on a weakly supervised method using point annotation unique to histopathological image segmentation, and tackled on weakly supervised domain adaptation to suppress domain gaps. Providing point level annotation instead of fine annotation decreases the high cost of labor normally required.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. The paper presents an emerging issue of fine-grained pedestrian action recognition that induces an advanced pre-crush safety to estimate a pedestrian intention in advance. The fine-grained pedestrian actions include visually slight differences (e.g., walking straight and crossing), which are difficult to distinguish from each other. It is believed that the fine-grained action recognition induces a pedestrian intention estimation for a helpful advanced driver-assistance systems (ADAS). The following difficulties have been studied to achieve a fine-grained and accurate pedestrian action recognition: (i) In order to analyze the fine-grained motion of a pedestrian appearance in the vehicle-mounted drive recorder, a method to describe subtle change of motion characteristics occurring in a short time is necessary; (ii) even when the background moves greatly due to the driving of the vehicle, it is necessary to detect changes in subtle motion of the pedestrian; (iii) the collection of large-scale fine-grained actions is very difficult, and therefore a relatively small database should be focused. We find out how to learn an effective recognition model with only a small-scale database. Here, we have thoroughly evaluated several types of configurations to explore an effective approach in fine-grained pedestrian action recognition without a large-scale database. Moreover, two different datasets have been collected in order to raise the issue. Finally, our proposal attained 91.01% on National Traffic Science and Environment Laboratory database (NTSEL) and 53.23% on the near-miss driving recorder database (NDRDB). The paper has improved +8.28% and +6.53% from baseline two-stream fusion convnets.

© 2018 IEEE. In this paper, we propose a novel approach for traffic accident anticipation through (i) Adaptive Loss for Early Anticipation (AdaLEA) and (ii) a large-scale self-annotated incident database for anticipation. The proposed AdaLEA allows a model to gradually learn an earlier anticipation as training progresses. The loss function adaptively assigns penalty weights depending on how early the model can anticipate a traffic accident at each epoch. Additionally, we construct a Near-miss Incident DataBase for anticipation. This database contains an enormous number of traffic near-miss incident videos and annotations for detail evaluation of two tasks, risk anticipation and risk-factor anticipation. In our experimental results, we found our proposal achieved the highest scores for risk anticipation (+6.6% better on mean average precision (mAP) and 2.36 sec earlier than previous work on the average time-to-collision (ATTC)) and risk-factor anticipation (+4.3% better on mAP and 0.70 sec earlier than previous work on ATTC).

© 2018 IEEE. The demand for sports video analysis is expanding. Sports video analysis is used to analyze their own play and play of opponent players, and visualize movement and ability of players. Scientific and objective analysis become possible by incorporating video analysis. It is desired to improve the competition level by feeding back the analysis results to the players. Therefore, in this research, in the case of feeding back the analysis result to the athlete, research purpose is to realize a method which can understand and evaluate the details of the form hitting the ball in detail. First, joint position coordinates are estimated from input RGB tennis images by a posture estimation method. The joint position coordinates in each frame at the time of shot are classified using unsupervised method and represented by BoW. The feature vector is designed by combining this with the shot position. The probability of shot success is predicted using this feature vector. By visualizing BoW of the shot with the high probability of success and the high failure probability, it is possible to extract and compare poses that are likely to appear in each case without giving correct labels.

© 2018 IEEE. In this paper, we propose a method to estimate 6D pose (3D position and pose) of the everyday objects, especially tools, even if there are not the same 3D model of the target object. We assume that the tools that belong to same category have common spatial relation in part-affordances, the role of each part. The proposed method uses this assumption for pose estimation. By minimizing two types of consistencies, the spatial relationship of part-affordances and the shapes, we have confirmed that the proposed method achieved higher performance compared with traditional pose estimation method.

© 2018 IEEE. In this paper, we propose a novel segmentation algorithm based on convolutional neural networks (CNNs) on superpix-else CNNs are powerful methods for several computer vision tasks, but spatial information disappears through the pooling process. Moreover, since pooling compresses different types of pixels into a single value, pooling sometimes negatively affects the results of inference in segmentation task. We use superpixel pooling instead of general pooling to resolve this problem. However, general CNNs can't use superpixel images in which the adjacency relationships between pixels are broken. Therefore, we define CNNs and Dilated Convolution on superpixels. Finally, we show the effectiveness of proposed method on an HKU-IS dataset.

Recently, the development of deep learning has enabled robots to grasp objects more reliably than ever. Given this fact, there is an increasing demand for helper robots or home robots. To make these robots real, robots need to understand not only how to grasp objects but also their functions. We propose a new representation for the functions of objects, task-oriented function, which is based on operational task input. This representation makes it possible to describe a variety of ways to use an object. We also propose a new dataset for task-oriented function and a network to detect it. This model reached 79.7% mean IOU in our dataset.

The use of image processing technology for sports is increasing. By analyzing athletes and teams based on video analysis, scientific and objective analysis can be conducted separately from subjective analysis of experts, and individuals or teams can be evaluated with the same index. As can be seen from such a trend, recently, technological progress has made it possible to detect and analyze what can be visually confirmed in the sports video. It is certain that these technologies are helping coaches in the analysis of movement. However, these cannot play the role of coaches. The role is to judge what is important from what can be visually confirmed. There are also empirical and qualitative parts in coach's judgment, and no reproducibility of understanding the important parts unless it is a specialist. Based on this background, we thought that it would be useful to extract more important scenes during the game using quantitative information. Such technology can be applied to various sports, but this paper, we focused on tennis. In this thesis, the aim is to estimate which play was largely contributed to the tennis game result (score, failure). In addition, we don't use supervised information that can be obtained from an empirical point of view. This is to eliminate dataset dependency caused by using data created from qualitative information. Specifically, based on quantitative information such as athletes' movement and score result, we attempted to estimate the attention from unsupervised method.

We address the problem of soft color segmentation, defined as decomposing a given image into several RGBA layers, each containing only homogeneous color regions. The resulting layers from decomposition pave the way for applications that benefit from layer-based editing, such as recoloring and compositing of images and videos. The current state-of-the-art approach for this problem is hindered by slow processing time due to its iterative nature, and consequently does not scale to certain real-world scenarios. To address this issue, we propose a neural network based method for this task that decomposes a given image into multiple layers in a single forward pass. Furthermore, our method separately decomposes the color layers and the alpha channel layers. By leveraging a novel training objective, our method achieves proper assignment of colors amongst layers. As a consequence, our method achieve promising quality without existing issue of inference speed for iterative approaches. Our thorough experimental analysis shows that our method produces qualitative and quantitative results comparable to previous methods while achieving a 300,000x speed improvement. Finally, we utilize our proposed method on several applications, and demonstrate its speed advantage, especially in video editing.

The analysis of assembly working scenes is an important tool for improving work efficiency and detecting worker error. Many factories choose to analyze videos of employees working by human observation. It is more efficient, however, to approach this as an action segmentation task, but this is difficult due to the fine-grained actions. In this paper, we focus on featuring the workers hands in action segmentation to describe the detailed moves in assembly work scenes and utilize the tool or product information that the worker is using. We propose two methods to draw attention to the hand from each image in the video: the first is by cutting out the workers hand image and extracting features, and the second is by using an attention module specialized to extract hand features by the training of a regression network. For both methods we combine different types of features - image features and pose features - which are both important information in fine-grained actions. Due to the lack of action datasets that focus on assembly work scenes, we create a new assembly work dataset for our experiments, and the results of those experiments show that our method is effective for analyzing fine-grained action segmentation tasks.

In this study, we proposed data-mining based bell-curve analogical hydrographs analysis with lag time vertical axes and bankfull discharge horizontal axes to make flood susceptibility prediction. We utilized flood data reports, hourly/daily rainfall data and daily water discharge of Hulu Langat district, Selangor Malaysia from the year 2013-2016 to do flood susceptibility. We implement data mining concept by sorting the database, followed by plotting hydrograph to identify flood patterns and establish relationships to predict flood trends. This method is an intersection between the knowledge field of hydrology and mathematical modeling. When an outlier from the graph is detected, the knowledge from hydrology can be applied to understand the reason behind the appearance of outliers. Besides, the knowledge of mathematical modeling is necessary to assist us in predicting flood susceptibility. The purpose of this study is to predict the flood susceptibility which is vital to prepare the users/public well prepared for smooth and efficient evacuation. In 4 years context, our flood depth predictions are nearly 100% accurate. Factor influencing the lag time and steepness of rising limb are related to land use and topographical features. Implications of the results and future research directions are also presented.

Optical flow estimation from an in-vehicle camera is an important task in automatic driving and advanced driver-assistance systems. However, there is a problem that optical flow estimation is mistakable with high contrast and high speed. Event camera can overcome these situations because it reports only the per-pixel intensity change with high dynamic range and low latency. However, the L1 smoothness regularization in the conventional optical flow estimation method is not suitable for radial optical flow in the driving scene. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical flow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical flow becomes radial from the FOE excluding the rotational component. Using the property, the optical flow can be regularized in the correct direction in the optimization process. We demonstrated that the optical flow was improved by introducing our regularization using the public dataset.

Optical flow estimation in onboard cameras is an important task in automatic driving and advanced driver- assistance systems. However, there is a problem that calculation is mistakable with high contrast and high speed. Event cameras have great features such as high dynamic range and low latency that can overcome these problems. Event cameras report only the change in the logarithmic intensity per pixel rather than the absolute brightness. There is a method of estimating the optical ow simultaneously with the luminance restoration from the event data. The regularization using the L1 norm of differentiation is insufficient for spatially sparse event data. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical ow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical ow becomes radial from the FOE excluding the rotational component. Using the property, the optical ow can be regularized in the correct direction in the optimization process. We demonstrated that the optical ow was improved by introducing our regularization using the public dataset.

Optical flow estimation from an in-vehicle camera is an important task in automatic driving and advanced driver- A ssistance systems. However, there is a problem that optical flow estimation is mistakable with high contrast and high speed. Event camera can overcome these situations because it reports only the per-pixel intensity change with high dynamic range and low latency. However, the LI smoothness regularization in the conventional optical flow estimation method is not suitable for radial optical flow in the driving scene. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical flow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical flow becomes radial from the FOE excluding the rotational component. Using the property, the optical flow can be regularized in the correct direction in the optimization process. We demonstrated that the optical flow was improved by introducing our regularization using the public dataset.

In the field of action recognition, when and where an interaction between a human and an object happens has the potential to be valid information in enhancing action recognition accuracy. Especially, in daily life where each activities are performed in longer time frame, conventional short term action recognition may fail to generalize do to the variety of shorter actions that could take place during the activity. In this paper, we propose a novel representation of human object interaction called Human-Object Maps (HOMs) for recognition of long term daily activities. HOMs are 2D probability maps that represents spatio-temporal information of human object interaction in a given scene. We analyzed the effectiveness of HOMs as well as features relating to the time of the day in daily activity recognition. Since there are no publicly available daily activity dataset that depicts daily routines needed for our task, we have created a new dataset that contains long term activities. Using this dataset, we confirm that our method enhances the prediction accuracy of the conventional 3D ResNeXt action recognition method from 86.31% to 97.89%.

The paper presents a pedestrian near-miss detector with temporal analysis that provides both pedestrian detection and risk-level predictions which are demonstrated on a self-collected database. Our work makes three primary contributions: (i) The framework of pedestrian near-miss detection is proposed by providing both a pedestrian detection and risk-level assignment. Specifically, we have created a Pedestrian Near-Miss (PNM) dataset that categorizes traffic near-miss incidents based on their risk levels (high-, low-, and no-risk). Unlike existing databases, our dataset also includes manually localized pedestrian labels as well as a large number of incident-related videos. (ii) Single-Shot MultiBox Detector with Motion Representation (SSD-MR) is implemented to effectively extract motion-based features in a detected pedestrian. (iii) Using the self-collected PNM dataset and SSD-MR, our proposed method achieved +19.38% (on risk-level prediction) and +13.00% (on joint pedestrian detection and risk-level prediction) higher scores than that of the baseline SSD and LSTM. Additionally, the running time of our system is over 50 fps on a graphics processing unit (GPU).

We propose an image reconstruction method from event data, assuming the target images belong to a prespecified class like QR codes. Instead of solving the reconstruction problem in the image space, we introduce a code space that covers all the noiseless target class images and solves the reconstruction problem on it. This restriction enormously reduces the number of optimizing parameters and makes the reconstruction problem well posed and robust to noise. We demonstrate fast and robust QR-code scanning in difficult, high-speed scenes with industrial high-speed cameras and other reconstruction methods.

In this paper, we present a framework thatjointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

Generative image extension has the advantage of extending the overall image size while preserving the target image because, unlike other image extensions using interpolation, it completes the surroundings of the target image. However, existing generative image extension methods tend to have poor quality in the generation of outer pixels. One method deals only with a limited number of scene classes because the extension repeats the same semantics. We propose a mirrored input, which sandwiches the extended region by mirroring a part of the target image. This replaces generative image extension with an image inpainting problem and thus helps to achieve higher quality pixel generation and can extend semantics with more complex shapes than horizontal repetition. Experimental results show that our proposed method achieves a scenery image extension that exceeds the state-of-the-art generative image extension methods in both visual quality and FID score for datasets containing diverse scenes.

In recent years, there has been a demand in the sports industry to reduce the burden of data collection and video editing for tactical analysis. To achieve these, a system that can recognize the game context is needed. In this study, we proposed a method to identify the player's shot timing at the frame level during a ball-striking sport. In this study, players' shots were detected in video of a tennis match. It was shown that shots could be detected with an F-score value of 87% or more within an error range of 1 frame (0.033 sec) by considering time-series information using a recurrent neural network. This technology is expected to be applied not only to tennis, but also to other sports that involve ball shots, such as table tennis, baseball, and volleyball. At the same time, it can be used to detect moments of a specific action (for example, touching or hitting an object).