Published papers

Digital images are frequently degraded by Gaussian noise while capturing photos. This paper proposes a rapid and high accurate Gaussian noise removal method by applying the learned linear filter used in RAISR for super-resolution. The denoising methods are classified into local, nonlocal methods and deep-learning-based methods. The conventional local processing has a problem that high-frequency components of the original image are lost while reducing the noise. The nonlocal and deep-learning-based methods achieve higher denoising performance but take a long time for training and implementation. To solve these problems, we apply a super-resolution method to the local denoising method as post-processing because it can efficiently recover the high-frequency components. The super-resolution method uses a learned linear filter according to the feature of patches. The novelty of this paper is that the same processing as super-resolution is incorporated into denoising. The proposed algorithm is a rapid local denoising method and can achieve comparable performance to the high-accurate nonlocal denoising methods. Experimental results show that our proposed method provides accurate denoising performance with a low computational cost compared to nonlocal processing like BM3D.

Under low-light conditions, captured images can be affected by unsatisfactory lighting conditions. Low-light image enhancement called LLIE is a digital image processing to obtain natural normal-light images from the low-light image. LLIE includes three main tasks: reducing noise and artifacts, preserving edges and textures, and reproducing natural brightness and color. In recent years, many types of research have focused on deep-learning-based approaches that can achieve excellent performance. However, one primary problem with these methods is that inference time is long owing to complex network structures. To solve the trade-off between the performance and implementation time, we propose a simple network with effective modules. We utilize a U-Net structure and pre-processing is added to preserve edges and textures. Moreover, we embed Channel Attention to restore color and illumination, Res FFT-ReLU to reduce noise, and Pixel Shuffler to preserve the high-frequency components. According to our experimental results, the proposed method achieves better performance and faster inference time than conventional LLIE methods.

Pose Guided Person Image Generation (PGPIG) is a task that involves generating an image of a person in a target pose, given an image in a source pose, source pose information and target pose information. Many of the existing PGPIG techniques need extra pose-related data or tasks, which cause the limitations on their applicability. In addition CNNs are used as the feature extractor which do not have long-range dependency. However, CNNs can only extract features from neighboring pixels and cannot consider image consistency. This paper introduces a PGPIG network that solves these challenges by incorporating modules that use Axial Transformers with wide receptive fields. The proposed approach disentangles the PGPIG task into two subtasks: 'rough pose transformation' and 'detailed texture generation.' In 'rough pose transformation,' lower-resolution feature maps is processed by Axial Transformer-based blocks. These blocks employ an Encoder-Decoder structure, which allows the network to use the pose information well and improves the stability and performance of the training. The latter subtask employs a CNN network with Adaptive Instance Normalization. Experimental results show the competitive performance of the proposed method compared to existing methods. The proposed method achieves lowest LPIPS in Deep Fashion dataset and FID in Market-1501 dataset. Remarkably, despite the great results obtained, the number of parameters of the proposed network is significantly less in contrast to existing methods.

Pose Guided Person Image Generation (PGPIG) is the task that transforms the pose of a person's image from the source image, its pose information, and the target pose information. Most existing PGPIG methods require additional pose information or tasks, which limits their application. Moreover, they use CNNs, which can only extract features from neighboring pixels and cannot consider the consistency of the entire image. This paper proposes a PGPIG network solving these problems by using a module containing Axial Transformers with large receptive field. The proposed method disentangles the PGPIG task into two subtasks: “rough pose transformation” and “detailed texture generation”. In the former task, low-resolution feature maps are transformed by blocks containing Axial Transformer. The latter task uses a CNN network with Adaptive Instance Normalization. Experiments show that the proposed method has competitive performance with other state-of-the-art methods. Furthermore, despite achieving excellent performance, the proposed network has a significantly fewer parameters than existing methods.

Convolutions in neural networks are still essential on various vision tasks. To develop neural convolutions, this study focuses on Structured Receptive Field (SRF), representing a convolution filter as a linear combination of widely acting designed components. Although SRF can represent convolution filters with fewer components than the number of filter bins, N-Jet, the sole component system implementation, requires ten trainable parameters per filter to improve accuracy even for 3×3 convolutions. Hence, we aim to formulate a new component system for SRF that can represent valid filters with fewer components. Our component system named 'OtX' is based on the Principal Component Analysis of well-trained filter weights because the extracted components will also be principal for neural convolution filters. In addition to proposing the component system, we develop a component scaling method to defuse massive scale differences among the coefficients in a linear combination of OtX components. In the experimental section, we train image classification models on CIFAR-100 dataset under the hyperparameters tuned for the original models with the standard convolutions. For NFNet-F0 classifier, OtX with six components performs 0.5% better than the standard convolution, 3.1% better than N-Jet with six components, and only 0.1% worse than N-Jet with ten components. Besides, OtX with nine components provides stabler training than N-Jet, performing 0.5% better than the standard for NFNet-F0. OtX suits when replacing standard convolutions because OtX performs at least comparably against N-Jet with further parameter efficiency and training stability.

RGB-D salient object detection is a task to detect visually significant objects in an image using RGB and depth images. Although many useful CNN-based methods have been proposed in the past, there are some problems such as blurring of object boundaries and inability to detect important parts of objects, leading to a decrease in detection accuracy. In this paper, we propose RGB-D salient object detection using Saliency and Edge Reverse Attention(SERA), which combines the fusion of saliency and edge features with reverse attention. The reverse attention process has the effect of making it easier to capture the boundaries of objects and undetected objects by inverting the up-sampled saliency features and weighting other saliency features, and the edge reverse attention process has the effect of making salient object regions stand out by inverting the edge features and weighting saliency features. The interaction between the salient object features and the edge features enhances each other's features and refines the information on the boundary of objects and salient object regions. In addition, to make it easier to refer to the global information of an image, we introduced the Multi-Scale Interactive Module(MSIM), which is capable of acquiring information at rich scales by converting feature maps to different resolutions and interacting with them. In addition to the salient object output, supervised learning is applied to multiple edge outputs of each resolution to improve the accuracy of both salient objects and boundary areas. Experimental results on five benchmarks show that the proposed method quantitatively performs better than the conventional method, and qualitatively improves the sharpness of object boundaries and the accuracy of detecting important parts of objects.

Single image reflection removal is a technique that removes undesirable reflections, which occur due to glass, from images. Various methods of reflection removal have been proposed, but unfortunately, they usually fail to remove reflections particularly with very high pixel values. In this paper, we define these saturated reflections and their characteristics, as well as discuss and propose a removal system. The proposed system detects areas of saturated reflections based on our proposed model of convolutional neural networks and restores them by a conventional method of image estimation. In our experiments, the proposed system shows better peak-signal-to-noise ratio scores and perceptual quality than conventional methods of reflection removal.

To remove shadowed region in a single image, it is important to obtain high accuracy in both two processes, shadow detection and removal. In order to improve the results, recent methods perform these two processes simultaneously and use GAN for the training. However, since these methods do not try to maintain the luminance of non-shadowed regions, the output images tend to be faded. In this paper, to overcome fading problem, we proposed a new GAN structure based on shadow model. Since our GAN-based method focus on the variation of the illuminance, the illuminances of the shadowed regions, whose amount of change are large, are effectively estimated. In addition, non-shadowed regions remain slightly faded due to our new GAN structure and training method. Owing to our novel GAN structure and training method, our method outperforms state-of-the-art methods in PSNR and SSIM.

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this problem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this prob-lem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.

With the increasing availability of high-resolution video recording and streaming, there is a need for fast and high-quality video denoising methods that can handle high-resolution videos. However, many existing methods fail to achieve high-quality denoising performance and computationally effecient at the same time. This paper proposes a video denoising network, Pseudo Temporal Fusion Network (PTFN), that satisfies these requirements. PTFN adopts a new Pseudo Temporal Fusion (PTF) module that captures pseudo-temporal relationships between video frames in combination with the Temporal Shift Module. PTFN also adopts a modern ConvBlock paradigm that breaks away from the classical ConvBlock paradigm, contributing to denoising performance and computationally effecient. PTFN achieves better performance than existing video denoising methods in terms of both video quality and computational effeciency. Specifically, PTFN has only about 16.7% of the computational cost of existing lightweight methods, while it improves denoising performance. PTFN is also superior in terms of memory consumption. It can process 1080p videos with a GPU with 24 GB RAM. In addition, a lighter version (PTFN Half) can process 2K videos at high speed under the same conditions.

With the recent widespread use of smartphones and digital video cameras, the opportunities to handle digital video have increased significantly. However, despite improvements in the performance of the hardware, the captured video often contains information that is not necessary for the purpose of the video. In particular, factors such as camera shake and object movement can cause blurring in video. So, we propose a method to deblur a video by software processing of the video after shooting. In conventional methods, video deblurring is often performed using a network whose main task is video super-resolution. However, in super-resolution, the size of the input and output images are different, while the input and output images are the same size in deblurring. In the case of deblurring, the input images are input to the network after a simple downsampling process, which is not optimized for the same size as the input and output images. Therefore, the proposed method constructs a multi-scale network based on UNet. The UNet-based network is a successful method for single image deblurring. Because a video is a sequence of multiple images, we use a method that has been successful in single image deblurring. Furthermore, we add improvements to the network based on the structure of MPRNet. The feature extraction modules of the bottom and the second stage of the network are replaced with a single-stage UNet. These improvements resulted in a 34.80dB in PSNR and 0.973 in SSIM on the GoPro dataset despite about 75% of the FLOPs of BasicVSR++ and 3% of the FLOPs of VRT. On the DVD dataset, the proposed model achieved 34.36dB in PSNR and 0.966 in SSIM. Further ablation studies show the effectiveness of various components in our proposed model.

Images and video captured in water or fog suffer from low contrast and color distortion due to light scattering and absorption. An image formation model for hazy images is commonly used to restore both underwater images and hazy images because of the similarity between the two types of images. However, red light is attenuated faster than blue and green light in underwater, and underwater images are distorted by changes of color tone. Therefore, most current methods are specialized for either hazy images or underwater images. In this paper, we propose a novel defogging method which is efficient for both hazy images and underwater images. Our method is composed of adaptive color correction and weighted generalization of dark channel prior (WGDCP). Experimental results show that our algorithm can recover both underwater images and hazy images.

CNNs have traditionally been applied in computer vision. Recently, applying Transformer networks, originally a technique in natural language processing, to computer vision has received much attention and produced superior results. However, Transformers and their derivation have drawbacks that the computational cost and memory usage increase rapidly with the image resolution. In this paper, we propose the Laplacian Pyramid Translation Transformer (LPTT) for image to image translation. The Laplacian Pyramid Translation Network, a previous study of this work, creates Laplacian pyramid of the input images and processes each component with CNNs. However, LPTT transforms the high-frequency components with CNNs and the low-frequency components with Axial Transformer blocks. LPTT can have Transformer's expressive power while reducing the computational cost and memory usage. LPTT significantly improves the quality of generated images and inference speed for high-resolution images over conventional methods. LPTT is the first network with a Transformer that can perform practical inference in real time on 4K resolution images. LPTT can also process 8K images in real time depending on the model conditions and the performance of the GPU. The ablation study in this paper suggests that even when processing high-resolution images, the performance is improved while maintaining the inference speed by computing the low-resolution component with a Transformer. LPTT improves PSNR value by 0.41 dB in MIT-Adobe FiveK dataset. The greater the number of layers in the Laplacian pyramid, the greater the improvement of LPTT over the Laplacian Pyramid Translation Network.

Even today, where many deep-learning-based methods have been published, single-image shadow removal is a challenging task to achieve high accuracy. This is because the shadow changes depending on various conditions such as the target material or the light source, and it is difficult to estimate all the physical parameters. In this paper, we propose a new single-image shadow removal method (Channel Attention GAN: CANet) using two networks for detecting shadows and removing shadows. Intensity change in shadowed regions has different characteristics depending on the wavelength of light. In addition, the image acquisition system of the camera acquires an image in a state where the RGB values influence each other. Therefore, our method focused on the physical properties of shadows and the camera's image acquisition system. The proposed network has a structure considering the relationship between color channels. When training this network, we modified the color and added some artifacts to the training images in order to make the training dataset more complex. These image processing are based on the shadow model, considering the camera image acquisition system. With these new proposals, our method can remove shadows in all ISTD, ISTD+, SRD, and SRD+ datasets with higher accuracy than the state-of-the-art methods. The code is available on GitHub: https://github.com/ryo-abiko/CANet.

Under severe weather conditions, the quality of the images taken outside is directly affected by floating atmospheric particles. To keep the quality of the images, haze removal methods play a critical role. The most difficult part of haze removal is removing the haze that spreads over the entire image. Many CNN-based methods have been proposed to remove the haze, and can be divided into two types. One is to use a multi-scale structure and the other is to stack layers. The former causes image degradation due to the loss of some of the original information in an image and the latter increases computational complexity due to not reducing the resolution. In addition, a large number of parameters is required to secure the expressive power of the model, which leads to a huge amount of memory. To tackle these problems, we tried to 1) downsample the image while saving parameters and maintaining the quality of the generated image, and 2) consider the information in the entire image to remove the haze. For the first problem, we tried to solve this by using a feature extractor that has been used in other tasks, learning to optimize the output image in low-resolution, and preparing kernels with various dilation rates to expand the receptive fields. For the second problem, we use the attention structure to determine which part of the image features should be focused on from the entire feature map. By incorporating such modules, our method achieves better results on both synthetic and real-world images when compared with state-of-the-art methods.

Unpaired image to image translation is the task of converting images in unpaired datasets. The primary goal of the task is to translate a source image into the image aligned with the target domain while keeping the fundamental content. Existing researches have introduced effective techniques to translate images with unpaired datasets, focusing on preserving the fundamental content. However, these techniques have limitations in dealing with significant shape changes and preserving backgrounds that should not be transformed. The proposed method attempts to address these problems by utilizing the saliency domain for translation and simultaneously learning the translation in the saliency domain as well as in the image domain. The saliency domain represents the shape and position of the main object. The explicit learning of transformations within the saliency domain improves network's ability to transform shapes while maintaining the background. Experimental results show that the proposed method successfully addresses the problems of unpaired image to image translation and achieves competitive metrics with existing methods.

In this paper, we propose a high quality image deblurring method that uses discrete cosine transform (DCT) and requires less computational complexity. We train our model on a new dataset which is customized to include images with large motion blurs. Recently, Convolutional Neural Network (CNN) and Generative Adversarial Network (GAN) based algorithms have been proposed for image deblurring. Moreover, multi-scale and multi-patch architectures of CNN restore blurred images clearly and suppress more ringing or blocking artifacts, but they take a longer time to process. To improve the quality of deblured images and reduce the computational time, we propose a method called 'DeblurDCTGAN' that preserves texture and suppresses ringing artifacts in the restored image without multi-scale or multi-patch architecture using DCT based loss. This loss compares the restored image and the ground truth image in the frequency domain. With this loss, DeblurDCTGAN can reduce block noise and ringing artifacts while maintaining deblurring performance. Our experimental results show that DeblurDCTGAN gets the highest performances in terms of PSNR, SSIM, and running time compared with conventional methods. In terms of real image datasets, DeblurDCTGAN shows a better performance by using a customized training dataset made from GoPro, DVD, NFS and HIDE training datasets. Experimented code with pre-trained weights, datasets and results are available at https://github.com/Hiroki-Tomosada/DCTGAN-master

Under severe weather conditions, outdoor images or videos captured by cameras can be affected by heavy rain and fog. In this paper, we address a single-image rain removal problem (de-raining). As compared to video-based methods, single-image based methods are challenging because of the lack of temporal information. Although many existing methods have tackled these challenges, they suffer from overfitting, over-smoothing, and unnatural hue change. To solve these problems, we propose a GAN-based de-raining method. The optimal generator is determined by experimental comparisons. To train the generator, we learn the mapping between rainy and residual images from the training dataset. Besides, we synthesize a variety of rainy images to train our network. In particular, we focus on not only the orientations and scales of rain streaks but also the rainy image composite models. Our method also achieves better performance on both synthetic and real-world images than state-of-the-art methods in terms of quantitative and visual performances.

In recent years, accurate Gaussian noise removal has attracted considerable attention for mobile applications, as in smart phones. Accurate conventional denoising methods have the potential ability to improve denoising performance with no additional time. Therefore, we propose a rapid post-processing method for Gaussian noise removal in this paper. Block matching and 3D filtering and weighted nuclear norm minimization are utilized to suppress noise. Although these nonlocal image denoising methods have quantitatively high performance, some fine image details are lacking due to the loss of high frequency information. To tackle this problem, an improvement to the pioneering RAISR approach (rapid and accurate image super-resolution), is applied to rapidly post-process the denoised image. It gives performance comparable to state-of-the-art super-resolution techniques at low computational cost, preserving important image structures well. Our modification is to reduce the hash classes for the patches extracted from the denoised image and the pixels from the ground truth to 18 filters by two improvements: geometric conversion and reduction of the strength classes. In addition, following RAISR, the census transform is exploited by blending the image processed by noise removal methods with the filtered one to achieve artifact-free results. Experimental results demonstrate that higher quality and more pleasant visual results can be achieved than by other methods, efficiently and with low memory requirements. [Figure not available: see fulltext.].