Published papers

In this paper, we propose a method of reflection removal that reduces high intensity reflection for single image. Various methods of reflection removal have been proposed, but they fail to reduce the high reflections due to their assumption. To tackle this issue, the proposed method detects the target areas with high reflections by the proposed convolutional neural network (CNN) model and estimates their background information by inpainting. It is observed that the reflection is strongly blurred because of its physical property, and hence the proposed CNN model utilizes edge features of pixels for the detection. In simulation, we compare state-of-the-art methods of reflection removal with and without the proposed method for natural images, and the proposed method improves peak signal-to-noise ratio and perceptual quality.

With the rapid development of photography and information processing technologies, we use more and more digital images in our daily lives. JPEG is one of the most widely used digital image formats because of its high efficiency and widespread support. JPEG Artifact Removal is a task that removes artifacts in JPEG images, such as noise and color distortion in JPEG images. Existing methods for JPEG Artifact Removal require high computational costs, which means stricter performance requirements or longer processing time. We propose a novel method for JPEG Artifact Removal named Multi-Stage style Artifacts Removal Net (MSARNet), which meets high performance, high versatility, and low computational cost. MSARNet adopts multi-stage processing, and images are processed in processing stages step by step. This process enables our proposed method to deal with different types of images effectively and efficiently. Additionally, MSARNet estimates the quality value of the input image, which contributes to effective feature processing. Experimental results show that our proposed method handles various artifacts well and outperforms existing methods in artifact removal tasks with lower computational costs.

Video super-resolution aims at generating high-resolution video frames using multiple adjacent low-resolution frames. An important aspect of video super-resolution is the alignment of neighboring frames to the reference frame. Previous methods directly align the frames either using optical flow or deformable convolution. However, directly estimating the motion from low-resolution inputs is hard since they often contain blur and noise that hinder the image quality. To address this problem, we propose to conduct feature alignment across multiple stages to more accurately align the frames. Furthermore, to fuse the aligned features, we introduce a novel Attentional Feature Fusion Block that applies a spatial attention mechanism to avoid areas with occlusion or misalignment. Experimental results show that the proposed method achieves competitive performance to other state-of-the-art super-resolution methods while reducing the network parameters.

In this paper, we propose a people removal method from a single image for privacy and other reasons using a three-stage network of depth estimation, semantic segmentation, and inpainting, as shown in Fig. 1. In this three-stage network, we improve semantic segmentation for detecting people. We focus on a special situation of a person and construct a network. It is known that the accuracy of conventional methods can be improved by using edge information. The accuracy of segmentation can be further improved by increasing the accuracy of the edge map. In addition, edge detection does not work well when the person and the background are of the similar color, because edge detects the brightness change of the image. Therefore, in this paper, an adversarial loss function for edge maps is proposed. In addition, since an image with people is expected to have a depth difference from the background image, we use a trained depth estimation network to include the depth image in the input. In this way, it is possible to construct a network for people removal with a high accuracy both quantitatively and qualitatively.

The quality of the images taken outside is directly affected by floating atmospheric particles. To keep the quality of the image, haze removal methods play a critical role. In this paper, we propose a new multi-scale network structure that is rarely used in haze removal and a new loss function to generate high-quality haze-free images. Despite a number of proposed dehazing methods, they are not able to capture the haze accurately without being confused by other objects that are similar in color to haze and completely remove the haze throughout the image. Our proposed network architecture takes the Scale Recurrent Network structure and we incorporate dilated convolution to catch the more global features like haze with the wide receptive field. Additionally, we train our network with a new loss function using dark channel prior to more effectively learn the haze features. Compared with state-of-the-art methods, our proposed method achieves better results on both synthetic and real-world images.

Fences extensively cover the image in a variety of shapes, making fence removal from the image is a challenging task. Most fence removal in a single image is performed in two stages: fence area estimation and inpainting as shown in the Fig 1. Conventional fence detection has utilized information such as edge information, luminance information, and fence regularity to deal with various shapes. However, those methods have resulted in missing fences and false detections of structures similar to fences. In this paper, we propose a network that incorporates a module that includes a fast Fourier transform. This allows us to extract global features of fences that extensively cover the image in various shapes. Furthermore, we create a dataset of rectangular fences, which is not available in previous datasets. Experimental results on existing and proposed datasets show that our method achieves better results, both quantitatively and qualitatively, by suppressing missing fences and false positives of other structures.

Stereo matching, which estimates the disparity of corresponding points in paired images, has problems with accuracy degraded by occlusion, repeated texture, texture-free areas, and gloss. To address this problem, we focused on object boundary information, which is less susceptible to such problems. This is because occlusion is more likely to occur at the boundaries, and even if the corresponding points disappear due to occlusion, the boundaries still have the corresponding parts. The boundary information is also robust to such problems because it is not affected by repeated textures, untextured areas, or gloss. Therefore, we propose a method for generating the boundary ground truth from the depth ground truth and develop a network and loss function for estimating the boundary information. We also propose that the accuracy can be improved by incorporating the estimated boundary information into a state-of-the-art network as an auxiliary.

Convolution is an essential component in neural networks for vision tasks. Convolutions on large receptive fields are more suitable than small kernel convolutions when aggregating global features in convolutional neural networks. However, large kernel convolutions require much computation and memory usage, slowing training neural networks. Then, we propose to train convolution weights with small images, resizing the convolution filters. While this idea shortens the time for training filters, simply applying this causes profound degradation. In this paper, we introduce four techniques that suppress degradation; weight scaling, removing Batch Normalization, defining a minimum resolution, and training with various-size images. In our experiment, we apply our proposals to train an image classification model based on RepLKNet-B on the image classification task of the CIFAR-100 dataset. Training with our proposals is approximately eight times faster than conventional training on the target spatial scale, keeping its accuracy.

Underwater images suffer from low contrast, color distortion and visibility degradation due to the light scattering and attenuation. Over the past few years, the importance of underwater image enhancement has increased because of ocean engineering and underwater robotics. Existing underwater image enhancement methods are based on various assumptions. However, it is almost impossible to define appropriate assumptions for underwater images due to the diversity of underwater images. Therefore, they are only effective for specific types of underwater images. Recently, underwater image enhancement algorisms using CNNs and GANS have been proposed, but they are not as advanced as other image processing methods due to the lack of suitable training data sets and the complexity of the issues. To solve the problems, we propose a novel underwater image enhancement method which combines the residual feature attention block and novel combination of multi-scale and multi-patch structure. Multi-patch network extracts local features to adjust to various underwater images which are often Non-homogeneous. In addition, our network includes multi-scale network which is often effective for image restoration. Experimental results show that our proposed method outperforms the conventional method for various types of images.

Object detection techniques are essential for identifying objects and accurately determining their locations in images and videos. This paper proposes an approach to enhance the performance of YOLOv7, the YOLO (You Only Look Once) series, by improving detection accuracy, particularly for small objects. The accurate detection of small objects is crucial in various real-world applications such as autonomous driving, surveillance, and medical imaging, where the identification of small, often critical objects can have significant implications. The proposed method balances the current one-step and two-step algorithms used in object detection, with the goal of improving the accuracy of small object detection. We apply several improvements to the YOLOv7 framework, including the introduction of an attention module, the Global-Local Fusion Transformer (GLF-Trans), and the addition of a tiny-head. These enhancements are designed to further enhance the accuracy of small object detection. The experimental results show that the proposed method achieves higher accuracy than the previous model, with a specific improvement of 3.8% in mAP0.5 on the VisDrone2019 dataset.

Face photo-sketch synthesis tasks have been developed with Generative Adversarial Networks (GANs) based on Convolutional Neural Network (CNN) and Vision Transformer (ViT). CNN is good at capturing local features, but its locality results in blurred images and contours. ViT is good at capturing global information, but is not as good as CNN in capturing local features, and while it can prevent blurring of contours and other lines, it does not reflect fine texture. Therefore, we propose a Face Drawing GAN, which generates high-quality face sketches by capturing both local and global features. Face Drawing GAN is a CNN-based model and it incorporates Channel Attention, which functionally adjusts the weights of channels, and Matrix Product Attention (MP Attention), which weights pixels based on the similarity between the vertical and horizontal sides of images obtained by matrix product. Through the experiments, we confirmed that our proposed MP Attention assists in capturing global features and Face Drawing GAN is capable of generating face sketches that outperform conventional methods.

Removing rain degradations in images is recognized as a significant issue. In this field, deep learning-based approaches, such as Convolutional Neural Networks (CNNs) and Transformers, have succeeded. Recently, State Space Models (SSMs) have exhibited superior performance across various tasks in both natural language processing and image processing due to their ability to model long-range dependencies. This study introduces SSM to image deraining with deraining-specific enhancements and proposes a Deraining Frequency-Enhanced State Space Model (DFSSM). To effectively remove rain streaks, which produce high-intensity frequency components in specific directions, we employ frequency domain processing concurrently with SSM. Additionally, we develop a novel mixed-scale gated-convolutional block, which uses convolutions with multiple kernel sizes to capture various scale degradations effectively and integrates a gating mechanism to manage the flow of information. Finally, experiments on synthetic and real-world rainy image datasets show that our method surpasses state-of-the-art methods. Code is available at https://github.com/ShugoYamashita/DFSSM.

In recent years, U-net-based models have demonstrated high performance in high-magnification face super resolution (FSR) tasks and are capable of outputting more vivid super resolution (SR) images. However, this model has some drawbacks, such as the possibility of unnecessary computations and uniform block selection at all stages of the decoder. Therefore, we propose a Direct and Stage-wise (DS) Net, which improves on CTCNet [1], a U-net based model with high qualitative results. This model outperforms previous networks by eliminating encoders to reduce computational waste, and by focusing on global feature extraction in the small tensor size stage.

Residual blocks have been widely used in the single image super-resolution (SR), boosting the performance of SR results. The stacking of residual blocks helps to grab more information needed for the recovery of high-resolution (HR) images, but at the cost of larger model size and computational complexity. To address this problem, we propose a lightweight enhanced residual fourier transformation network (ERFTN). To ensure the network is lightweight enough while being capable of capturing both spatial and frequency information, we introduce a novel enhanced residual fourier transformation block (ERFTB) with an enhanced spatial attention (ESA) block. In addition, we adopt the contrastive loss for training acceleration without additional parameters. Extensive experiments demonstrate that our method can achieve SR performance superior to the state-of-the-art SR methods while reducing approximately half of the number of parameters (e.g., with 190K number of parameters, achieving 32.39dB in PSNR on Urban100 dataset).

Removing artifacts caused by multiple adverse weather, including rain, fog, and snow, is crucial for image processing in outdoor environments. Conventional high-performing methods face challenges, such as requiring pre-specification of weather types and slow processing times. In this study, we propose a novel convolutional neural network-based hierarchical encoder-decoder model that addresses these issues effectively. Our model utilizes knowledge of feature representations obtained from masked-based pre-training on a large-scale dataset. To remove diverse degradations efficiently, we employ a proposed dual-pooling adaptive convolution, which improves representational capability of weight generating network by using average pooling, max pooling, and filter-wise global response normalization. Experiments conducted on both synthetic and real image datasets show that our model achieves promising results. The performance on real images is also improved by a novel learning strategy, in which a model trained on the synthetic image dataset is fine-tuned to the real image dataset. The proposed method is notably cost-effective in terms of computational complexity and inference speed. Moreover, ablation studies show the effectiveness of various components in our method.

Computational requirements for training state-of-the-art neural network models are increasing on vision tasks because high computational factors have become known to be effective in improving quality. While research in the image-processing field requires a lot of trials, this trend makes proving hypotheses difficult for researchers in computationally restricted environments. Neural convolution with a wide receptive field is one of the high-computational factors with quality improvement. This study aims to accelerate the training of the large kernel convolutions by resizing both training images and convolution filters to a smaller scale. Applying this strategy requires careful training designs to replace conventional training on the target scale, and we propose four techniques to improve the quality of the trained models. In our experiment, we apply our proposals to train an image classifier model modified from RepLKNet-31B on the image classification task of the CIFAR-10, CIFAR-100, and STL-10 datasets. Our proposed framework trains almost the same models 4.62-4.91 times faster than the standard training on the target spatial scale, keeping its accuracy, and provides 2.61-2.79 times further training acceleration and stability in accuracy compared to Progressive Learning. In addition to the training acceleration, our framework can simultaneously train models for multiple scales without any scale-specific tuning, which provides scalable usage considering the computational costs.

Image compression technologies such as JPEG enable efficient use of digital images. However, these technologies also degrade the quality of the image, resulting in a poor visual appearance and reduced usability. One way to solve the quality problem and achieve high efficiency in using high-quality JPEG images is to use quality improvement technologies based on artifact removal. In recent years, various methods using machine-learning-optimized neural networks have been devised to achieve high performance whereas maintaining versatility for JPEG images with various qualities. On the other hand, methods that achieve high-quality images need enormous computing costs, which is a barrier to easy use, especially when using images with large size. Given this situation, this paper proposes a method that achieves higher performance with smaller computational resources while maintaining the image quality in broader situations. The proposed method consists of a Quality Estimation Part (QE Part) that estimates the quality of the input image and Image Processing Parts (IP Parts) that process the image based on the estimated quality representation. Sequential processing is carried out by connecting multiple IP Parts in a cascade, which enables efficient and effective processing of different features. Each IP Part consists of multiple Processing Blocks, which enable effective quality improvement while maintaining efficiency. These measures enable the proposed method to achieve state-of-the-art quantitative results and better qualitative results that outperform conventional methods for images with various qualities. The code and the pre-trained models are released at https://github.com/ezumi-keio/Sequential_Processing-main.

Under low-light conditions, the images taken may not be satisfactorily bright and may degrade in appearance. Low-light image enhancement (LLIE) is a process that converts such dim images into images taken under normal lighting conditions. The primary objectives of LLIE are to diminish noise and artifacts, maintain the integrity of edges and textures, and restore the image’s natural brightness and colors. Deep learning-based methods have shown remarkable success in this field recently but are hindered by their lengthy processing times due to intricate network architectures. To address the balance between performance and processing speed, we introduce a streamlined network equipped with efficient modules. Our approach incorporates a GAN (Generative Adversarial Network) framework enhanced with preprocessing for edge and texture extraction. We also integrate Channel Attention for color and illumination correction, Res FFT-ReLU for noise reduction, and Pixel Shuffler for high-frequency detail preservation. Our experiments show that our method surpasses traditional LLIE techniques in both quality and processing speed.

Single image deblurring aims to restore a sharp image by removing blurred areas in the single image. Such blurred images are not only visually unpleasant, but cause various problems in many applications like image recognition. In recent years, with the development of deep learning, neural networks are used in many single image deblurring. Especially, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. However, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, this paper proposes a new lightweight network (IRFTNet) based on UNet, which is a widely used basic network of encoder-decoder structures. Our proposed network has three features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) is introduced to efficiently transfer encoder information from lower layers to upper layers. Finally, multiple outputs structure proposed in MIMO-UNet is introduced. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT. Further ablation studies show the effectiveness of various components in our proposed model.

In recent years, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. How-ever, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, we propose a new lightweight network (IRFTNet) based on DeepRFT. This network has two features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) was introduced to efficiently transfer encoder information from lower layers to upper layers. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT.