Skip-Concatenated Image Super-Resolution Network for Mobile Devices (original) (raw)
Related papers
Cornell University - arXiv, 2022
Image super-resolution is a common task on mobile and IoT devices, where one often needs to upscale and enhance low-resolution images and video frames. While numerous solutions have been proposed for this problem in the past, they are usually not compatible with low-power mobile NPUs having many computational and memory constraints. In this Mobile AI challenge, we address this problem and propose the participants to design an efficient quantized image super-resolution solution that can demonstrate a real-time performance on mobile NPUs. The participants were provided with the DIV2K dataset and trained INT8 models to do a high-quality 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated edge NPU capable of accelerating quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 60 FPS rate when reconstructing Full HD resolution images. A detailed description of all models developed in the challenge is provided in this paper.
Fast and Accurate Image Super Resolution by Deep CNN with Skip Connection and Network in Network
Neural Information Processing
We propose a highly efficient and faster Single Image Super-Resolution (SISR) model with Deep Convolutional neural networks (Deep CNN). Deep CNN have recently shown that they have a significant reconstruction performance on single-image super-resolution. The current trend is using deeper CNN layers to improve performance. However, deep models demand larger computation resources and are not suitable for network edge devices like mobile, tablet and IoT devices. Our model achieves state-of-the-art reconstruction performance with at least 10 times lower calculation cost by Deep CNN with Residual Net, Skip Connection and Network in Network (DCSCN). A combination of Deep CNNs and Skip connection layers are used as a feature extractor for image features on both local and global areas. Parallelized 1x1 CNNs, like the one called Network in Network, are also used for image reconstruction. That structure reduces the dimensions of the previous layer's output for faster computation with less information loss, and make it possible to process original images directly. Also we optimize the number of layers and filters of each CNN to significantly reduce the calculation cost. Thus, the proposed algorithm not only achieves stateof-the-art performance but also achieves faster and more efficient computation.
LMSN:a lightweight multi-scale network for single image super-resolution
Multimedia Systems, 2020
With the development of deep learning (DL), convolutional neural networks (CNNs) have shown great reconstruction performance in single image super-resolution (SISR). However, some methods blindly deepen the networks to purchase the performance, which neglect to make full use of the multi-scale information of different receptive fields and ignore the efficiency in practice. In this paper, a lightweight SISR network with multi-scale information fusion blocks (MIFB) is proposed to fully extract information via a multiple ranges of receptive fields. The features are refined in a coarse-to-fine manner within each block. Group convolutional layers are employed in each block to reduce the number of parameters and operations. Results of extensive experiments on the benchmarks show that our method achieves better performance than the state-of-the-arts with comparable parameters and multiply-accumulate (MAC) operations.
Memory‐ and time‐efficient dense network for single‐image super‐resolution
IET Signal Processing, 2021
Dense connections in convolutional neural networks (CNNs), which connect each layer to every other layer, can compensate for mid/high-frequency information loss and further enhance high-frequency signals. However, dense CNNs suffer from high memory usage due to the accumulation of concatenating feature-maps stored in memory. To overcome this problem, a two-step approach is proposed that learns the representative concatenating feature-maps. Specifically, a convolutional layer with many more filters is used before concatenating layers to learn richer feature-maps. Therefore, the irrelevant and redundant feature-maps are discarded in the concatenating layers. The proposed method results in 24% and 6% less memory usage and test time, respectively, in comparison to single-image super-resolution (SISR) with the basic dense block. It also improves the peak signal-to-noise ratio by 0.24 dB. Moreover, the proposed method, while producing competitive results, decreases the number of filters in concatenating layers by at least a factor of 2 and reduces the memory consumption and test time by 40% and 12%, respectively. These results suggest that the proposed approach is a more practical method for SISR. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
LDCSIR: Lightweight Deep CNN-based Approach for Single Image Super-Resolution
2021
Single image super-resolution (SISR) is an image processing technique, and its main target is to reconstruct the high-quality or high-resolution (HR) image from the low-quality or lowresolution (LR) image. Currently, deep learning-based convolutional neural network (CNN) image super-resolution approaches achieved remarkable improvement over the previous approaches. Furthermore, earlier approaches used hand designed filter to upscale the LR image into HR image. The design architecture of such approaches is easy, but it introduces the extra unwanted pixels in the reconstructed image. To resolve these issues, we propose novel deep learning-based approach known as Lightweight deep CNN-based approach for Single Image Super-Resolution (LDCSIR). In this paper, we propose a new architecture which is inspired by ResNet with Inception blocks, which significantly drop the computational cost of the model and increase the processing time for reconstructing the HR image. Compared with the other s...
Deep Networks for Image and Video Super-Resolution
2022
Efficiency of gradient propagation in intermediate layers of convolutional neural networks is of key importance for super-resolution task. To this end, we propose a deep architecture for single image super-resolution (SISR), which is built using efficient convolutional units we refer to as mixed-dense connection blocks (MDCB). The design of MDCB combines the strengths of both residual and dense connection strategies, while overcoming their limitations. To enable super-resolution for multiple factors, we propose a scale-recurrent framework which reutilizes the filters learnt for lower scale factors recursively for higher factors. This leads to improved performance and promotes parametric efficiency for higher factors. We train two versions of our network to enhance complementary image qualities using different loss configurations. We further employ our network for video super-resolution task, where our network learns to aggregate information from multiple frames and maintain spatio-t...
AIM 2020 Challenge on Efficient Super-Resolution: Methods and Results
Computer Vision – ECCV 2020 Workshops, 2020
This paper reviews the AIM 2020 challenge on efficient single image super-resolution with focus on the proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor ×4 based on a set of prior examples of low and corresponding high resolution images. The goal is to devise a network that reduces one or several aspects such as runtime, parameter count, FLOPs, activations, and memory consumption while at least maintaining PSNR of MSRRes-Net. The track had 150 registered participants, and 25 teams submitted the final results. They gauge the state-of-the-art in efficient single image super-resolution.
Single Image Super Resolution Model Using Learnable Weight Factor in Residual Skip Connection
IEEE Access
In single image super resolution problems, the recent feed forward deep learning architectures use residual connections in order to preserve local features and carry them through the next layer. In a simple residual skip connection, all the features of the earlier layer are concatenated with the features of the current layer. A simple concatenation of features does not exploit the fact that some features may be more useful than other features and vice versa. To overcome this limitation, we propose an extended architecture (baby neural network) which will have input as the features learned from the previous layer and output a multiplication factor. This multiplication factor will give importance to the given feature and thus help in training the current layer's features more accurately. The proposed model clearly outperforms existing works.
Frequency-Based Enhancement Network for Efficient Super-Resolution
IEEE Access
Recently, deep convolutional neural networks (CNNs) have provided outstanding performance in single image super-resolution (SISR). Despite their remarkable performance, the lack of high-frequency information in the recovered images remains a core problem. Moreover, as the networks increase in depth and width, deep CNN-based SR methods are faced with the challenge of computational complexity in practice. A promising and under-explored solution is to adapt the amount of compute based on the different frequency bands of the input. To this end, we present a novel Frequency-based Enhancement Block (FEB) which explicitly enhances the information of high frequencies while forwarding low-frequencies to the output. In particular, this block efficiently decomposes features into low-and high-frequency and assigns more computation to high-frequency ones. Thus, it can help the network generate more discriminative representations by explicitly recovering finer details. Our FEB design is simple and generic and can be used as a direct replacement of commonly used SR blocks with no need to change network architectures. We experimentally show that when replacing SR blocks with FEB we consistently improve the reconstruction error, while reducing the number of parameters in the model. Moreover, we propose a lightweight SR model-Frequency-based Enhancement Network (FENet)-based on FEB that matches the performance of larger models. Extensive experiments demonstrate that our proposal performs favorably against the stateof-the-art SR algorithms in terms of visual quality, memory footprint, and inference time. The code is available at https://github.com/pbehjatii/FENet INDEX TERMS Deep learning, frequency-based methods, lightweight architectures, single image super-resolution.