Kuldeep Purohit | Indian Institute of Technology Mandi (original) (raw)

Papers by Kuldeep Purohit

Image restoration is the task of recovering a clean image from a degraded version. In most cases,... more Image restoration is the task of recovering a clean image from a degraded version. In most cases, the degradation is spatially varying, and it requires the restoration network to both localize and restore the affected regions. In this paper, we present a new approach suitable for handling the image-specific and spatially-varying nature of degradation in images affected by practically occurring artifacts such as blur, rain-streaks. We decompose the restoration task into two stages of degradation localization and degraded region-guided restoration, unlike existing methods which directly learn a mapping between the degraded and clean images. Our premise is to use the auxiliary task of degradation mask prediction to guide the restoration process. We demonstrate that the model trained for this auxiliary task contains vital region knowledge, which can be exploited to guide the restoration network's training using attentive knowledge distillation technique. Further, we propose mask-gui...

Lecture Notes in Computer Science, 2019

In this paper, we describe the Perceptual Image Restoration and Manipulation (PIRM) workshop chal... more In this paper, we describe the Perceptual Image Restoration and Manipulation (PIRM) workshop challenge on spectral image superresolution, motivate its structure and conclude on results obtained by the participants. The challenge is one of the first of its kind, aiming at leveraging modern machine learning techniques to achieve spectral image super-resolution. It comprises of two tracks. The first of these (Track 1) is about example-based single spectral image super-resolution. The second one (Track 2) is on colour-guided spectral image super-resolution. In this manner, Track 1 focuses on the problem of super-resolving the spatial resolution of spectral images given training pairs of low and high spatial resolution spectral images. Track 2, on the other hand, aims to leverage the inherently higher spatial resolution of colour (RGB) cameras and the link between spectral and trichromatic images of the scene. The challenge in both tracks is then to recover a super-resolved image making use of low-resolution imagery at the input. We also elaborate upon the methods used by the participants, summarise the results and discuss their rankings. Mehrdad Sheoiby, Antonio Robles-Kelly, and Radu Timofte are the PIRM2018 organizers, while the other authors participated in the challenge.

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021

Image inpainting methods have shown significant improvements by using deep neural networks recent... more Image inpainting methods have shown significant improvements by using deep neural networks recently. However, many of these techniques often create distorted structures or blurry inconsistent textures. The problem is rooted in the encoder layers' ineffectiveness in building a complete and faithful embedding of the missing regions from scratch. Existing solutions like course-to-fine, progressive refinement, structural guidance, etc. suffer from huge computational overheads owing to multiple generator networks, limited ability of handcrafted features, and sub-optimal utilization of the information present in the ground truth. We propose a distillation-based approach for inpainting, where we provide direct feature level supervision while training. We deploy cross and self-distillation techniques and design a dedicated completion-block in encoder to produce more accurate encoding of the holes. Next, we demonstrate how an inpainting network's attention module can improve by leveraging a distillation-based attention transfer technique and further enhance coherence by using a pixeladaptive global-local feature fusion. We conduct extensive evaluations on multiple datasets to validate our method. Along with achieving significant improvements over previous SOTA methods, the proposed approach's effectiveness is also demonstrated through its ability to improve existing inpainting works.

Carbon Nanotubes (CNTs) are the allotropes of carbon which belong to the fulle rene structural fa... more Carbon Nanotubes (CNTs) are the allotropes of carbon which belong to the fulle rene structural family. These are cylindrical structures with at least one end closed with a buckyball structure hemisphere. They are few nano meter in diameter and have tensile strength of ~63GPa and young's modulus of ~1TPa. On the basis of structures carbon nanotubes can be classified as Single-walled (SWNT), Multi -walled (MWNT), Polymerized SWNT, Nanotorus and Nanobuds. Carbon Nanotubes can behav e

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

This paper reviews the video extreme super-resolution challenge associated with the AIM 2019 work... more This paper reviews the video extreme super-resolution challenge associated with the AIM 2019 workshop, with emphasis on submitted solutions and results. Video extreme super-resolution (×16) is a highly challenging problem, because 256 pixels need to be estimated for each single pixel in the low-resolution (LR) input. Contrary to single image super-resolution (SISR), video provides temporal information, which can be additionally leveraged to restore the heavily downscaled videos and is imperative for any video super-resolution (VSR) method. The challenge is composed of two tracks, to find the best performing method for fully supervised VSR (track 1) and to find the solution which generates the perceptually best looking outputs (track 2). A new video dataset, called Vid3oC, is introduced together with the challenge.

Image inpainting methods have shown significant improvements by using deep neural networks recent... more Image inpainting methods have shown significant improvements by using deep neural networks recently. However, many of these techniques often create distorted structures or blurry textures inconsistent with surrounding areas. The problem is rooted in the encoder layers' ineffectiveness in building a complete and faithful embedding of the missing regions. To address this problem, two-stage approaches deploy two separate networks for a coarse and fine estimate of the inpainted image. Some approaches utilize handcrafted features like edges or contours to guide the reconstruction process. These methods suffer from huge computational overheads owing to multiple generator networks, limited ability of handcrafted features, and sub-optimal utilization of the information present in the ground truth. Motivated by these observations, we propose a distillation based approach for inpainting, where we provide direct feature level supervision for the encoder layers in an adaptive manner. We dep...

This paper tackles the problem of dynamic scene deblurring. Although end-to-end fully convolution... more This paper tackles the problem of dynamic scene deblurring. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by a simple increment in the number of generic convolution layers, kernel-size, which comes with the burden of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations within and across different images. We also propose an effective content-aware global-local filtering module that significantly improves the performance by considering not only the global dependencies of the pixel but also dynamically using the neighboring pixels. We use a patch hierarchical attentive architecture composed of the above module that implicitly discover the spatial variations in the blur present in the in...

We present a solution for the goal of extracting a video from a single motion blurred image to se... more We present a solution for the goal of extracting a video from a single motion blurred image to sequentially reconstruct the clear views of a scene as beheld by the camera during the time of exposure. We first learn motion representation from sharp videos in an unsupervised manner through training of a convolutional recurrent video autoencoder network that performs a surrogate task of video reconstruction. Once trained, it is employed for guided training of a motion encoder for blurred images. This network extracts embedded motion information from the blurred image to generate a sharp video in conjunction with the trained recurrent video decoder. As an intermediate step, we also design an efficient architecture that enables real-time single image deblurring and outperforms competing methods across all factors: accuracy, speed, and compactness. Experiments on real scenes and standard datasets demonstrate the superiority of our framework over the state-of-the-art and its ability to gen...

Efficiency of gradient propagation in intermediate layers of convolutional neural networks is of ... more Efficiency of gradient propagation in intermediate layers of convolutional neural networks is of key importance for super-resolution task. To this end, we propose a deep architecture for single image super-resolution (SISR), which is built using efficient convolutional units we refer to as mixed-dense connection blocks (MDCB). The design of MDCB combines the strengths of both residual and dense connection strategies, while overcoming their limitations. To enable super-resolution for multiple factors, we propose a scale-recurrent framework which reutilizes the filters learnt for lower scale factors recursively for higher factors. This leads to improved performance and promotes parametric efficiency for higher factors. We train two versions of our network to enhance complementary image qualities using different loss configurations. We further employ our network for video super-resolution task, where our network learns to aggregate information from multiple frames and maintain spatio-t...

In this paper, we address the problem of dynamic scene deblurring in the presence of motion blur.... more In this paper, we address the problem of dynamic scene deblurring in the presence of motion blur. Restoration of images affected by severe blur necessitates a network design with a large receptive field, which existing networks attempt to achieve through simple increment in the number of generic convolution layers, kernel-size, or the scales at which the image is processed. However, increasing the network capacity in this manner comes at the expense of increase in model size and inference time, and ignoring the non-uniform nature of blur. We present a new architecture composed of spatially adaptive residual learning modules that implicitly discover the spatially varying shifts responsible for non-uniform blur in the input image and learn to modulate the filters. This capability is complemented by a self-attentive module which captures non-local relationships among the intermediate features and enhances the receptive field. We then incorporate a spatiotemporal recurrent module in the design to also facilitate efficient video deblurring. Our networks can implicitly model the spatially-varying deblurring process, while dispensing with multi-scale processing and large filters entirely. Extensive qualitative and quantitative comparisons with prior art on benchmark dynamic scene deblurring datasets clearly demonstrate the superiority of the proposed networks via reduction in model-size and significant improvements in accuracy and speed, enabling almost real-time deblurring.

arXiv: Computer Vision and Pattern Recognition, Aug 19, 2021

Existing works on motion deblurring either ignore the effects of depth-dependent blur or work wit... more Existing works on motion deblurring either ignore the effects of depth-dependent blur or work with the assumption of a multi-layered scene wherein each layer is modeled in the form of fronto-parallel plane. In this work, we consider the case of 3D scenes with piecewise planar structure i.e., a scene that can be modeled as a combination of multiple planes with arbitrary orientations. We first propose an approach for estimation of normal of a planar scene from a single motion blurred observation. We then develop an algorithm for automatic recovery of number of planes, the parameters corresponding to each plane, and camera motion from a single motion blurred image of a multiplanar 3D scene. Finally, we propose a first-of-its-kind approach to recover the planar geometry and latent image of the scene by adopting an alternating minimization framework built on our findings. Experiments on synthetic and real data reveal that our proposed method achieves state-of-the-art results.

Recent advances in the design of convolutional neural network (CNN) have yielded significant impr... more Recent advances in the design of convolutional neural network (CNN) have yielded significant improvements in the performance of image super-resolution (SR). The boost in performance can be attributed to the presence of residual or dense connections within the intermediate layers of these networks. The efficient combination of such connections can reduce the number of parameters drastically while maintaining the restoration quality. In this paper, we propose a scale recurrent SR architecture built upon units containing series of dense connections within a residual block (Residual Dense Blocks (RDBs)) that allow extraction of abundant local features from the image. Our scale recurrent design delivers competitive performance for higher scale factors while being parametrically more efficient as compared to current state-of-the-art approaches. To further improve the performance of our network, we employ multiple residual connections in intermediate layers (referred to as Multi-Residual D...

ArXiv, 2021

In practice, images can contain different amounts of noise for different color channels, which is... more In practice, images can contain different amounts of noise for different color channels, which is not acknowledged by existing super-resolution approaches. In this paper, we propose to super-resolve noisy color images by considering the color channels jointly. Noise statistics are blindly estimated from the input low-resolution image and are used to assign different weights to different color channels in the data cost. Implicit low-rank structure of visual data is enforced via nuclear norm minimization in association with adaptive weights, which is added as a regularization term to the cost. Additionally, multi-scale details of the image are added to the model through another regularization term that involves projection onto PCA basis, which is constructed using similar patches extracted across different scales of the input image. The results demonstrate the super-resolving capability of the approach in real scenarios. Additional

Machine Vision and Applications

This paper reviews the AIM 2019 challenge on real world super-resolution. It focuses on the parti... more This paper reviews the AIM 2019 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided in the challenge. In Track 1: Source Domain the aim is to super-resolve such images while preserving the low level image characteristics of the source input domain. In Track 2: Target Domain a set of high-quality images is also provided for training, that defines the output domain and desired quality of the super-resolved images. To allow for quantitative evaluation, the source input images in both tracks are constructed using artificial, but realistic, image degradations. The challenge is the first of its kind, aiming to advance the state-of-the-art and provide a standard benchmark for this newly emerging task. In total 7 teams competed in the final testing phase, demo...

Existing works on motion deblurring either ignore the effects of depth-dependent blur or work wit... more Existing works on motion deblurring either ignore the effects of depth-dependent blur or work with the assumption of a multi-layered scene wherein each layer is modeled in the form of fronto-parallel plane. In this work, we consider the case of 3D scenes with piecewise planar structure i.e., a scene that can be modeled as a combination of multiple planes with arbitrary orientations. We first propose an approach for estimation of normal of a planar scene from a single motion blurred observation. We then develop an algorithm for automatic recovery of a number of planes, the parameters corresponding to each plane, and camera motion from a single motion blurred image of a multiplanar 3D scene. Finally, we propose a first-of-its-kind approach to recover the planar geometry and latent image of the scene by adopting an alternating minimization framework built on our findings. Experiments on synthetic and real data reveal that our proposed method achieves state-of-the-art results.

This paper tackles the problem of motion deblurring of dynamic scenes. Although end-to-end fully ... more This paper tackles the problem of motion deblurring of dynamic scenes. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by increasing the number of generic convolution layers and kernel-size, but this comes at the expense of of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations across different spatial locations and process each test image adaptively. We also propose an effective content-aware global-local filtering module that significantly improves performance by considering not only global dependencies but also by dynamically exploiting neighbouring pixel information. We use a patch-hierarchical attentive architecture composed of the above module that implicitly discovers the spati...

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing

In practice, images can contain different amounts of noise for different color channels, which is... more In practice, images can contain different amounts of noise for different color channels, which is not acknowledged by existing superresolution approaches. In this paper, we propose to super-resolve noisy color images by considering the color channels jointly. Noise statistics are blindly estimated from the input low-resolution image and are used to assign different weights to different color channels in the data cost. Implicit low-rank structure of visual data is enforced via nuclear norm minimization in association with adaptive weights, which is added as a regularization term to the cost. Additionally, multi-scale details of the image are added to the model through another regularization term that involves projection onto PCA basis, which is constructed using similar patches extracted across different scales of the input image. The results demonstrate the super-resolving capability of the approach in real scenarios.

Lecture Notes in Computer Science, 2019

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Image inpainting methods have shown significant improvements by using deep neural networks recent... more Image inpainting methods have shown significant improvements by using deep neural networks recently. However, many of these techniques often create distorted structures or blurry textures inconsistent with surrounding areas. The problem is rooted in the encoder layers' ineffectiveness in building a complete and faithful embedding of the missing regions. To address this problem, two-stage approaches deploy two separate networks for a coarse and fine estimate of the inpainted image. Some approaches utilize handcrafted features like edges or contours to guide the reconstruction process. These methods suffer from huge computational overheads owing to multiple generator networks, limited ability of handcrafted features, and sub-optimal utilization of the information present in the ground truth. Motivated by these observations, we propose a distillation based approach for inpainting, where we provide direct feature level supervision for the encoder layers in an adaptive manner. We dep...

arXiv: Computer Vision and Pattern Recognition, Aug 19, 2021

ArXiv, 2021

Machine Vision and Applications

Existing works on motion deblurring either ignore the effects of depth-dependent blur or work wit... more Existing works on motion deblurring either ignore the effects of depth-dependent blur or work with the assumption of a multi-layered scene wherein each layer is modeled in the form of fronto-parallel plane. In this work, we consider the case of 3D scenes with piecewise planar structure i.e., a scene that can be modeled as a combination of multiple planes with arbitrary orientations. We first propose an approach for estimation of normal of a planar scene from a single motion blurred observation. We then develop an algorithm for automatic recovery of a number of planes, the parameters corresponding to each plane, and camera motion from a single motion blurred image of a multiplanar 3D scene. Finally, we propose a first-of-its-kind approach to recover the planar geometry and latent image of the scene by adopting an alternating minimization framework built on our findings. Experiments on synthetic and real data reveal that our proposed method achieves state-of-the-art results.

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing

In practice, images can contain different amounts of noise for different color channels, which is... more In practice, images can contain different amounts of noise for different color channels, which is not acknowledged by existing superresolution approaches. In this paper, we propose to super-resolve noisy color images by considering the color channels jointly. Noise statistics are blindly estimated from the input low-resolution image and are used to assign different weights to different color channels in the data cost. Implicit low-rank structure of visual data is enforced via nuclear norm minimization in association with adaptive weights, which is added as a regularization term to the cost. Additionally, multi-scale details of the image are added to the model through another regularization term that involves projection onto PCA basis, which is constructed using similar patches extracted across different scales of the input image. The results demonstrate the super-resolving capability of the approach in real scenarios.