GitHub - gladzhang/ART: PyTorch code for our ICLR 2023 paper "Accurate Image Restoration with Attention Retractable Transformer". (original) (raw)

Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan, "Accurate Image Restoration with Attention Retractable Transformer", ICLR, 2023 (Spotlight)

[paper] [arXiv] [supplementary material] [visual results] [pretrained models]

This code is the PyTorch implementation of ART model. Our ART achieves state-of-the-art performance in

• bicubic image SR
• gaussian color image denoising
• real image denoising
• jpeg compression artifact reduction

Abstract: Recently, Transformer-based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independent global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group of tokens are always from a dense area of the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose Attention Retractable Transformer (ART) for image restoration, which presents both dense and sparse attention modules in the network. The sparse attention module allows tokens from sparse areas to interact and thus provides a wider receptive field. Furthermore, the alternating application of dense and sparse attention modules greatly enhances representation ability of Transformer while providing retractable attention on the input image.We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks. Experimental results validate that our proposed ART outperforms state-of-the-art methods on various benchmark datasets both quantitatively and visually. We also provide code and models at https://github.com/gladzhang/ART.

img_092 (x4)	img_098 (x4)	HR	LR	SwinIR	ART (ours)

Requirements

• python 3.8
• pyTorch >= 1.8.0
• NVIDIA GPU + CUDA

Installation

git clone https://github.com/gladzhang/ART.git cd ART pip install -r requirements.txt python setup.py develop

TODO

• Testing on Image SR
• Testing on Color Image Denoising
• Testing on Real Image Denoising
• Testing on JPEG compression artifact reduction
• Training
• More tasks

Contents

1. Models
2. Datasets
3. Training
4. Testing
5. Results
6. Citation
7. Acknowledgement

Models

• We provide the performance on Urban100 (x4, SR), Urban100 (level=50, Color-DN) LIVE1 (q=10, CAR), and SIDD (Real-DN). We use the input 160 × 160 to calculate FLOPS.
• Download the models and put them into the folder experiments/pretrained_models . Go to the folder to find details of directory structure.

Datasets

Used training and testing sets can be downloaded as follows:

Download training and testing datasets and put them into the folder datasets/. Go to the folder to find details of directory structure.

Training

Train on SR

1. Please download the corresponding training datasets and put them in the folder datasets/DF2K. Download the testing datasets and put them in the folder datasets/SR.
2. Follow the instructions below to begin training our ART model.

train ART for SR task, cropped input=64×64, 4 GPUs, batch size=8 per GPU

python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_SR_x2.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_SR_x3.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_SR_x4.yml --launcher pytorch

train ART-S for SR task, cropped input=64×64, 4 GPUs, batch size=8 per GPU

python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_S_SR_x2.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_S_SR_x3.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_S_SR_x4.yml --launcher pytorch
Run the script then you can find the generated experimental logs in the folder experiments.

Train on Gaussian Color Image Denoising

1. Please download the corresponding training datasets and put them in the folder datasets/DFWB_RGB. Download the testing datasets and put them in the folder datasets/ColorDN.
2. Follow the instructions below to begin training our ART model.

train ART for ColorDN task, cropped input=128×128, 4 GPUs, batch size=2 per GPU

python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_ColorDN_level15.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_ColorDN_level25.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_ColorDN_level50.yml --launcher pytorch
Run the script then you can find the generated experimental logs in the folder experiments.

Train on Real Image Denoising

1. Please download the corresponding training datasets and put them in the folder datasets/SIDD. Note that we provide both training and validating files, which are already processed.
2. Go to folder 'realDenoising'. Follow the instructions below to train our ART model.

go to the folder

cd realDenoising

set the new environment (BasicSRv1.2.0), which is the same with Restormer for training.

python setup.py develop --no_cuda_ext

train ART for RealDN task, 8 GPUs

python -m torch.distributed.launch --nproc_per_node=8 --master_port=2414 basicsr/train.py -opt options/train_ART_RealDN.yml --launcher pytorch
Run the script then you can find the generated experimental logs in the folder realDenoising/experiments. 3. Remember to go back to the original environment if you finish all the training or testing about real image denoising task. This is a friendly hint in order to prevent confusion in the training environment.

Tips here. Go back to the original environment (BasicSRv1.3.5) after finishing all the training or testing about real image denoising.

cd ..
python setup.py develop

Train on JPEG Compression Artifact Reduction

1. Please download the corresponding training datasets and put them in the folder datasets/DFWB_CAR. Download the testing datasets and put them in the folder datasets/CAR.
2. Follow the instructions below to begin training our ART model.

train ART for CAR task, cropped input=126×126, 4 GPUs, batch size=2 per GPU

python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_CAR_q10.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_CAR_q30.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=4 --master_port=2414 basicsr/train.py -opt options/train/train_ART_CAR_q40.yml --launcher pytorch
Run the script then you can find the generated experimental logs in the folder experiments.

Testing

Test on SR

Test with ground-truth images

1. Please download the corresponding testing datasets and put them in the folder datasets/SR. Download the corresponding models and put them in the folder experiments/pretrained_models.
2. Follow the instructions below to begin testing our ART model.

test ART model for image SR. You can find corresponding results in Table 2 of the main paper.

python basicsr/test.py -opt options/test/test_ART_SR_x2.yml
python basicsr/test.py -opt options/test/test_ART_SR_x3.yml
python basicsr/test.py -opt options/test/test_ART_SR_x4.yml

test ART-S model for image SR. You can find corresponding results in Table 2 of the main paper.

python basicsr/test.py -opt options/test/test_ART_S_SR_x2.yml
python basicsr/test.py -opt options/test/test_ART_S_SR_x3.yml
python basicsr/test.py -opt options/test/test_ART_S_SR_x4.yml

Test without ground-truth images

1. Please upload the images that need to be upscaled, and put them in the folder datasets/example. Download the corresponding models and put them in the folder experiments/pretrained_models.
2. Choose the upscale size and follow the instructions below to apply our ART model to upscale the provided images.

apply ART model for image SR.

python basicsr/test.py -opt options/apply/test_ART_SR_x2_without_groundTruth.yml
python basicsr/test.py -opt options/apply/test_ART_SR_x3_without_groundTruth.yml
python basicsr/test.py -opt options/apply/test_ART_SR_x4_without_groundTruth.yml
Run the script then you can find the output visual results in the automatically generated folder results.

Test on Gaussian Color Image Denoising

1. Please download the corresponding testing datasets and put them in the folder datasets/ColorDN. Download the corresponding models and put them in the folder experiments/pretrained_models.
2. Follow the instructions below to begin testing our ART model.

test ART model for Color Image Denoising. You can find corresponding results in Table 4 of the main paper.

python basicsr/test.py -opt options/test/test_ART_ColorDN_level15.yml
python basicsr/test.py -opt options/test/test_ART_ColorDN_level25.yml
python basicsr/test.py -opt options/test/test_ART_ColorDN_level50.yml

Test on Real Image Denoising

1. Download the SIDD test and DND test. Place them in datasets/RealDN. Download the corresponding models and put them in the folder experiments/pretrained_models.
2. Go to folder 'realDenoising'. Follow the instructions below to test our ART model. The output is in realDenoising/results/Real_Denoising.

go to the folder

cd realDenoising

set the new environment (BasicSRv1.2.0), which is the same with Restormer for testing.

python setup.py develop --no_cuda_ext

test our ART (training total iterations = 300K) on SSID

python test_real_denoising_sidd.py

test our ART (training total iterations = 300K) on DND

python test_real_denoising_dnd.py 3. Run the scripts below to reproduce PSNR/SSIM on SIDD. You can find corresponding results in Table 7 of the main paper. 4. For PSNR/SSIM scores on DND, you can upload the genetated DND mat files to the online server and get the results. 5. Remerber to go back to the original environment if you finish all the training or testing about real image denoising task. This is a friendly hint in order to prevent confusion in the training environment.

Tips here. Go back to the original environment (BasicSRv1.3.5) after finishing all the training or testing about real image denoising.

cd ..
python setup.py develop

Test on JPEG Compression Artifact Reduction

1. Please download the corresponding testing datasets and put them in the folder datasets/CAR. Download the corresponding models and put them in the folder experiments/pretrained_models.
2. Follow the instructions below to begin testing our ART model.

ART model for JPEG CAR. You can find corresponding results in Table 5 of the main paper.

python basicsr/test.py -opt options/test/test_ART_CAR_q10.yml
python basicsr/test.py -opt options/test/test_ART_CAR_q30.yml
python basicsr/test.py -opt options/test/test_ART_CAR_q40.yml

Results

We provide the results on image SR, color image denoising, real image denoising, and JPEG compression artifact reduction here. More results can be found in the main paper. The visual results of ART can be downloaded here.

Image SR (click to expan)

• Results of Table 2 in the main paper

• Visual results

Color Image Denoising(click to expan)

• Results of Table 4 in the main paper

• Visual results

Real Image Denoising (click to expan)

• Results of Table 7 in the main paper

JPEG Compression Artifact Reduction (click to expan)

• Results of Table 5 in the main paper

Citation

If you find the code helpful in your resarch or work, please cite the following paper(s).

@inproceedings{zhang2023accurate,
  title={Accurate Image Restoration with Attention Retractable Transformer},
  author={Zhang, Jiale and Zhang, Yulun and Gu, Jinjin and Zhang, Yongbing and Kong, Linghe and Yuan, Xin},
  booktitle={ICLR},
  year={2023}
}

Acknowledgement

This work is released under the Apache 2.0 license. The codes are based on BasicSR and Restormer. Please also follow their licenses. Thanks for their awesome works.