Eugene Siow
commited on
Commit
•
118b5fe
1
Parent(s):
2480f55
Initial commit.
Browse files- README.md +140 -0
- config.json +5 -0
- images/drln_2_4_compare.png +0 -0
- images/drln_4_4_compare.png +0 -0
- pytorch_model_4x.pt +3 -0
README.md
ADDED
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
tags:
|
4 |
+
- super-image
|
5 |
+
- image-super-resolution
|
6 |
+
datasets:
|
7 |
+
- eugenesiow/Div2k
|
8 |
+
- eugenesiow/Set5
|
9 |
+
- eugenesiow/Set14
|
10 |
+
- eugenesiow/BSD100
|
11 |
+
- eugenesiow/Urban100
|
12 |
+
metrics:
|
13 |
+
- pnsr
|
14 |
+
- ssim
|
15 |
+
---
|
16 |
+
# Densely Residual Laplacian Super-Resolution (DRLN)
|
17 |
+
DRLN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Densely Residual Laplacian Super-resolution](https://arxiv.org/abs/1906.12021) by Anwar et al. (2020) and first released in [this repository](https://github.com/saeed-anwar/DRLN).
|
18 |
+
|
19 |
+
The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling.
|
20 |
+
|
21 |
+
![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4](images/drln_4_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4")
|
22 |
+
## Model description
|
23 |
+
Super-Resolution convolutional neural networks have recently demonstrated high-quality restoration for single images. However, existing algorithms often require very deep architectures and long training times. Furthermore, current convolutional neural networks for super-resolution are unable to exploit features at multiple scales and weigh them equally, limiting their learning capability. In this exposition, we present a compact and accurate super-resolution algorithm namely, Densely Residual Laplacian Network (DRLN). The proposed network employs cascading residual on the residual structure to allow the flow of low-frequency information to focus on learning high and mid-level features. In addition, deep supervision is achieved via the densely concatenated residual blocks settings, which also helps in learning from high-level complex features. Moreover, we propose Laplacian attention to model the crucial features to learn the inter and intra-level dependencies between the feature maps. Furthermore, comprehensive quantitative and qualitative evaluations on low-resolution, noisy low-resolution, and real historical image benchmark datasets illustrate that our DRLN algorithm performs favorably against the state-of-the-art methods visually and accurately.
|
24 |
+
## Intended uses & limitations
|
25 |
+
You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
|
26 |
+
### How to use
|
27 |
+
The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
|
28 |
+
```bash
|
29 |
+
pip install super-image
|
30 |
+
```
|
31 |
+
Here is how to use a pre-trained model to upscale your image:
|
32 |
+
```python
|
33 |
+
from super_image import DrlnModel, ImageLoader
|
34 |
+
from PIL import Image
|
35 |
+
import requests
|
36 |
+
|
37 |
+
url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
|
38 |
+
image = Image.open(requests.get(url, stream=True).raw)
|
39 |
+
|
40 |
+
model = DrlnModel.from_pretrained('eugenesiow/drln', scale=2) # scale 2, 3 and 4 models available
|
41 |
+
inputs = ImageLoader.load_image(image)
|
42 |
+
preds = model(inputs)
|
43 |
+
|
44 |
+
ImageLoader.save_image(preds, './scaled_2x.png') # save the output 2x scaled image to `./scaled_2x.png`
|
45 |
+
ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png') # save an output comparing the super-image with a bicubic scaling
|
46 |
+
```
|
47 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Upscale_Images_with_Pretrained_super_image_Models.ipynb "Open in Colab")
|
48 |
+
## Training data
|
49 |
+
The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://huggingface.co/datasets/eugenesiow/Div2k), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
|
50 |
+
## Training procedure
|
51 |
+
### Preprocessing
|
52 |
+
We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
|
53 |
+
Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
|
54 |
+
During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
|
55 |
+
Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
|
56 |
+
|
57 |
+
We need the huggingface [datasets](https://huggingface.co/datasets?filter=task_ids:other-other-image-super-resolution) library to download the data:
|
58 |
+
```bash
|
59 |
+
pip install datasets
|
60 |
+
```
|
61 |
+
The following code gets the data and preprocesses/augments the data.
|
62 |
+
|
63 |
+
```python
|
64 |
+
from datasets import load_dataset
|
65 |
+
from super_image.data import EvalDataset, TrainDataset, augment_five_crop
|
66 |
+
|
67 |
+
augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\
|
68 |
+
.map(augment_five_crop, batched=True, desc="Augmenting Dataset") # download and augment the data with the five_crop method
|
69 |
+
train_dataset = TrainDataset(augmented_dataset) # prepare the train dataset for loading PyTorch DataLoader
|
70 |
+
eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation')) # prepare the eval dataset for the PyTorch DataLoader
|
71 |
+
```
|
72 |
+
### Pretraining
|
73 |
+
The model was trained on GPU. The training code is provided below:
|
74 |
+
```python
|
75 |
+
from super_image import Trainer, TrainingArguments, DrlnModel, DrlnConfig
|
76 |
+
|
77 |
+
training_args = TrainingArguments(
|
78 |
+
output_dir='./results', # output directory
|
79 |
+
num_train_epochs=1000, # total number of training epochs
|
80 |
+
)
|
81 |
+
|
82 |
+
config = DrlnConfig(
|
83 |
+
scale=4, # train a model to upscale 4x
|
84 |
+
)
|
85 |
+
model = DrlnModel(config)
|
86 |
+
|
87 |
+
trainer = Trainer(
|
88 |
+
model=model, # the instantiated model to be trained
|
89 |
+
args=training_args, # training arguments, defined above
|
90 |
+
train_dataset=train_dataset, # training dataset
|
91 |
+
eval_dataset=eval_dataset # evaluation dataset
|
92 |
+
)
|
93 |
+
|
94 |
+
trainer.train()
|
95 |
+
```
|
96 |
+
|
97 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Train_super_image_Models.ipynb "Open in Colab")
|
98 |
+
## Evaluation results
|
99 |
+
The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm).
|
100 |
+
|
101 |
+
Evaluation datasets include:
|
102 |
+
- Set5 - [Bevilacqua et al. (2012)](https://huggingface.co/datasets/eugenesiow/Set5)
|
103 |
+
- Set14 - [Zeyde et al. (2010)](https://huggingface.co/datasets/eugenesiow/Set14)
|
104 |
+
- BSD100 - [Martin et al. (2001)](https://huggingface.co/datasets/eugenesiow/BSD100)
|
105 |
+
- Urban100 - [Huang et al. (2015)](https://huggingface.co/datasets/eugenesiow/Urban100)
|
106 |
+
|
107 |
+
The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
|
108 |
+
|
109 |
+
|Dataset |Scale |Bicubic |drln |
|
110 |
+
|--- |--- |--- |--- |
|
111 |
+
|Set5 |2x |33.64/0.9292 |**** |
|
112 |
+
|Set5 |3x |30.39/0.8678 |**** |
|
113 |
+
|Set5 |4x |28.42/0.8101 |**32.55/0.899** |
|
114 |
+
|Set14 |2x |30.22/0.8683 |**** |
|
115 |
+
|Set14 |3x |27.53/0.7737 |**** |
|
116 |
+
|Set14 |4x |25.99/0.7023 |**28.96/0.7901** |
|
117 |
+
|BSD100 |2x |29.55/0.8425 |**** |
|
118 |
+
|BSD100 |3x |27.20/0.7382 |**** |
|
119 |
+
|BSD100 |4x |25.96/0.6672 |**28.65/0.7692** |
|
120 |
+
|Urban100 |2x |26.66/0.8408 |**** |
|
121 |
+
|Urban100 |3x | |**** |
|
122 |
+
|Urban100 |4x |23.14/0.6573 |**26.56/0.7998** |
|
123 |
+
|
124 |
+
![Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2](images/drln_2_4_compare.png "Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 2")
|
125 |
+
|
126 |
+
You can find a notebook to easily run evaluation on pretrained models below:
|
127 |
+
|
128 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugenesiow/super-image-notebooks/blob/master/notebooks/Evaluate_Pretrained_super_image_Models.ipynb "Open in Colab")
|
129 |
+
|
130 |
+
## BibTeX entry and citation info
|
131 |
+
```bibtex
|
132 |
+
@misc{anwar2019densely,
|
133 |
+
title={Densely Residual Laplacian Super-Resolution},
|
134 |
+
author={Saeed Anwar and Nick Barnes},
|
135 |
+
year={2019},
|
136 |
+
eprint={1906.12021},
|
137 |
+
archivePrefix={arXiv},
|
138 |
+
primaryClass={eess.IV}
|
139 |
+
}
|
140 |
+
```
|
config.json
ADDED
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"bam": false,
|
3 |
+
"data_parallel": false,
|
4 |
+
"model_type": "DRLN"
|
5 |
+
}
|
images/drln_2_4_compare.png
ADDED
images/drln_4_4_compare.png
ADDED
pytorch_model_4x.pt
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b910a591f5ac494cfb278c680b0313cb6c15ffe51e9dc806246c0d846f623726
|
3 |
+
size 138494249
|