Initial commit.

README.md

@@ -0,0 +1,146 @@
+---
+license: apache-2.0
+tags:
+- image-super-resolution
+datasets:
+- div2k
+metrics:
+- pnsr
+- ssim
+---
+# Multi-scale Residual Network for Image Super-Resolution (MSRN)
+MSRN model pre-trained on DIV2K (800 images training, augmented to 4000 images, 100 images validation) for 2x, 3x and 4x image super resolution. It was introduced in the paper [Multi-scale Residual Network for Image Super-Resolution](https://openaccess.thecvf.com/content_ECCV_2018/html/Juncheng_Li_Multi-scale_Residual_Network_ECCV_2018_paper.html) by Li et al. (2018) and first released in [this repository](https://github.com/MIVRC/MSRN-PyTorch). 
+
+The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling x2 and model upscaling x2.
+
+![Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 4](images/msrn_4_4_compare.png "Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 4")
+## Model description
+The MSRN model proposes a feature extraction structure called the multi-scale residual block. This module can "adaptively detect image features at different scales" and "exploit the potential features of the image".
+
+This model also applies the balanced attention (BAM) method invented by [Wang et al. (2021)](https://arxiv.org/abs/2104.07566) to further improve the results.
+## Intended uses & limitations
+You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
+### How to use
+The model can be used with the [super_image](https://github.com/eugenesiow/super-image) library:
+```bash
+pip install super-image
+```
+Here is how to use a pre-trained model to upscale your image:
+```python
+from super_image import MsrnModel, ImageLoader
+from PIL import Image
+import requests
+
+url = 'https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'
+image = Image.open(requests.get(url, stream=True).raw)
+
+model = MsrnModel.from_pretrained('eugenesiow/msrn-bam', scale=2)      # scale 2, 3 and 4 models available
+inputs = ImageLoader.load_image(image)
+preds = model(inputs)
+
+ImageLoader.save_image(preds, './scaled_2x.png')                        # save the output 2x scaled image to `./scaled_2x.png`
+ImageLoader.save_compare(inputs, preds, './scaled_2x_compare.png')      # save an output comparing the super-image with a bicubic scaling
+```
+## Training data
+The models for 2x, 3x and 4x image super resolution were pretrained on [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/), a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of  100 validation images (images numbered 801 to 900). 
+## Training procedure
+### Preprocessing
+We follow the pre-processing and training method of [Wang et al.](https://arxiv.org/abs/2104.07566).
+Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
+During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches. 
+Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image. 
+
+The following code provides some helper functions to preprocess the data.
+```python
+from super_image.data import EvalDataset, TrainAugmentDataset, DatasetBuilder
+
+DatasetBuilder.prepare(
+    base_path='./DIV2K/DIV2K_train_HR',
+    output_path='./div2k_4x_train.h5',
+    scale=4,
+    do_augmentation=True
+)
+DatasetBuilder.prepare(
+    base_path='./DIV2K/DIV2K_val_HR',
+    output_path='./div2k_4x_val.h5',
+    scale=4,
+    do_augmentation=False
+)
+train_dataset = TrainAugmentDataset('./div2k_4x_train.h5', scale=4)
+val_dataset = EvalDataset('./div2k_4x_val.h5')
+```
+### Pretraining
+The model was trained on GPU. The training code is provided below:
+```python
+from super_image import Trainer, TrainingArguments, MsrnModel, MsrnConfig
+
+training_args = TrainingArguments(
+    output_dir='./results',                 # output directory
+    num_train_epochs=1000,                  # total number of training epochs
+)
+
+config = MsrnConfig(
+    scale=4,                                # train a model to upscale 4x
+    bam=True,                               # apply balanced attention to the network
+    supported_scales=[2, 3, 4],
+)
+model = MsrnModel(config)
+
+trainer = Trainer(
+    model=model,                         # the instantiated model to be trained
+    args=training_args,                  # training arguments, defined above
+    train_dataset=train_dataset,         # training dataset
+    eval_dataset=val_dataset             # evaluation dataset
+)
+
+trainer.train()
+```
+## Evaluation results
+The evaluation metrics include [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio#Quality_estimation_with_PSNR) and [SSIM](https://en.wikipedia.org/wiki/Structural_similarity#Algorithm). 
+
+Evaluation datasets include:
+- Set5 - [Bevilacqua et al. (2012)](http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html)
+- Set14 - [Zeyde et al. (2010)](https://sites.google.com/site/romanzeyde/research-interests)
+- BSD100 - [Martin et al. (2001)](https://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/)
+- Urban100 - [Huang et al. (2015)](https://sites.google.com/site/jbhuang0604/publications/struct_sr)
+
+The results columns below are represented below as `PSNR/SSIM`. They are compared against a Bicubic baseline.
+
+|Dataset  	    |Scale      |Bicubic  	        |msrn-bam  	                    |
+|---	        |---	    |---	            |---	                        |
+|Set5  	        |2x         |33.64/0.9292       |**38.023705/0.960794**         |
+|Set5  	        |3x  	    |30.39/0.8678  	    |**35.155403/0.940999**  	    |
+|Set5  	        |4x  	    |28.42/0.8101  	    |**32.263668/0.89554**          |
+|Set14  	    |2x         |30.22/0.8683  	    |**33.635643/0.917744**  	    |
+|Set14  	    |3x         |27.53/0.7737  	    |**30.974932/0.857354**  	    |
+|Set14  	    |4x         |25.99/0.7023  	    |**28.660543/0.782889**  	    |
+|BSD100  	    |2x  	    |29.55/0.8425  	    |**32.208752/0.899763**  	    |
+|BSD100  	    |3x  	    |27.20/0.7382  	    |**29.668056/0.820912**  	    |
+|BSD100  	    |4x  	    |25.96/0.6672  	    |**27.614033/0.736893**  	    |
+|Urban100  	    |2x  	    |26.66/0.8408  	    |**32.084557/0.927621**  	    |
+|Urban100  	    |3x  	    |  	                |**29.314505/0.873682**  	    |
+|Urban100  	    |4x  	    |23.14/0.6573  	    |**26.100685/0.785711**  	    |
+
+![Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 2](images/msrn_2_4_compare.png "Comparing Bicubic upscaling against the models x2 upscaling on Set5 Image 2")
+
+## BibTeX entry and citation info
+```bibtex
+@misc{wang2021bam,
+    title={BAM: A Lightweight and Efficient Balanced Attention Mechanism for Single Image Super Resolution}, 
+    author={Fanyi Wang and Haotian Hu and Cheng Shen},
+    year={2021},
+    eprint={2104.07566},
+    archivePrefix={arXiv},
+    primaryClass={eess.IV}
+}
+```
+
+```bibtex
+@InProceedings{Li_2018_ECCV,
+    author = {Li, Juncheng and Fang, Faming and Mei, Kangfu and Zhang, Guixu},
+    title = {Multi-scale Residual Network for Image Super-Resolution},
+    booktitle = {The European Conference on Computer Vision (ECCV)},
+    month = {September},
+    year = {2018}
+}
+```

config.json

@@ -0,0 +1,10 @@
+{
+  "_name_or_path": "eugenesiow/msrn-bam",
+  "data_parallel": true,
+  "model_type": "MSRN",
+  "bam": true,
+  "n_feats": 64,
+  "n_blocks": 8,
+  "rgb_range": 255,
+  "supported_scales": [2,3,4]
+}

pytorch_model_2x.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cab3f5885433b1871da3e061af773a65cb59f1aa58ca5d14be3e5ef01dec8e5
+size 23786629

pytorch_model_3x.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84a5c6b9993b0bfde05bba2aea52d45be7d6333e9ab2cd9e6fe64b452da0bdba
+size 24525189

pytorch_model_4x.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1182755427e413d8914c9833b44d02e30653998fc01a7876f48f21c8247105e7
+size 24378241