metadata

license: etalab-2.0
tags:
  - segmentation
  - pytorch
  - aerial imagery
  - landcover
  - IGN
model-index:
  - name: FLAIR-INC_rgbie_12cl_resnet34-unet
    results:
      - task:
          type: semantic-segmentation
        dataset:
          name: IGNF/FLAIR/
          type: earth-observation-dataset
        metrics:
          - name: mIoU
            type: mIoU
            value: 62.716
          - name: Overall Accuracy
            type: OA
            value: 76.509
          - name: Fscore
            type: Fscore
            value: 75.907
          - name: Precision
            type: Precision
            value: 76.525
          - name: Recall
            type: Recall
            value: 75.714
          - name: IoU Buildings
            type: IoU
            value: 82.564
          - name: IoU Pervious surface
            type: IoU
            value: 54.149
          - name: IoU Impervious surface
            type: IoU
            value: 73.807
          - name: IoU Bare soil
            type: IoU
            value: 59.013
          - name: IoU Water
            type: IoU
            value: 87.216
          - name: IoU Coniferous
            type: IoU
            value: 61.591
          - name: IoU Deciduous
            type: IoU
            value: 72.225
          - name: IoU Brushwood
            type: IoU
            value: 31.187
          - name: IoU Vineyard
            type: IoU
            value: 76.105
          - name: IoU Herbaceous vegetation
            type: IoU
            value: 51.34
          - name: IoU Agricultural land
            type: IoU
            value: 57.558
          - name: IoU Plowed land
            type: IoU
            value: 45.84
pipeline_tag: image-segmentation

FLAIR model collection

The FLAIR models are a collection of semantic segmentation models initially developed to classify land cover on very high resolution aerial images (more specifically the French BD ORTHO® product). The distributed pre-trained models differ in their :

dataset for training : FLAIR dataset or the increased version of this dataset FLAIR-INC (x 3.5 patches). Only the FLAIR dataset is open at the moment.
input modalities : RGB (natural colours), RGBI (natural colours + infrared), RGBIE (natural colours + infrared + elevation)
model architecture : resnet34_unet (U-Net with a Resnet-34 encoder), deeplab, fpn, mit
target class nomenclature : 12cl (12 land cover classes) or 15cl (15 land cover classes)

FLAIR-INC_rgbie_12cl_resnet34-unet

The general characteristics of this specific model FLAIR-INC_rgbie_12cl_resnet34-unet are :

Trained with the FLAIR-INC dataset
Trained with the SegmentationModelsPytorch library
RGBIE images (true colours + infrared + elevation)
U-Net with a Resnet-34 encoder
12 class nomenclature : [building, pervious surface, impervious surface, bare soil, water, coniferous, deciduous, brushwood, vineyard, herbaceous, agricultural land, plowed land]

Model Informations

Code repository: https://github.com/IGNF/FLAIR-1
Paper: https://arxiv.org/pdf/2211.12979.pdf
Developed by: IGN
Compute infrastructure:
- software: python, pytorch-lightning
- hardware: HPC/AI resources provided by GENCI-IDRIS
License: Etalab 2.0

Uses

Although the model can be applied to other type of very high spatial earth observation images, it was initially developed to tackle the problem of classifying aerial images acquired on the French Territory. The product called (BD ORTHO®) has its own spatial and radiometric specifications. The model is not intended to be generic to other type of very high spatial resolution images but specific to BD ORTHO images. Consequently, the model’s prediction would improve if the user images are similar to the original ones.

Radiometry of input images : The BD ORTHO input images are distributed in 8-bit encoding format per channel. When traning the model, input normalization was performed (see section Training Details). It is recommended that the user apply the same type of input normalization while inferring the model.

Multi-domain model : The FLAIR-INC dataset that was used for training is composed of 75 radiometric domains. In the case of aerial images, domain shifts are frequent and are mainly due to : the date of acquisition of the aerial survey (from april to november), the spatial domain (equivalent to a french department administrative division) and downstream radiometric processing. By construction (sampling 75 domains) the model is robust to these shifts, and can be applied to any images of the (BD ORTHO® product).

Specification for the Elevation channel : The fifth dimension of the RGBIE images is the Elevation (height of building and vegetation). This information is encoded in a 8-bit encoding format. When decoded to [0,255] ints, a difference of 1 should coresponds to 0.2 meters step of elevation difference.

Land Cover classes of prediction : The orginial class nomenclature of the FLAIR Dataset encompasses 19 classes (See the FLAIR dataset page for details). This model was trained to be coherent withe the FLAIR#1 scientific challenge in which contestants were evaluated of the first 12 classes of the nomenclature. Classes with label greater than 12 were desactivated during training. As a result, the logits produced by the model are of size 19x1, but classes n° 13, 14, 15, 16, 17, 18 and 19 should appear at 0 in the logits and should not be present in the final argmax product.

Bias, Risks, Limitations and Recommendations

Using the model on input images with other spatial resolution : The FLAIR-INC_rgbie_12cl_resnet34-unet model was trained with fixed scale conditions. All patches used for training are derived from aerial images with 0.2 meters spatial resolution. Only flip and rotate augmentations were performed during the training process.
No data augmentation method concerning scale change was used during training. The user should pay attention that generalization issues can occur while applying this model to images that have different spatial resolutions.

Using the model for other remote sensing sensors : The FLAIR-INC_rgbie_12cl_resnet34-unet was trained with aerial images of the (BD ORTHO® product) that encopass very specific radiometric image processing. Using the model on other type of aerial images or satellite images may imply the use of transfer learning or domain adaptation techniques.

Using the model on other spatial areas : The FLAIR-INC_rgbie_12cl_resnet34-unet model was trained on patches reprensenting the French Metropolitan territory. The user should be aware that applying the model to other type of landscapes may imply a drop in model metrics.

How to Get Started with the Model

Visit (https://github.com/IGNF/FLAIR-1) to use the model. Fine-tuning and prediction tasks are detailed in the README file.

Training Details

Training Data

218 400 patches of 512 x 512 pixels were used to train the FLAIR-INC_rgbie_12cl_resnet34-unet model. The train/validation split was performed patchwise to obtain a 80% / 20% distribution between train and validation. Annotation was performed at the zone level (~100 patches per zone). Spatial independancy between patches is guaranted as patches from the same zone were assigned to the same set (TRAIN or VALIDATION). The following number of patches were used for train and validation : | TRAIN set | 174 700 patches | | VALIDATION set | 43 700 patchs |

Training Procedure

Preprocessing

For traning the model, input normalization was performed to center-reduce (a mean=0 and a standard deviation = 1, channel wise) the dataset. We used the statistics of TRAIN+VALIDATION for input normalization. It is recommended that the user apply the same type of input normalization.

Statistics of the TRAIN+VALIDATION set :

Modalities	Mean (Train + Validation)	Std (Train + Validation)
Red Channel (R)	105.08	52.17
Green Channel (G)	110.87	45.38
Blue Channel (B)	101.82	44.00
Infrared Channel (I)	106.38	39.69
Elevation Channel (E)	53.26	79.30

Training Hyperparameters

- Model architecture: Unet #(implementation from the [Segmentation Models Pytorch library](https://segmentation-modelspytorch.readthedocs.io/en/latest/docs/api.html#unet))
- Encoder : Resnet-34 pre-trained with ImageNet
- Augmentation :
  - VerticalFlip(p=0.5)
  - HorizontalFlip(p=0.5)
  - RandomRotate90(p=0.5)
- Input normalization (mean=0 | std=1):
  - norm_means: [105.08, 110.87, 101.82, 106.38, 53.26]
  - norm_stds: [52.17, 45.38, 44, 39.69, 79.3]
- Seed: 2022
- Batch size: 10
- Number of epochs : 200
- Early stopping : patience 30 and val_loss as monitor criterium 
- Optimizer : SGD
- Schaeduler : mode = "min", factor = 0.5, patience = 10, cooldown = 4, min_lr = 1e-7
- Learning rate : 0.02
- Class Weights : [1-building: 1.0 , 2-pervious surface: 1.0 , 3-impervious surface: 1.0 , 4-bare soil: 1.0 , 5-water: 1.0 , 6-coniferous: 1.0 , 7-deciduous: 1.0 , 8-brushwood: 1.0 , 9-vineyard: 1.0 , 10-herbaceous vegetation: 1.0 , 11-agricultural land: 1.0 , 12-plowed land: 1.0 , 13-swimming_pool: 1.0 , 14-snow: 1.0 , 15-clear cut: 0.0 , 16-mixed: 0.0 , 17-ligneous: 0.0 , 18-greenhouse: 1.0 , 19-other: 0.0]

Speeds, Sizes, Times

The FLAIR-INC_rgbie_12cl_resnet34-unet model was trained on a HPC/AI resources provided by GENCI-IDRIS (Grant 2022-A0131013803). 16 V100 GPUs were used ( 4 nodes, 4 GPUS per node). With this configuration the approximate learning time is 6 minutes per epoch.

FLAIR-INC_rgbie_12cl_resnet34-unet was obtained for num_epoch=65 with corresponding val_loss=0.55.

TRAIN loss

VALIDATION loss

Evaluation

Testing Data, Factors & Metrics

Testing Data

The evaluation was performed on a TEST set of 31 750 patches that are independant from the TRAIN and VALIDATION patches. They represent 15 spatio-temporal domains. The TEST set corresponds to the reunion of the TEST set of scientific challenges FLAIR#1 and FLAIR#2. See the FLAIR challenge page for more details.

The choice of a separate TEST set instead of cross validation was made to be coherent with the FLAIR challenges.

Metrics

With the evaluation protocol, the FLAIR-INC_rgbie_12cl_resnet34-unet have been evaluated to OA=76.509% and mIoU=62.716%.

The following table give the class-wise metrics :

Classes	IoU (%)	Fscore (%)	Precision (%)	Recall (%)
building	82.564	90.449	90.412	90.486
pervious_surface	54.149	70.255	70.723	69.794
impervious_surface	73.807	84.930	85.848	84.032
bare_soil	59.013	74.224	78.795	70.154
water	87.216	93.172	91.692	94.699
coniferous	61.591	76.231	79.304	73.387
deciduous	72.225	83.873	81.728	86.133
brushwood	31.187	47.546	57.056	40.754
vineyard	76.105	86.432	84.264	88.714
herbaceous	51.340	67.847	70.134	65.705
agricultural_land	57.558	73.063	67.249	79.978
plowed_land	45.840	62.863	61.094	64.737
average	62.716	75.907	76.525	75.714

The following illustration gives the resulting confusion matrix :

Top : normalised acording to columns, columns sum at 100% and the precision is on the diagonal of the matrix
Bottom : normalised acording to rows, rows sum at 100% and the recall is on the diagonal of the matrix

Normalized Confusion Matrix (precision)

Normalized Confusion Matrix (recall)

Results

Samples of results

Citation

BibTeX:

@inproceedings{ign-flair,
      title={FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery}, 
      author={Anatol Garioud and Nicolas Gonthier and Loic Landrieu and Apolline De Wit and Marion Valette and Marc Poupée and Sébastien Giordano and Boris Wattrelos},
      year={2023},
      booktitle={Advances in Neural Information Processing Systems (NeurIPS) 2023},
      doi={https://doi.org/10.48550/arXiv.2310.13336},
}

APA:

Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sébastien Giordano and Boris Wattrelos. 2023. 
FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery. (2023).
In proceedings of Advances in Neural Information Processing Systems (NeurIPS) 2023.
DOI: https://doi.org/10.48550/arXiv.2310.13336

IGNF
/

FLAIR-INC_rgbie_12cl_resnet34-unet