Prepare Datasets for FrozenSeg

A dataset can be used by accessing DatasetCatalog for its data, or MetadataCatalog for its metadata (class names, etc). This document explains how to setup the builtin datasets so they can be used by the above APIs. Use Custom Datasets gives a deeper dive on how to use DatasetCatalog and MetadataCatalog, and how to add new datasets to them.

FrozenSeg has builtin support for a few datasets. The datasets are assumed to exist in a directory specified by the environment variable DETECTRON2_DATASETS. Under this directory, detectron2 will look for datasets in the structure described below, if needed.

$DETECTRON2_DATASETS/
  # panoptic datasets
  ADEChallengeData2016/
  coco/
  cityscapes/
  mapillary_vistas/
  bdd100k/
  # semantic datasets
  VOCdevkit/
  ADE20K_2021_17_01/
  pascal_ctx_d2/
  pascal_voc_d2/

You can set the location for builtin datasets by export DETECTRON2_DATASETS=/path/to/datasets. If left unset, the default is ./datasets relative to your current working directory.

Expected dataset structure for COCO:

coco/
  annotations/
    instances_{train,val}2017.json
    panoptic_{train,val}2017.json
  {train,val}2017/
    # image files that are mentioned in the corresponding json
  panoptic_{train,val}2017/  # png annotations
  panoptic_semseg_{train,val}2017/  # generated by the script mentioned below

Install panopticapi by:

pip install git+https://github.com/cocodataset/panopticapi.git

Then, run python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py, to extract semantic annotations from panoptic annotations (only used for evaluation).

Expected dataset structure for cityscapes:

cityscapes/
  gtFine/
    train/
      aachen/
        color.png, instanceIds.png, labelIds.png, polygons.json,
        labelTrainIds.png
      ...
    val/
    test/
    # below are generated Cityscapes panoptic annotation
    cityscapes_panoptic_train.json
    cityscapes_panoptic_train/
    cityscapes_panoptic_val.json
    cityscapes_panoptic_val/
    cityscapes_panoptic_test.json
    cityscapes_panoptic_test/
  leftImg8bit/
    train/
    val/
    test/

Install cityscapes scripts by:

pip install git+https://github.com/mcordts/cityscapesScripts.git

Note: to create labelTrainIds.png, first prepare the above structure, then run cityscapesescript with:

CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createTrainIdLabelImgs.py

These files are not needed for instance segmentation.

Note: to generate Cityscapes panoptic dataset, run cityscapesescript with:

CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createPanopticImgs.py

These files are not needed for semantic and instance segmentation.

Expected dataset structure for ADE20k (A150):

ADEChallengeData2016/
  images/
  annotations/
  objectInfo150.txt
  # download instance annotation
  annotations_instance/
  # generated by prepare_ade20k_sem_seg.py
  annotations_detectron2/
  # below are generated by prepare_ade20k_pan_seg.py
  ade20k_panoptic_{train,val}.json
  ade20k_panoptic_{train,val}/
  # below are generated by prepare_ade20k_ins_seg.py
  ade20k_instance_{train,val}.json

The directory annotations_detectron2 is generated by running python datasets/prepare_ade20k_sem_seg.py.

Install panopticapi by:

pip install git+https://github.com/cocodataset/panopticapi.git

Download the instance annotation from http://sceneparsing.csail.mit.edu/:

wget http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar

Then, run python datasets/prepare_ade20k_pan_seg.py, to combine semantic and instance annotations for panoptic annotations.

And run python datasets/prepare_ade20k_ins_seg.py, to extract instance annotations in COCO format.

Expected dataset structure for Mapillary Vistas:

mapillary_vistas/
  training/
    images/
    instances/
    labels/
    panoptic/
  validation/
    images/
    instances/
    labels/
    panoptic/

No preprocessing is needed for Mapillary Vistas on semantic and panoptic segmentation.

Expected dataset structure for BDD100K

bdd100k/
  images/
    10k/
      train/
      val/
      test/
  json
  labels/
    pan_seg/
    sem_seg/

coco-format annotations is obtained by running:

cd $DETECTRON2_DATASETS
wget https://github.com/chenxi52/FrozenSeg/releases/download/latest/bdd100k_json.zip
unzip bdd100k_json.zip

Expected dataset structure for ADE20k-Full (A-847):

ADE20K_2021_17_01/
  images/
  index_ade20k.pkl
  objects.txt
  # generated by prepare_ade20k_full_sem_seg.py
  images_detectron2/
  annotations_detectron2/

cd $DETECTRON2_DATASETS
wget your/personal/download/link/{username}_{hash}.zip
unzip {username}_{hash}.zip

Generate the directories ADE20K_2021_17_01/images_detectron2 and ADE20K_2021_17_01/annotations_detectron2 by running:

python datasets/prepare_ade20k_full_sem_seg.py

Expected dataset structure for PASCAL Context Full (PC-459) and PASCAL VOC (PAS-21):

VOCdevkit/
  VOC2012/
    Annotations/
    JPEGImages/
    ImageSets/
      Segmentation/
  VOC2010/
    JPEGImages/
    trainval/
    trainval_merged.json
# generated by prepare_pascal_voc_sem_seg.py
pascal_voc_d2/
  images/
  annotations_pascal21/
  # pascal 20 excludes the background class
  annotations_pascal20/
# generated by prepare_pascal_ctx_sem_seg.py
pascal_ctx_d2/
  images/
  annotations_ctx59/
  # generated by prepare_pascal_ctx_full_sem_seg.py
  annotations_ctx459/

PASCAL VOC (PAS-21)

Download the dataset from http://host.robots.ox.ac.uk/pascal/VOC/:

cd $DETECTRON2_DATASETS
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
# generate folder VOCdevkit/VOC2012
tar -xvf VOCtrainval_11-May-2012.tar

Generate directory pascal_voc_d2 running:

python datasets/prepare_pascal_voc_sem_seg.py

PASCAL Context Full (PC-459)

Download the dataset from http://host.robots.ox.ac.uk/pascal/VOC/ and annotation from https://www.cs.stanford.edu/~roozbeh/pascal-context/:

cd $DETECTRON2_DATASETS
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2010/VOCtrainval_03-May-2010.tar
# generate folder VOCdevkit/VOC2010
tar -xvf VOCtrainval_03-May-2010.tar 
wget https://www.cs.stanford.edu/~roozbeh/pascal-context/trainval.tar.gz
# generate folder VOCdevkit/VOC2010/trainval
tar -xvzf trainval.tar.gz -C VOCdevkit/VOC2010 
wget https://codalabuser.blob.core.windows.net/public/trainval_merged.json -P VOCdevkit/VOC2010/

Install Detail API by:

git clone https://github.com/zhanghang1989/detail-api.git
rm detail-api/PythonAPI/detail/_mask.c
pip install -e detail-api/PythonAPI/

Generate directory pascal_ctx_d2/images running:

python datasets/prepare_pascal_ctx_sem_seg.py

Generate directory pascal_ctx_d2/annotations_ctx459 running:

python datasets/prepare_pascal_ctx_full_sem_seg.py