Model: PhilEO Bench

A novel evaluation framework for EO Foundation Models.

Model Details

Model Description

The PhilEO Bench evaluation framework comprises of a testbed that can be used to test any EO Foundation Model. The three downstream tasks are building density estimation, road segmentation, and land cover classification.

Developed by: ESA, Phi-lab
Model type: Evaluation Framework
License: MIT

The aim of Foundation Models is to improve the performance on several diverse downstream tasks. However, these models are often evaluated on a range of datasets with different characteristics (size, resolution, locations, satellite sources, and capture dates). There is also a focus on evaluating classification downstream tasks, while omitting image-to-image downstream tasks (such as segmentation). Therefore, it is challenging to fairly compare the performance of these burgeoning EO FMs and draw meaningful conclusions. To evaluate FMs, we propose the PhilEO Bench, an evaluation framework with the aim of providing a flexible, consistent, and fair benchmark for EO Sentinel-2 FMs.

Uses

The PhilEO Bench is used to evaluate EO Foundation Models. - We introduce a new flexible evaluation framework focused on generating comparable, fair, and reproducible results.

Model Sources

The basic links for the model are:

Paper: https://arxiv.org/pdf/2401.04464.pdf
Code: http://github.com/ESA-PhiLab/PhilEO-Bench
Project Website: http://phileo-bench.github.io
Repository: http://huggingface.co/ESA-philab/PhilEO-Bench
arXiv: https://arxiv.org/abs/2401.04464
Pre-trained models: http://huggingface.co/ESA-philab/PhilEO-Bench/tree/main/pretrained_philab_models
Data in: http://huggingface.co/datasets/ESA-philab/PhilEO-downstream

Citation

Casper Fibaek, Luke Camilleri, Andreas Luyts, Nikolaos Dionelis, and Bertrand Le Saux, “PhilEO Bench: Evaluating Geo-Spatial Foundation Models,” arXiv:2401.04464, 2024.