GEO-Bench 2

Overview

GEO-Bench-2 is a framework for robust evaluation of Geospatial Foundation Models (GeoFMs) which expands on the work of GEO-Bench. It has been carefully curated for evaluation of state-of-the-art model features such as such as multi-modality and multi-temporality. This library aims to facilitate extensive benchmarking of GeoFMs on the GEO-Bench-2 datasets, including features such band re-ordering, changing normalizations, and more.

For details on the GEO-Bench-2 methodology, please see the paper.

The Geo-Bench-2-Leaderbaord tracks the performance of state-of-the-art models on GeoBench-2 datasets. It further acts as a public repository of model performance. We strongly encourage users of the this library to submit experimental results to the leaderboard.

Installation

For a stable release, install with:

pip install GeoBenchV2

For the most recent version of the main branch, install with:

pip install git+https://github.com/The-AI-Alliance/GEO-Bench-2.git

To use the package as a developer, install in editable mode with:

git clone https://github.com/The-AI-Alliance/GEO-Bench-2.git
cd GEO-Bench-2
pip install -e .

Documentation

The latest documentation can be found at this link.

Downloading the Benchmark

Datasets can be downloaded using the geobench-download command-line tool, which is installed automatically with the package.

First, make sure you have installed the GEO-Bench-2 package:

pip install geobenchv2

To download all datasets to a specified directory:

geobench-download --root /path/to/data

To download only selected datasets:

geobench-download --root /path/to/data spacenet7 caffe dynamic_earthnet

The following datasets are available for download:

• benv2, biomassters, burn_scars, caffe, cloudsen12
• dynamic_earthnet, everwatch, flair2, forestnet, fotw
• kuro_siwo, pastis, spacenet2, spacenet7, substation
• treesatai, wind_turbine, so2sat

Note: The --root argument is required to ensure you explicitly choose where to store the datasets, as they can be quite large.

Code License

This code is licensed under the Apache License 2.0. By contributing to this repository, you agree that your contributions will be licensed under the Apache 2.0 License unless otherwise stated.

Dataset Licenses

All dataset are distributed under open-licenses. For license details please see the respective Huggingface repository of each dataset. A summary of the license files can be found in this file. License information is also contained in each README.md file that will be downloaded with the data from Huggingface. For our disclaimer about the licenses and our takedown policy, please see disclaimer file.

How was the Benchmark Generated?

All scripts that were used to generate the GEO-Bench-2 dataset versions are included in the generate_benchmark directory. These scripts are included for transparency purposes, but leaderboard submissions are solely accepted based on official dataset versions on HuggingFace.

Credits

This project was developed as part of the AI Alliance with involvement from IBM and ServiceNow.

Citation

If you are using GEO-Bench-2 in your work, please cite the paper.