LILocBench Challenge Evaluation Tool

Codabench Challenge | Dataset Website | Paper

LILocBench is a long-term indoor localization dataset that comes with a CodaBench challenge for an automated evaluation of localization algorithms. This repo contains the script that we use for evaluating localization algorithms in the challenge.

Setup & Quick Start

To directly run the evaluation script, we provide an example submission of localizer poses on the part of our dataset with public ground truth.

To run the script and download the data do the following steps:

1. Clone the repository:

   git clone https://github.com/PRBonn/LILocBench
   cd LILocBench
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Download and unzip the example submission and public ground truth pose files to example_data/:

   python3 src/download_example_data.py example_data/
   

4. Run the evaluation script:

   python3 src/evaluate.py  \
   --config-file config/config_public.yaml \
   --gt-poses-dir example_data/ref_data_public/ \
   --localizer-dir example_data/amcl_example_submission_public/ \
   --output-dir output/
   
   

This generates a folder output/ in which the following results are stored:

output/
├── dynamic/                # figures with results for dynamic sequences
├── static/                 # figures with results for static sequences
├── lt_changes/             # figures with results for sequences with long-term changes
├── lt_changes_dynamics/    # figures with results for sequences with long-term changes + dynamics
├── *.svg                   # figures with results over all sequences
├── detailed_results.html   # detailed results page also provided in CodaBench with detailed scores for all sequences and the generated figures
└── scores.json             # the overall score of the evaluated localization algorithm

Generating & Evaluating Your own Results + CodaBench Participation

Here we provide instructions how to generate and evaluate results for your own localization algorithm. For the part of our dataset with public ground truth, you can directly evaluate your results locally using the provided evaluation script. We use the non-public part of our dataset for the CodaBench challenge to evaluate submitted localizer results, where we run the same evaluation script with a hidden set of ground truth poses.

To generate your own results, do the following steps:

1. Download the data from our dataset website. We provide full ROS bags with all sensors, a reduced versions without any camera images and individual human-readable files.
2. Generate an initial map based on the provided mapping run and the given ground truth poses. The map's coordinate frame must align with the reference frame with our ground truth for the evaluation, see the format section. Alternatively, we also provide a 2D occupancy grid map based on the mapping run on our dataset website.
3. Run your localization algorithm on the data. For each provided sequence, we evaluate multiple runs with varying starting points, where each run lasts from the starting point until the end of the sequence. The initial time stamps are defined in the config files:
   • in config_public.yaml for the public part of the dataset
   • in config.yaml for the non-public part

Format of the results

The evaluation script expects the pose files of the localizer results to be in the following format:

localizer_results/
    ├── description.json        # JSON file with information about the localization algorithm
    ├── <sequence_name_1>/      # for public part: (dynamics_0, ..., static_0) / for non-public part: (dynamics_1, dynamics_2, ..., static_6)            
        ├── run_1.txt           # files with poses estimated by the localizer 
        ├── run_2.txt
        └── run_3.txt
    ├── <sequence_name_2>/      
        ├── run_1.txt           
        ├── run_2.txt
        └── run_3.txt
    ...

The description.json should provide key information about the submitted localization algorithm and should contain the following fields in JSON format:

{
    "name": "<your algorithm's name>",
    "pdf_url": "<url to a pdf of your paper, if available>",
    "code_url": "<url to a code repository, if available>",
    "short_description": "<a short description of the fundamental principle of your algorithm>"
}

You may leave some of the fields empty, by simply putting an empty string "".

Each of the pose files for the individual runs in the folders of each sequence must follow the TUM-format, i.e. each line in the file should follow the format timestamp tx ty tz qx qy qz qw, where timestamp is a float with the number of seconds since the Unix epoch, tx ty tz are three floats for the position of the robot base_link, and qx qy qz qw are four floats that represent the robot base_link's orientation as a unit quaternion. For more information on the robot frames, we refer to our dataset website. For reference, check the format of the provided sample submission for the public part of our dataset here.

Note, that we do not align the pose trajectory of the localization algorithm to the ground truth, but assume that it is already in the right coordinate frame. This is automatically achieved by generating a map from the mapping run with the provided ground truth poses, which is then used to run the localizer.

Local evaluation on data with public ground truth

You can locally evaluate your results on the public part by first following the Setup & Quickstart section to install the dependencies and get the public ground truth poses, and then run the evaluation script:

python3 src/evaluate.py  \
--config-file config/config_public.yaml \
--gt-poses-dir example_data/ref_data_public/ \
--localizer-dir <path/to/your/results> \
--output-dir output/

CodaBench challenge participation for data with non-public ground truth

For the CodaBench challenge, we evaluate algorithms on the part of our dataset with non-public ground. To participate and evaluate your results, create a .zip-file of your results, which again have to follow the format specified above. More details about the submission can be found on the challenge website on the Participate page.

Citation

If you use our dataset for your research, please cite the following paper:

@inproceedings{trekel2025iros,
	author = {N. Trekel and T. Guadagnino and T. L\"abe and L. Wiesmann and P. Aguiar and J. Behley and C. Stachniss},
	title = {{Benchmark for Evaluating Long-Term Localization in Indoor Environments Under Substantial Static and Dynamic Scene Changes}},
	booktitle = {Proc.~of the IEEE/RSJ Intl.~Conf.~on Intelligent Robots and Systems (IROS)},
	year = {2025},
}