Add support for evaluating pre-computed predictions

.gitignore

@@ -13,4 +13,6 @@ docs/py_docs/build/.buildinfo*
 local/
 third_party/
 
-.DS_Store
+.DS_Store
+
+*.pyc

README.md

@@ -107,7 +107,7 @@ poetry install
 ```
 
 
-Activate the environment: 
+Activate the environment:
 
 Depending on your Poetry version, use one of the following (you can leave the environment by running deactivate or exit):
 
@@ -177,17 +177,17 @@ For detailed GUI documentation, see our [GUI guide](https://jderobot.github.io/P
 You can check the `examples` directory for further inspiration. If you are using *poetry*, you can run the scripts provided either by activating the created environment using `poetry shell` or directly running `poetry run python examples/<some_python_script.py>`.
 
 ## Command-line interface
-PerceptionMetrics provides a CLI with two commands, `pm_evaluate` and `pm_batch`. Thanks to the configuration in the `pyproject.toml` file, we can simply run `poetry install` from the root directory and use them without explicitly invoking the Python files. More details are provided in [PerceptionMetrics website](https://jderobot.github.io/PerceptionMetrics/usage/#command-line-interface).
+PerceptionMetrics provides a CLI with several commands (e.g. `pm_eval_model` and `pm_batch`). Thanks to the configuration in the `pyproject.toml` file, we can simply run `poetry install` from the root directory and use them without explicitly invoking the Python files. More details are provided in [PerceptionMetrics website](https://jderobot.github.io/PerceptionMetrics/usage/#command-line-interface).
 
 ### Example Usage
 **Segmentation:**
 ```bash
-pm_evaluate segmentation image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format rellis3d --dataset_dir /path/to/dataset --dataset_ontology /path/to/ontology.json --out_fname /path/to/results.csv
+pm_eval_model segmentation image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format rellis3d --dataset_dir /path/to/dataset --dataset_ontology /path/to/ontology.json --out_fname /path/to/results.csv
 ```
 
 **Detection:**
 ```bash
-pm_evaluate detection image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format coco --dataset_dir /path/to/coco/dataset --out_fname /path/to/results.csv
+pm_eval_model detection image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format coco --dataset_dir /path/to/coco/dataset --out_fname /path/to/results.csv
 ```
 
 <h1 id="DetectionMetrics">DetectionMetrics</h1>

additional_envs/pyproject-lsk3dnet.toml

@@ -38,8 +38,10 @@ sphinx-rtd-theme = "^3.0.2"
 pytest = "^8.0.0"
 
 [tool.poetry.scripts]
-pm_evaluate = "perceptionmetrics.cli.evaluate:evaluate"
+pm_eval_model = "perceptionmetrics.cli.eval_model:eval_model"
+pm_eval_preds = "perceptionmetrics.cli.eval_preds:eval_preds"
 pm_batch = "perceptionmetrics.cli.batch:batch"
+pm_computational_cost = "perceptionmetrics.cli.computational_cost:computational_cost"
 
 [build-system]
 requires = ["poetry-core"]

additional_envs/pyproject-sphereformer.toml

@@ -38,8 +38,10 @@ sphinx-rtd-theme = "^1.0.0"
 pytest = "^6.2.5"
 
 [tool.poetry.scripts]
-pm_evaluate = "perceptionmetrics.cli.evaluate:evaluate"
+pm_eval_model = "perceptionmetrics.cli.eval_model:eval_model"
+pm_eval_preds = "perceptionmetrics.cli.eval_preds:eval_preds"
 pm_batch = "perceptionmetrics.cli.batch:batch"
+pm_computational_cost = "perceptionmetrics.cli.computational_cost:computational_cost"
 
 [build-system]
 requires = ["poetry-core"]

docs/_pages/usage.md

@@ -45,24 +45,24 @@ You can check the [`examples` directory](https://github.com/JdeRobot/PerceptionM
 #### [Full docs for the Python library](https://jderobot.github.io/PerceptionMetrics/py_docs/build/html/index.html)
 
 ## Command-line interface
-PerceptionMetrics currently provides a CLI with two commands, `pm_evaluate` and `pm_batch`. Thanks to the configuration in the `pyproject.toml` file, we can simply run `poetry install` from the root directory and use them without explicitly invoking the Python files.
+PerceptionMetrics currently provides a CLI with three commands, `pm_eval_model`, `pm_eval_preds` and `pm_batch`. Thanks to the configuration in the `pyproject.toml` file, we can simply run `poetry install` from the root directory and use them without explicitly invoking the Python files.
 
-#### `pm_evaluate`
+#### `pm_eval_model`
 Run a single evaluation job given a model and dataset configurations.
 
 **Segmentation Example:**
 ```shell
-pm_evaluate segmentation image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format rellis3d --dataset_dir /path/to/dataset  --dataset_ontology /path/to/ontology.json --out_fname /path/to/results.csv
+pm_eval_model segmentation image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format rellis3d --dataset_dir /path/to/dataset  --dataset_ontology /path/to/ontology.json --out_fname /path/to/results.csv
 ```
 
 **Detection Example:**
 ```shell
-pm_evaluate detection image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format coco --dataset_dir /path/to/coco/dataset --out_fname /path/to/results.csv
+pm_eval_model detection image --model_format torch --model /path/to/model.pt --model_ontology /path/to/ontology.json --model_cfg /path/to/cfg.json --dataset_format coco --dataset_dir /path/to/coco/dataset --out_fname /path/to/results.csv
 ```
 
 Docs:
 ```shell
-Usage: pm_evaluate [OPTIONS] {segmentation|detection} {image|lidar}
+Usage: pm_eval_model [OPTIONS] {segmentation|detection} {image|lidar}
 
   Evaluate model on dataset
 
@@ -113,6 +113,67 @@ Options:
   --help                          Show this message and exit.
 ```
 
+#### `pm_eval_preds`
+Evaluate pre-computed predictions stored on disk against a ground truth dataset.
+
+**Example:**
+```shell
+pm_eval_preds segmentation image --dataset_format gaia --dataset_fname /path/to/dataset.parquet --predictions_dir /path/to/predictions --split test --out_fname /path/to/results.csv
+```
+
+Docs:
+```shell
+Usage: pm_eval_preds [OPTIONS] {segmentation|detection} {image|lidar}
+
+  Evaluate pre-computed predictions stored on disk against a GT dataset
+
+Options:
+  --predictions_dir DIRECTORY     Root directory containing prediction files,
+                                  organized in the same split/filename
+                                  structure as the GT dataset  [required]
+  --dataset_format [gaia|rellis3d|goose|generic|rugd|coco]
+                                  Dataset format  [default: gaia]
+  --dataset_fname FILE            Parquet dataset file
+  --dataset_dir DIRECTORY         Dataset directory (used for 'Rellis3D',
+                                  'Wildscenes', and 'COCO' formats)
+  --split_dir DIRECTORY           Directory containing .lst or .csv split
+                                  files (used for 'Rellis3D' and 'Wildscenes'
+                                  formats, respectively)
+  --train_dataset_dir DIRECTORY   Train dataset directory (used for 'GOOSE'
+                                  and 'Generic' formats)
+  --val_dataset_dir DIRECTORY     Validation dataset directory (used for
+                                  'GOOSE' and 'Generic' formats)
+  --test_dataset_dir DIRECTORY    Test dataset directory (used for 'GOOSE' and
+                                  'Generic' formats)
+  --images_dir TEXT               Directory containing data (used for 'RUGD'
+                                  format)
+  --labels_dir TEXT               Directory containing annotations (used for
+                                  'RUGD' format)
+  --data_suffix TEXT              Data suffix to be used to filter data (used
+                                  for 'Generic' format)
+  --label_suffix TEXT             Label suffix to be used to filter labels
+                                  (used for 'Generic' format)
+  --dataset_ontology FILE         JSON containing dataset ontology (used for
+                                  'Generic' and 'Rellis3D' formats)
+  --split TEXT                    Name of the split or splits separated by
+                                  commas to be evaluated  [default: test]
+  --pred_ontology FILE            JSON file containing the prediction ontology
+                                  (only needed when it differs from the
+                                  dataset ontology)
+  --ontology_translation FILE     JSON file containing translation between
+                                  dataset and prediction ontologies
+  --translation_direction [dataset_to_model|model_to_dataset]
+                                  Direction of the ontology translation
+                                  [default: dataset_to_model]
+  --ignored_classes TEXT          Class name(s) to ignore during evaluation
+                                  (repeat for multiple)
+  --out_fname PATH                CSV file where the evaluation results will
+                                  be stored  [required]
+  --results_per_sample            Store per-sample CSV results next to each
+                                  prediction file
+  --help                          Show this message and exit.
+```
+
 #### `pm_batch`
 Execute requested jobs sequentially. It must be configured by means of a YAML file.
 

examples/eval_preds.py

@@ -0,0 +1,74 @@
+import argparse
+import os
+
+from perceptionmetrics.datasets.gaia import GaiaImageSegmentationDataset
+
+
+def parse_args() -> argparse.Namespace:
+    """Parse user input arguments.
+
+    :return: Parsed arguments
+    :rtype: argparse.Namespace
+    """
+    parser = argparse.ArgumentParser(
+        description="Evaluate pre-computed prediction labels against a GT dataset."
+    )
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        required=True,
+        help="Parquet dataset file (GT)",
+    )
+    parser.add_argument(
+        "--predictions_dir",
+        type=str,
+        required=True,
+        help="Root directory containing prediction labels, organized in the "
+        "same split/filename structure as the GT dataset",
+    )
+    parser.add_argument(
+        "--split",
+        type=str,
+        default="test",
+        help="Name of the split to evaluate (default: test)",
+    )
+    parser.add_argument(
+        "--out_fname",
+        type=str,
+        required=True,
+        help="CSV file where the evaluation results will be stored",
+    )
+    parser.add_argument(
+        "--ignored_classes",
+        type=str,
+        nargs="+",
+        default=None,
+        help="List of class names to ignore during evaluation",
+    )
+    parser.add_argument(
+        "--results_per_sample",
+        action="store_true",
+        help="Store per-sample results as CSV files next to each prediction",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    """Main function."""
+    args = parse_args()
+
+    dataset = GaiaImageSegmentationDataset(args.dataset)
+    results = dataset.eval_preds(
+        predictions_dir=args.predictions_dir,
+        split=args.split,
+        ignored_classes=args.ignored_classes,
+        results_per_sample=args.results_per_sample,
+    )
+
+    os.makedirs(os.path.dirname(args.out_fname), exist_ok=True)
+    results.to_csv(args.out_fname)
+    print(f"Results saved to {args.out_fname}")
+
+
+if __name__ == "__main__":
+    main()

perceptionmetrics/cli/__init__.py

@@ -1,11 +1,13 @@
 from perceptionmetrics import datasets
 from perceptionmetrics import models
-from perceptionmetrics.cli.evaluate import evaluate
+from perceptionmetrics.cli.eval_model import eval_model
+from perceptionmetrics.cli.eval_preds import eval_preds
 from perceptionmetrics.cli.computational_cost import computational_cost
 from perceptionmetrics.datasets.coco import find_img_dir_and_ann_file
 
 REGISTRY = {
-    "evaluate": evaluate,
+    "eval_model": eval_model,
+    "eval_preds": eval_preds,
     "computational_cost": computational_cost,
 }
 

perceptionmetrics/cli/batch.py

@@ -16,9 +16,14 @@ def batch(command, jobs_cfg):
     """Perform PerceptionMetrics jobs in batch mode"""
     jobs_cfg = uio.read_yaml(jobs_cfg)
 
-    # If a single model has been provided, convert it to a list
-    if not isinstance(jobs_cfg["model"], list):
-        jobs_cfg["model"] = [jobs_cfg["model"]]
+    # If a single model/predictions has been provided, convert it to a list
+    target_key = "model" if "model" in jobs_cfg else "predictions"
+    if not isinstance(jobs_cfg[target_key], list):
+        jobs_cfg[target_key] = [jobs_cfg[target_key]]
+
+    # Alias predictions to model so the rest of the iteration logic works
+    if target_key == "predictions":
+        jobs_cfg["model"] = jobs_cfg.pop("predictions")
 
     # Same for dataset
     has_dataset = "dataset" in jobs_cfg
@@ -107,14 +112,23 @@ def batch(command, jobs_cfg):
             }
 
             model_cfg = job_components[0]
-            params.update(
-                {
-                    "model_format": model_cfg["format"],
-                    "model": model_cfg["path"],
-                    "model_ontology": model_cfg["ontology"],
-                    "model_cfg": model_cfg["cfg"],
-                }
-            )
+            if command == "eval_model" or command == "computational_cost":
+                params.update(
+                    {
+                        "model_format": model_cfg["format"],
+                        "model": model_cfg["path"],
+                        "model_ontology": model_cfg["ontology"],
+                        "model_cfg": model_cfg.get("cfg"),
+                    }
+                )
+            elif command == "eval_preds":
+                params.update(
+                    {
+                        "predictions_dir": model_cfg["path"],
+                    }
+                )
+                if "ontology" in model_cfg:
+                    params["pred_ontology"] = model_cfg["ontology"]
 
             if command == "computational_cost":
                 if jobs_cfg["input_type"] == "image":

perceptionmetrics/cli/evaluate.py → perceptionmetrics/cli/eval_model.py

@@ -16,7 +16,7 @@ def parse_split(ctx, param, value):
     return splits
 
 
-@click.command(name="evaluate", help="Evaluate model on dataset")
+@click.command(name="eval_model", help="Evaluate model on dataset")
 @click.argument("task", type=click.Choice(["segmentation", "detection"], case_sensitive=False))
 @click.argument(
     "input_type", type=click.Choice(["image", "lidar"], case_sensitive=False)
@@ -138,7 +138,7 @@ def parse_split(ctx, param, value):
     help="Directory where predictions (images/points and CSV) per sample will be "
     "stored. If not provided, predictions per sample will not be saved",
 )
-def evaluate(
+def eval_model(
     task,
     input_type,
     model_format,
@@ -163,7 +163,7 @@ def evaluate(
     predictions_outdir,
 ):
     """Evaluate model on dataset"""
-    if isinstance(split, str):  # if evaluate has been called directly
+    if isinstance(split, str):  # if eval_model has been called directly
         split = parse_split(None, None, split)
 
     model = cli.get_model(
@@ -200,4 +200,4 @@ def evaluate(
 
 
 if __name__ == "__main__":
-    evaluate()
+    eval_model()