File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change 44Each task is grounded in a separate subprocess so that a per-task timeout can
55be enforced (grounding a large task can take a long time or a lot of memory).
66The results are written as JSON so that two revisions can be compared with
7- ``compare_benchmarks .py``.
7+ ``compare-benchmarks .py``.
88
99The ``--src`` option selects which pyperplan source tree to benchmark; it is put
1010on ``PYTHONPATH`` for the worker subprocesses, so the script itself can live
1111outside the revision under test.
1212
1313Examples:
14- dev/benchmark_grounding .py --out grounding-new.json
15- dev/benchmark_grounding .py --src /tmp/pyperplan-base --out grounding-base.json
14+ dev/benchmark-grounding .py --out grounding-new.json
15+ dev/benchmark-grounding .py --src /tmp/pyperplan-base --out grounding-base.json
1616"""
1717
1818import argparse
Original file line number Diff line number Diff line change 44For each configuration and task, ``python -m pyperplan`` is invoked as a
55subprocess with a per-run timeout, and the planner's reported "Overall time" is
66recorded. The results are written as JSON so that two revisions can be compared
7- with ``compare_benchmarks .py``.
7+ with ``compare-benchmarks .py``.
88
99The configurations below cover every search algorithm and every heuristic at
1010least once. By default the first task of each benchmark domain is used; pass
1414on ``PYTHONPATH`` for the subprocesses.
1515
1616Examples:
17- dev/benchmark_search .py --out search-new.json
18- dev/benchmark_search .py --src /tmp/pyperplan-base --out search-base.json
17+ dev/benchmark-search .py --out search-new.json
18+ dev/benchmark-search .py --src /tmp/pyperplan-base --out search-base.json
1919"""
2020
2121import argparse
Original file line number Diff line number Diff line change 11#! /usr/bin/env python3
22"""Compare two benchmark result files and plot relative runtimes.
33
4- Reads two JSON files produced by ``benchmark_grounding .py`` or
5- ``benchmark_search .py`` (an "old" and a "new" revision) and, for the tasks that
4+ Reads two JSON files produced by ``benchmark-grounding .py`` or
5+ ``benchmark-search .py`` (an "old" and a "new" revision) and, for the tasks that
66both revisions completed successfully, prints a runtime summary and writes a
77log-log scatter plot of old vs. new runtime. Points below the diagonal were
88faster on the new revision.
99
1010Examples:
11- dev/compare_benchmarks .py grounding-base.json grounding-new.json \
11+ dev/compare-benchmarks .py grounding-base.json grounding-new.json \
1212 --out grounding.png --title "Grounding time"
13- dev/compare_benchmarks .py search-base.json search-new.json \
13+ dev/compare-benchmarks .py search-base.json search-new.json \
1414 --out search.png --title "Search time" --success solved
1515"""
1616
You can’t perform that action at this time.
0 commit comments