|
| 1 | +# QQA4CO benchmark datasets |
| 2 | + |
| 3 | +This directory is the **single home for benchmark instance data** consumed |
| 4 | +by the `qqa.datasets.*` loaders, the CLI (`qqa solve --graph-file ...`), |
| 5 | +and the GUI. Each subdirectory is one *benchmark family* with its own |
| 6 | +README, optional download/conversion scripts, and a small bundled smoke |
| 7 | +sample (when feasible). |
| 8 | + |
| 9 | +``` |
| 10 | +data/ |
| 11 | +├── README.md ← this file: layout + how to add a benchmark |
| 12 | +├── gset/ ← Stanford G-set MaxCut (G1, G2, ..., G81) |
| 13 | +├── mis/ ← bundled small/large ER MIS instances |
| 14 | +├── fig/ ← static figures used by docs (NOT instance data) |
| 15 | +└── discs/ ← unified DISCS NeurIPS-2023 CO suite |
| 16 | + (gitignored except .gitignore + README.md) |
| 17 | +``` |
| 18 | + |
| 19 | +## What lives here vs what is git-ignored |
| 20 | + |
| 21 | +* **Tracked**: tiny demo datasets (≤ a few MB, like `mis/er-small/`), |
| 22 | + per-family `README.md`, the `.gitignore` that forbids large blobs, and |
| 23 | + any conversion script that the contributor wrote by hand. |
| 24 | +* **Git-ignored**: bulk data downloaded or generated by setup scripts |
| 25 | + (e.g. all of `discs/`, see `discs/.gitignore`). The pattern is |
| 26 | + |
| 27 | + ```gitignore |
| 28 | + * |
| 29 | + !.gitignore |
| 30 | + !README.md |
| 31 | + ``` |
| 32 | + |
| 33 | + which keeps the directory present (so loaders can refer to it without |
| 34 | + surprising contributors) while excluding every payload file. |
| 35 | + |
| 36 | +## Resolving the data root at run time |
| 37 | + |
| 38 | +`qqa.datasets._default_data_dir()` resolves the root in this order: |
| 39 | + |
| 40 | +1. ``$QQA_DATA_DIR`` environment variable (highest priority). |
| 41 | +2. ``<repo_root>/data`` when running from a source checkout. |
| 42 | +3. ``./data`` next to the working directory (PyPI wheel install). |
| 43 | + |
| 44 | +So you can keep huge datasets on a scratch filesystem and just point |
| 45 | +``QQA_DATA_DIR`` at it. |
| 46 | + |
| 47 | +--- |
| 48 | + |
| 49 | +## Adding a new benchmark family — checklist |
| 50 | + |
| 51 | +Suppose you want to add a `tsp` benchmark (Travelling Salesman). Follow |
| 52 | +the *exact* same pattern as `discs/`: |
| 53 | + |
| 54 | +### 1. Create the folder + `.gitignore` + `README.md` |
| 55 | + |
| 56 | +```bash |
| 57 | +mkdir -p data/tsp |
| 58 | +cat > data/tsp/.gitignore <<'EOF' |
| 59 | +* |
| 60 | +!.gitignore |
| 61 | +!README.md |
| 62 | +EOF |
| 63 | +$EDITOR data/tsp/README.md |
| 64 | +``` |
| 65 | + |
| 66 | +The README **must** document: |
| 67 | + |
| 68 | +* the upstream source (paper, URL, license), |
| 69 | +* how to download / regenerate the raw data, |
| 70 | +* the on-disk layout produced by your converter, |
| 71 | +* the `manifest.jsonl` schema, |
| 72 | +* citation snippet (BibTeX) for the original authors. |
| 73 | + |
| 74 | +Use [`data/discs/README.md`](discs/README.md) as the canonical template. |
| 75 | + |
| 76 | +### 2. Standardise on `*.gpickle + manifest.jsonl` |
| 77 | + |
| 78 | +QQA4CO has converged on one on-disk format for *all* graph problems: |
| 79 | + |
| 80 | +``` |
| 81 | +data/<family>/<problem>/<graph_type>/<subset>/ |
| 82 | + 0001.gpickle # pickle.dump(networkx.Graph) |
| 83 | + 0002.gpickle |
| 84 | + ... |
| 85 | + manifest.jsonl # one JSON object per line |
| 86 | +``` |
| 87 | + |
| 88 | +Each `manifest.jsonl` line is: |
| 89 | + |
| 90 | +```json |
| 91 | +{ |
| 92 | + "id": "tsp-rue-100-0001", |
| 93 | + "file": "0001.gpickle", |
| 94 | + "problem": "tsp", |
| 95 | + "graph_type": "rue", |
| 96 | + "subset": "100", |
| 97 | + "num_nodes": 100, |
| 98 | + "num_edges": 4950, |
| 99 | + "best_known": 2756.4, // null if unknown |
| 100 | + "source": "tsplib/eil101.tsp" |
| 101 | +} |
| 102 | +``` |
| 103 | + |
| 104 | +Why this format? |
| 105 | + |
| 106 | +* **One reader**: `qqa.datasets._load_graphs_from_manifest` already |
| 107 | + consumes it; new families plug in for free. |
| 108 | +* **Streamable**: each line of `manifest.jsonl` is independent → |
| 109 | + `--limit N` smoke runs read the first N lines only. |
| 110 | +* **Reproducible**: `source` keeps a paper trail back to the original |
| 111 | + artefact; converters become trivially auditable. |
| 112 | + |
| 113 | +### 3. Write a converter (`scripts/convert_<family>_to_qqa.py`) |
| 114 | + |
| 115 | +Copy `scripts/convert_discs_to_qqa.py` as a starting point. The required |
| 116 | +helpers (`_normalize_graph`, `_Subset.open/emit/close`) can be *copied |
| 117 | +verbatim* — they encode the contract above. Most converters are 100-300 |
| 118 | +lines; one function per source file shape. |
| 119 | + |
| 120 | +**Idempotency rule**: re-running the converter on an existing destination |
| 121 | +must overwrite each instance with byte-identical content (we depend on |
| 122 | +this for CI smoke tests). The provided `_Subset` helper already does this. |
| 123 | + |
| 124 | +### 4. Add a setup wrapper (`scripts/setup_<family>_data.sh`) |
| 125 | + |
| 126 | +Should support at minimum: |
| 127 | + |
| 128 | +* `--source hf | gdrive | local` (or whatever is appropriate), |
| 129 | +* `--limit N` (subset for smoke), |
| 130 | +* `--problem <foo>` and `--subsets a,b,c` (passed through to the |
| 131 | + converter), |
| 132 | +* graceful fallback when the primary source is unreachable. |
| 133 | + |
| 134 | +Again, `scripts/setup_discs_data.sh` is the reference implementation. |
| 135 | + |
| 136 | +### 5. Add Python loader(s) in `src/qqa/datasets.py` |
| 137 | + |
| 138 | +Expose one function per top-level problem under the family. The |
| 139 | +canonical signature mirrors the existing DISCS loaders: |
| 140 | + |
| 141 | +```python |
| 142 | +def <family>_<problem>( |
| 143 | + graph_type: str = "<default>", |
| 144 | + subset: str | None = None, |
| 145 | + *, |
| 146 | + device: str | torch.device = "cpu", |
| 147 | + limit: int | None = None, |
| 148 | + root: str | os.PathLike | None = None, |
| 149 | +) -> DiscsBenchmark: # rename the dataclass if "DISCS" is wrong here |
| 150 | + sdir, records = _resolve_<family>_subset(...) |
| 151 | + graphs, bests, recs = _load_graphs_from_manifest(sdir, records, limit=limit) |
| 152 | + problems = [<QQA problem class>(g, device=device) for g in graphs] |
| 153 | + return DiscsBenchmark(problems, bests, recs, sdir) |
| 154 | +``` |
| 155 | + |
| 156 | +If the benchmark needs a brand-new QQA problem class (e.g. TSP isn't a |
| 157 | +plain `MaxCut`), add it under `src/qqa/problems/` first — see |
| 158 | +[the problem-class checklist below](#adding-a-new-problem-class). |
| 159 | + |
| 160 | +### 6. Wire it into the runner |
| 161 | + |
| 162 | +`scripts/bench_discs.py` is intentionally generic but currently routes by |
| 163 | +hard-coded family map (`_PROBLEM_LOADER`). When you add a family: |
| 164 | + |
| 165 | +* For DISCS-family additions (new problem inside DISCS), extend the |
| 166 | + `_PROBLEM_LOADER` dict and the suite-resolution code. |
| 167 | +* For a *new* family (TSP, FAP, …), copy `scripts/bench_discs.py` to |
| 168 | + `scripts/bench_<family>.py`, swap the loaders/`_objective_from_result`, |
| 169 | + and add a `make bench-<family>{,-setup,-smoke}` triplet in the |
| 170 | + `Makefile`. |
| 171 | + |
| 172 | +### 7. Add a fake-graph smoke test under `tasks/test/` |
| 173 | + |
| 174 | +Mirror `tasks/test/discs_bench_fake_smoke.py`: build a temporary |
| 175 | +`<family>/<problem>/<gtype>/<subset>/` tree with NetworkX toy graphs, |
| 176 | +point `QQA_DATA_DIR` at it, run the bench script, and assert the JSON |
| 177 | +schema. This runs in CI without any network/large-file dependency. |
| 178 | + |
| 179 | +A real-data sister script (`*_real_smoke.py`) — *non-CI*, requires the |
| 180 | +download — should also be checked in, with documentation of how to |
| 181 | +seed the data (see `tasks/test/discs_bench_real_smoke.py`). |
| 182 | + |
| 183 | +### 8. Documentation |
| 184 | + |
| 185 | +* Add a paragraph + one example command in the top-level |
| 186 | + [`README.md`](../README.md) under the existing "DISCS combinatorial- |
| 187 | + optimization benchmark suite" subsection (or a sibling subsection for |
| 188 | + a new family). |
| 189 | +* Append the BibTeX of the original benchmark to the `## Cite` section. |
| 190 | +* If the family pulls a new optional dependency, register it as |
| 191 | + `[<family>]` in `pyproject.toml::project.optional-dependencies` and |
| 192 | + add it to the `all` extra. |
| 193 | + |
| 194 | +--- |
| 195 | + |
| 196 | +## Adding a new problem class |
| 197 | + |
| 198 | +(See also the in-source design notes in `src/qqa/problems/base.py`.) |
| 199 | + |
| 200 | +When a new benchmark exposes an objective that no existing |
| 201 | +`COProblem` subclass covers (DISCS NormCut was a real example): |
| 202 | + |
| 203 | +1. Inherit from `COProblem` (or `QUBOProblem` for a binary QUBO). |
| 204 | +2. Set `self.relaxation` to one of `BinaryRelaxation`, |
| 205 | + `SpinRelaxation`, `CategoricalRelaxation`, or your own |
| 206 | + `Relaxation`-protocol object in `__init__`. |
| 207 | +3. Implement `loss_fn(x)` returning shape `(B,)` or `(B, I)`. **Keep the |
| 208 | + sign convention "lower is better"** — for a max-objective problem, |
| 209 | + negate (`loss = -obj`) or add a non-negative penalty. |
| 210 | +4. Provide `score_summary(x_disc) -> dict` with keys |
| 211 | + `label, value, unit, feasible, extra`. The benchmark runner uses |
| 212 | + `value` for the human-readable objective and `feasible` to flag |
| 213 | + constraint violations — *do not skip this*, otherwise a penalised |
| 214 | + QUBO will overstate its objective whenever it fails to satisfy a |
| 215 | + constraint (this happened to MIS/MaxClique in early DISCS bench |
| 216 | + iterations). |
| 217 | +5. Define the right size attribute for your relaxation: |
| 218 | + * Binary / Spin → `num_nodes` (or `num_spins` / `num_vars`). |
| 219 | + * Categorical → `num_node` (singular!) **and** `num_category >= 2`. |
| 220 | +6. If you want SA / PA support, the problem must be **single-instance |
| 221 | + binary or spin** with no `CategoricalRelaxation`. Optionally expose |
| 222 | + `Q_mat` ((N, N) torch tensor) for the Glauber + greedy 1-flip fast |
| 223 | + path and the `polish` step in `qqa.anneal`. |
| 224 | +7. `device`: load all per-problem tensors onto the requested device in |
| 225 | + `__init__`. Solvers will then route everything through that device |
| 226 | + without surprises. |
| 227 | +8. Register in `src/qqa/problems/__init__.py` (`__all__` + import) and |
| 228 | + `src/qqa/__init__.py` (re-export, only if it is part of the public |
| 229 | + API). |
| 230 | +9. Add a unit test under `tests/test_<problem>.py` covering at least |
| 231 | + (a) `loss_fn` on a hand-checked tiny case, (b) `score_summary` |
| 232 | + feasibility detection, and (c) a `qqa.anneal` smoke that asserts |
| 233 | + `result.best_obj` is finite. Mirror `tests/test_normcut.py`. |
| 234 | + |
| 235 | +--- |
| 236 | + |
| 237 | +## Reference layouts (existing families) |
| 238 | + |
| 239 | +| Family | Problems | Source | Layout | |
| 240 | +|---------|-------------------------------------------|-----------------------------------------|---------------------------------------------| |
| 241 | +| `gset` | MaxCut | Stanford G-set | one `<name>` text file per instance | |
| 242 | +| `mis` | Maximum Independent Set | bundled (ER-small, ER-large) | per-instance directories with `meta.json` | |
| 243 | +| `discs` | MaxCut, MIS, MaxClique, NormCut | Goshvadi et al., NeurIPS 2023 | unified `gpickle + manifest.jsonl` (above) | |
| 244 | + |
| 245 | +When in doubt, **mimic `discs/` exactly** — it is the most general layout |
| 246 | +and the loader code is already battle-tested. |
0 commit comments