Skip to content

Commit 23780b9

Browse files
author
Yuma Ichikawa
committed
Wire setup_discs_data.sh to the pre-converted HF Hub dataset
The Hugging Face dataset Yuma-Ichikawsa/discs-co-bench now hosts the already-converted `.gpickle` + `manifest.jsonl` tree (~3.9 GB, 15486 instances across 32 subsets), so users no longer need to download the 6.7 GB raw tarball and run the converter when starting from a clean checkout. Changes: * fix the default HF_REPO_ID to the actual namespace (`Yuma-Ichikawsa/discs-co-bench`). * `fetch_hf` now snapshots straight into `data/discs/` (DST) using `allow_patterns` to fetch only the four problem trees + manifests, and skips `convert_discs_to_qqa.py` entirely. * the gdrive code path is unchanged and still does download + extract + convert for users who explicitly opt out of the Hub. Verified end-to-end with a fresh `snapshot_download` of the mis/satlib subset (500 .gpickle + 1 manifest.jsonl) into a tempdir, which loads correctly via `pickle.load` -> networkx.Graph.
1 parent 9e9b46a commit 23780b9

1 file changed

Lines changed: 15 additions & 6 deletions

File tree

scripts/setup_discs_data.sh

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ SOURCE="hf"
3434
PROBLEM="all"
3535
SUBSETS=""
3636
LIMIT=""
37-
HF_REPO_ID="${DISCS_HF_REPO_ID:-yuma-ichikawa/discs-co-bench}"
37+
HF_REPO_ID="${DISCS_HF_REPO_ID:-Yuma-Ichikawsa/discs-co-bench}"
3838
# The published DISCS Drive folder (1nEppx...) contains exactly two files:
3939
# * 2dtsp.zip (TSP data, NOT used by us)
4040
# * DISCS-DATA.tar.gz ID=1lbpdEqs_rDqaLmS3YkFrbn7iK8z1K1it (~6.7 GB)
@@ -69,15 +69,19 @@ echo "[setup_discs_data] source=${SOURCE} problem=${PROBLEM} subsets=${SUBSETS
6969
# fetch #
7070
# --------------------------------------------------------------------------- #
7171
fetch_hf() {
72-
echo "[setup_discs_data] fetching from Hugging Face Hub: ${HF_REPO_ID}"
72+
# The HF dataset hosts the *converted* `.gpickle` + `manifest.jsonl`
73+
# tree directly (~3.9 GB). We snapshot it straight into ${DST} and
74+
# skip the `convert_discs_to_qqa.py` step entirely.
75+
echo "[setup_discs_data] fetching pre-converted dataset from Hugging Face Hub: ${HF_REPO_ID}"
7376
python - <<PY
7477
from huggingface_hub import snapshot_download
7578
import os
79+
allow = ["maxcut/**", "mis/**", "maxclique/**", "normcut/**", "**/manifest.jsonl", "README.md"]
7680
local = snapshot_download(
7781
repo_id=os.environ["HF_REPO_ID"],
7882
repo_type="dataset",
79-
local_dir=os.environ["RAW"],
80-
local_dir_use_symlinks=False,
83+
local_dir=os.environ["DST"],
84+
allow_patterns=allow,
8185
)
8286
print("[setup_discs_data] HF snapshot at:", local)
8387
PY
@@ -116,8 +120,13 @@ case "${SOURCE}" in
116120
echo "[setup_discs_data] huggingface_hub not installed; run: pip install huggingface_hub" >&2
117121
exit 3
118122
fi
119-
export HF_REPO_ID RAW
120-
if ! fetch_hf; then
123+
export HF_REPO_ID DST
124+
if fetch_hf; then
125+
echo "[setup_discs_data] done. HF snapshot is already in the unified gpickle+manifest format."
126+
echo "[setup_discs_data] Try:"
127+
echo " python scripts/bench_discs.py --suite mis-satlib --backend qqa --instances 3"
128+
exit 0
129+
else
121130
echo "[setup_discs_data] HF download failed, falling back to gdrive..." >&2
122131
SOURCE="gdrive"
123132
fi

0 commit comments

Comments
 (0)