COMPASS

COMmunity-guided Placement And Shard Selection for distributed vector databases.

COMPASS places each fact on the shard of its knowledge graph community (Louvain) and routes each query to a few relevant shards using community-level summaries, so a distributed vector database can answer scientific queries with selective search instead of broadcasting to every shard.

Requirements

• Python >= 3.12 and the packages in requirements.txt
• A Rust toolchain (cargo) for the gRPC data-plane client
• Qdrant v1.16.1, pulled as an Apptainer image
• A PBS cluster for the distributed experiments; a single node suffices for data preparation and the offline benchmarks

Layout

configs/      scicueval.yaml, smoke.txt, sweeps/e0..e3
scripts/      prepare_scicueval.py, leakage_ablation.py, rag_eval.py, run_overhead.sh, run_leakage_ablation.sh
src/compass/  config, dataset, scicueval_sources, communities, placement, prepare_run,
              cluster, build_index, segments, finalize, util, bench/, analysis/
qdrant/       download_sif.sh, launch_node.sh, client/
hpc/          env.sh, setup.sh, pbs/ (job + sweep submitter)

Quickstart

Paths are configured in hpc/env.sh. Set your allocation in the #PBS -A line of hpc/pbs/job_experiment.pbs. On ALCF systems, set COMPASS_USE_ALCF_PROXY=1 before sourcing hpc/env.sh.

1. Set up

git clone https://github.com/song-oh/compass.git
cd compass
./hpc/setup.sh
qdrant/download_sif.sh v1.16.1

2. Prepare data (downloads the SciCUEval data once, then embeds each KG)

python scripts/prepare_scicueval.py --config configs/scicueval.yaml --download
for DS in PriKG HipKG PhaKG GoKG; do
  python scripts/prepare_scicueval.py --config configs/scicueval.yaml -o data.dataset=$DS
done

3. Offline benchmarks

./scripts/run_overhead.sh
bash scripts/run_leakage_ablation.sh

4. One cluster run (smoke test)

qsub -v RUN_TAG=smoke,OVERRIDE_FILE=configs/smoke.txt hpc/pbs/job_experiment.pbs

5. Full sweeps: retrieval quality (e0), placement frontier (e1), index/worker shape (e2), node scaling (e3)

nohup ./hpc/pbs/submit_sweep.sh \
    configs/sweeps/e0_main.yaml configs/sweeps/e1_balance.yaml \
    configs/sweeps/e2_shape.yaml configs/sweeps/e3_scale.yaml > sweep.log 2>&1 &

6. Aggregate

PYTHONPATH=src python -m compass.analysis.aggregate --runs-dir runs \
    --out results.csv --stats results_stats.csv
PYTHONPATH=src python -m compass.analysis.balance --runs-dir runs \
    --glob '*-pri-*-s*' --frontier --workload factret --recall-floor 0.8

7. Downstream RAG accuracy

python scripts/rag_eval.py --config configs/scicueval.yaml --runs-dir runs \
    --glob '*' --plans all --backend hf --model Qwen/Qwen2.5-7B-Instruct \
    --out runs/rag_accuracy.csv

Method

• Placement (placement.method): kg (COMPASS: Louvain communities of the graph, each fact on its subject entity's shard), kmeans (IVF-style semantic centroids), default (Qdrant ID-hash).
• Strategy (placement.strategy, the locality<->balance tradeoff): whole -> cap_split (bounded splitting) -> scatter.
• Online insertion: personalized PageRank over the knowledge graph places new entities based on their connections to existing entities.
• Query routing: community-level summaries rank the shards for each query; centroid ranking of the query vector is the embedding-only baseline.
• Selective search: workload.fanout_probes controls the number of shards searched per query, versus full broadcast.

Each run writes a summary.json with recall@10/@50 per competency, corpus fraction searched, shard-load Gini, query throughput, p50/p95/p99 latency, and peak per-node memory.

Execution Environment

The experiments were conducted on the Aurora supercomputer using PBS. GPUs are used only for embedding, while Qdrant workers serve data from in-memory storage. The launch scripts under hpc/ and qdrant/ are cluster-specific; data preparation and offline benchmarks can be run on a single node.