Clustering step uses 5 CPUs and around N hours. Mem ~340MB.
Not able to see progress.
(a) Make sequencesearch.py async / non-blocking (a search must not block other requests).
(b) Replace repeated .uc linear scans with a parse-once in-memory index (lookup, not re-scan).
(c) Add incremental indexing — re-cluster only new/changed parts instead of the whole corpus.
Re-run the harness → produce the before/after speedup figure (latency / throughput / rebuild time).
Write a 2–3 page report describing the changes and the measured gains.
✅ Exit criterion: measurable speedup on all three metrics, identical (or documented-equivalent) search results vs baseline.
Deliverables
Correctness guard (don't skip)
Before/after must return the same clusters / hits on the benchmark corpus (or any difference must be explained). Add a small diff check comparing old vs new output on a fixed input.
Clustering step uses 5 CPUs and around N hours. Mem ~340MB.
Not able to see progress.
(a) Make sequencesearch.py async / non-blocking (a search must not block other requests).
(b) Replace repeated .uc linear scans with a parse-once in-memory index (lookup, not re-scan).
(c) Add incremental indexing — re-cluster only new/changed parts instead of the whole corpus.
Re-run the harness → produce the before/after speedup figure (latency / throughput / rebuild time).
Write a 2–3 page report describing the changes and the measured gains.
✅ Exit criterion: measurable speedup on all three metrics, identical (or documented-equivalent) search results vs baseline.
Deliverables
Reproducible benchmark harness + the corpus snapshot used.
Before/after speedup figure (this becomes a paper figure).
2–3 page written report.
Optimized code as a PR/branch (final wiring into production stays with @cl117).
Correctness guard (don't skip)
Before/after must return the same clusters / hits on the benchmark corpus (or any difference must be explained). Add a small diff check comparing old vs new output on a fixed input.