Optimize the VSEARCH sequence pipeline (async + incremental indexing + benchmark)

[cl117]

Summary
The sequence-search and clustering pipeline is slow due to engineering debt, not the algorithm. This task makes it fast and produces a clean before/after benchmark. We keep VSEARCH — a prior group paper established VSEARCH > BLAST for our data; switching tools is explicitly out of scope.

Background / where the slowness is
Relevant files: [flask/sequencesearch.py], [flask/cluster.py]
Three known hotspots:

Synchronous blocking subprocess — a sequence search blocks the whole service; other requests queue behind it.
Repeated linear scans over .uc files — the cluster file is re-parsed/scanned on every lookup instead of being parsed once into memory.
Full-corpus re-cluster on every index rebuild — no incremental path; adding one part re-clusters all ~100k sequences.
Goal
Same results, much faster, no service blocking — and a reproducible benchmark proving it.

Milestone 1 (Month 1) — Profile, baseline & harness

• Export the current part sequence corpus (FASTA) used by the pipeline.
• Profile the existing pipeline and confirm/quantify the three hotspots above (measure where wall-clock goes).
• Build a reproducible benchmark harness: a script that runs the pipeline on a given corpus and emits CSV with search latency, throughput (queries/sec), peak memory, and full-rebuild time.
• Record baseline numbers on a fixed corpus snapshot (this is the "before").

✅ Exit criterion: harness runs end-to-end on a sample corpus and reproduces stable baseline numbers.