Found while reviewing #267 (sec spac download registration|8k|everything). Deliberately kept out of the fix PRs on that branch (#268, #270) because it is a design change to the producer, not a bug fix.
The problem
DownloadSpacCandidateDocsTask.execute selects every filing of every high+medium SPAC candidate and builds the entire worklist in memory before issuing a single fetch:
const filings: Filing[] = [];
for (chunk of ciks) filings.push(...(await filingRepo.query({ cik: { value: chunk, operator: "in" } })));
const matchedRows = filings.filter(...);
const todo: CacheOneInput[] = [];
for (const row of matchedRows) { ...; todo.push({...}); }That is three live copies of the same set — filings, matchedRows, todo — held simultaneously, and there is no bound of any kind: no --limit, no --from/--to filing-date range, no --cik, no form filter beyond the three fixed sets. sec spac download everything means literally every filing for every candidate.
This reintroduces exactly the pattern formsSweep.ts / ComputeFormsWorklistTask were rewritten to remove. From ComputeFormsWorklistTask's own comments, measured on this corpus:
The candidate set is far too large to materialize: form 4 alone is ~4.6M filings and the full 55-form worklist ~6.4M, at a measured ~460 bytes per 15-column row — ~3 GB per process […]
it previously materialized every matching filing before the first fetch, ~1M entries and 158 MB for one shard of today's corpus, and growing without bound. Batching caps that at ~5k entries (~0.8 MB) and, more importantly, starts real work immediately instead of after a multi-minute scan.
The SPAC candidate set is smaller than the full corpus, but it is thousands of CIKs with unbounded filing histories, it grows monotonically, and the second half of that quote applies regardless of size: today the operator waits out a full scan before the first byte is fetched.
These are ONE change, not two
It is tempting to file "add --limit" and "stream the producer" separately. They are the same change.
--limit is a stop condition on a lazy producer. Applied to a materialized array it is a .slice() — the full scan is already paid, all three copies already exist, and the flag saves only fetches. The memory and time-to-first-fetch costs, which are the actual problems, are untouched. So the producer has to become lazy first, and then --limit is one line of it.
Two constraints found while reviewing
Anyone picking this up should know both up front, because they shape the design:
1. SearchCriteria allows one condition per column and has no OR. So filing_date >= from AND filing_date <= to is not expressible as a query — only one side of a --from/--to range can be pushed down to the database, and the other must be a JS predicate applied to rows already read. ComputeFormsWorklistTask.readPage documents the same limitation for its keyset resume:
SearchCriteria allows one condition per column and has no OR, so the exact keyset predicate (cik, accession) > (lastCik, lastAccession) is not expressible.
The consequence is not cosmetic: --limit must be counted on YIELDED items, never on rows read. With half the range filter running in JS, rows-read and items-yielded diverge arbitrarily, and a limit counted on the former would stop early on a range whose rows mostly fail the predicate — silently downloading fewer documents than asked for.
2. The honest test is a temporal one. Asserting on peak RSS is flaky, and asserting that a limited run fetched N documents passes just as well against a .slice() of a fully materialized array. The assertion that actually pins the memory behaviour is:
the first fetch is issued before the last page of filings is read.
Instrument the repo query and the fetch seam with a shared sequence counter and assert the ordering. That test fails today and is the only one that cannot be satisfied by a materialize-then-trim implementation.
Suggested shape
Mirror ComputeFormsWorklistTask: a paged producer over filings in primary-key order with a last-key resume, emitting batches into the existing FORMS_SWEEP_CONCURRENCY_LIMIT worker pool, with --limit / --from / --to / --cik / form filters evaluated inside the producer. The counters (matched, skipped, downloaded, failed) accumulate across batches rather than being derived from array lengths.
Related
Found while reviewing #267 (
sec spac download registration|8k|everything). Deliberately kept out of the fix PRs on that branch (#268, #270) because it is a design change to the producer, not a bug fix.The problem
DownloadSpacCandidateDocsTask.executeselects every filing of every high+medium SPAC candidate and builds the entire worklist in memory before issuing a single fetch:That is three live copies of the same set —
filings,matchedRows,todo— held simultaneously, and there is no bound of any kind: no--limit, no--from/--tofiling-date range, no--cik, no form filter beyond the three fixed sets.sec spac download everythingmeans literally every filing for every candidate.This reintroduces exactly the pattern
formsSweep.ts/ComputeFormsWorklistTaskwere rewritten to remove. FromComputeFormsWorklistTask's own comments, measured on this corpus:The SPAC candidate set is smaller than the full corpus, but it is thousands of CIKs with unbounded filing histories, it grows monotonically, and the second half of that quote applies regardless of size: today the operator waits out a full scan before the first byte is fetched.
These are ONE change, not two
It is tempting to file "add
--limit" and "stream the producer" separately. They are the same change.--limitis a stop condition on a lazy producer. Applied to a materialized array it is a.slice()— the full scan is already paid, all three copies already exist, and the flag saves only fetches. The memory and time-to-first-fetch costs, which are the actual problems, are untouched. So the producer has to become lazy first, and then--limitis one line of it.Two constraints found while reviewing
Anyone picking this up should know both up front, because they shape the design:
1.
SearchCriteriaallows one condition per column and has no OR. Sofiling_date >= from AND filing_date <= tois not expressible as a query — only one side of a--from/--torange can be pushed down to the database, and the other must be a JS predicate applied to rows already read.ComputeFormsWorklistTask.readPagedocuments the same limitation for its keyset resume:The consequence is not cosmetic:
--limitmust be counted on YIELDED items, never on rows read. With half the range filter running in JS, rows-read and items-yielded diverge arbitrarily, and a limit counted on the former would stop early on a range whose rows mostly fail the predicate — silently downloading fewer documents than asked for.2. The honest test is a temporal one. Asserting on peak RSS is flaky, and asserting that a limited run fetched N documents passes just as well against a
.slice()of a fully materialized array. The assertion that actually pins the memory behaviour is:Instrument the repo query and the fetch seam with a shared sequence counter and assert the ordering. That test fails today and is the only one that cannot be satisfied by a materialize-then-trim implementation.
Suggested shape
Mirror
ComputeFormsWorklistTask: a paged producer overfilingsin primary-key order with a last-key resume, emitting batches into the existingFORMS_SWEEP_CONCURRENCY_LIMITworker pool, with--limit/--from/--to/--cik/ form filters evaluated inside the producer. The counters (matched,skipped,downloaded,failed) accumulate across batches rather than being derived from array lengths.Related
primary_doccrash in this same worklist loop (also what makes feat(spac): pre-download candidate registration, 8-K, and all filings #267's CI red)--forceactually evict the cached accession document #270 —--forcenever evicted the cache entry. Note that PR makes--forcedelete files first, which raises the stakes here: an unbounded--force everythingevicts a large cache before re-fetching it at the SEC rate limit. Worth landing this bounding work before--force everythingis documented as a normal thing to run.