feat: add BootstrapSubmissions command and task for processing SEC submissions by sroussey · Pull Request #62 · workglow-dev/sec

sroussey · 2026-03-02T18:22:40Z

Introduced a new command bootstrap-submissions to bootstrap submissions from pre-downloaded files in the SEC_RAW_DATA_FOLDER.
Created BootstrapSubmissionsTask to handle the logic for reading CIK files, filtering unprocessed submissions, and executing fetch and store operations.
Updated command index to include the new BootstrapSubmissions command.

…bmissions - Introduced a new command `bootstrap-submissions` to bootstrap submissions from pre-downloaded files in the SEC_RAW_DATA_FOLDER. - Created `BootstrapSubmissionsTask` to handle the logic for reading CIK files, filtering unprocessed submissions, and executing fetch and store operations. - Updated command index to include the new BootstrapSubmissions command.

Copilot

Pull request overview

Adds a new CLI entrypoint to bootstrap SEC “company submissions” from already-downloaded JSON files under SEC_RAW_DATA_FOLDER/submissions, processing only CIKs that have not yet been recorded in the processed-submissions table.

Changes:

Introduces BootstrapSubmissionsTask to scan the local submissions cache, filter out already-processed CIKs, and run fetch+store per remaining CIK.
Adds a new bootstrap-submissions CLI command wiring to run the task.
Registers the new command in the commands index.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
`src/task/submissions/BootstrapSubmissionsTask.ts`	New task that enumerates cached CIK submission files and processes unprocessed ones via the existing fetch/store pipeline.
`src/commands/BootstrapSubmissions.ts`	New commander CLI command that runs the bootstrap task.
`src/commands/index.ts`	Adds the new command to the CLI command registry.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-02T18:26:54Z

+      await wf.run({
+        cik: unprocessedCiks,
+        date: unprocessedCiks.map(() => ""),
+      });


wf.run passes date: "" for each CIK. FetchSubmissionsTask uses TypeOptionalSecDate which validates the sec-date format when the field is present, so an explicit empty string can fail schema validation before the task executes. Since date is optional here, omit it from the map inputs (or pass undefined/make it optional in the map item type) so bootstrapping doesn't inject an invalid date value.

Copilot · 2026-03-02T18:26:55Z

+async function fetchAndStoreSubmission(
+  input: { cik: number; date: string },
+  ctx: IExecuteContext
+): Promise<{ success: boolean }> {
+  const pipeline = ctx.own(pipe([new FetchSubmissionsTask(), new StoreSubmissionsTask()]));
+  try {
+    await pipeline.run(input);
+  } catch (e) {
+    await processUpdateProcessing(input.cik, false);
+  }
+  return { success: true };


fetchAndStoreSubmission is effectively a copy of the helper in UpdateAllSubmissionsTask. Consider extracting this into a shared helper (e.g., in src/task/submissions/), or reusing an exported function, to avoid the two implementations drifting (concurrency, error handling, processing updates) over time.

sroussey · 2026-03-02T18:27:41Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2026-03-02T18:27:48Z

@sroussey I've opened a new pull request, #63, to work on those changes. Once the pull request is ready, I'll request review from you.

…empty date in BootstrapSubmissionsTask (#63) * Initial plan * fix: extract shared fetchAndStoreSubmission helper and fix invalid empty date in BootstrapSubmissionsTask Co-authored-by: sroussey <127349+sroussey@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sroussey <127349+sroussey@users.noreply.github.com>

…files

Add BootstrapDownload command to download and extract SEC bulk ZIP archives (submissions.zip, companyfacts.zip) and BootstrapCompanyFacts command to process extracted company facts files into the database. Extract shared fetchAndStoreCompanyFacts helper and refactor UpdateAllCompanyFactsTask to use it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ackage.json - Bumped versions of all @workglow/* packages to 0.0.109 for consistency across the project. - Updated task cleanup methods in tests to be asynchronous for improved reliability. - Added exit process call after stopping task queues in sec.ts for graceful shutdown.

Copilot

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-03T15:16:15Z

+    const processedFactsRepo = globalServiceRegistry.get(PROCESSED_FACTS_REPOSITORY_TOKEN);
+    await processedFactsRepo.put({
+      cik: input.cik,
+      last_processed: todayYYYYdMMdDD(),


In the catch block last_processed is always set to todayYYYYdMMdDD(), even when a date was provided on input. On the success path, StoreCompanyFactsTask currently records last_processed as input.date (when present), so the meaning of last_processed differs depending on success vs failure and can change update/skip behavior. Recommend making the value consistent (e.g., input.date ?? today...) for both paths.

Suggested change

last_processed: todayYYYYdMMdDD(),

last_processed: input.date ?? todayYYYYdMMdDD(),

Copilot · 2026-03-03T15:16:16Z

+      loop.pipe(fetchAndStoreCompanyFacts);
+      loop.endMap();
+      await wf.run({
+        cik: unprocessedCiks,


BootstrapCompanyFactsTask runs the map with only cik values (no date). Because StoreCompanyFactsTask only records processed facts when input.date is present, bootstrapped companies may never be marked as processed. Either pass a date here (e.g., today) or adjust the store/processing-record logic so bootstraps also persist processed status.

Suggested change

cik: unprocessedCiks,

cik: unprocessedCiks,

date: new Date().toISOString().slice(0, 10),

Copilot · 2026-03-03T15:16:16Z

+    const rawDataFolder = globalServiceRegistry.get(SEC_RAW_DATA_FOLDER);
+    const targetDir = resolve(rawDataFolder, input.targetFolder);
+    mkdirSync(targetDir, { recursive: true });
+


BootstrapDownloadTask resolves targetDir from user-provided input.targetFolder without validating it. If this task is ever called with untrusted input, values like ../... or an absolute path could write/extract outside SEC_RAW_DATA_FOLDER. Consider validating targetFolder (no path separators / no .. / not absolute) and/or asserting that the resolved path starts with rawDataFolder before proceeding.

Copilot · 2026-03-03T15:16:17Z

+    const proc = Bun.spawn(["unzip", "-o", zipPath, "-d", targetDir], {
+      stdout: "inherit",
+      stderr: "inherit",
+    });
+    const exitCode = await proc.exited;
+
+    if (exitCode !== 0) {
+      throw new Error(`unzip exited with code ${exitCode}`);
+    }


This task shells out to the external unzip binary. On environments where unzip isn't installed (or behaves differently), bootstrap-download will fail even though Bun/Node are present. Consider extracting with a JS ZIP library (or Bun APIs if available), or at least detect missing unzip and throw a clearer error with remediation steps.

Copilot · 2026-03-03T15:16:17Z

+        console.error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);
+        process.exit(1);


Using process.exit(1) inside the commander action prevents the normal CLI shutdown path in src/sec.ts (e.g., stopQueues()) from running. Prefer throwing an error (so commander prints usage) or setting process.exitCode = 1 and returning, so the top-level cleanup still happens.

Suggested change

console.error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);

process.exit(1);

throw new Error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);

Copilot · 2026-03-03T15:16:17Z


-getTaskQueueRegistry().stopQueues();
+await getTaskQueueRegistry().stopQueues();
+process.exit(0);


Calling process.exit(0) forces an immediate shutdown and can skip pending async cleanup (stdout/stderr flush, telemetry, etc.) even after stopQueues() resolves. If the goal is just to ensure the process terminates, consider setting process.exitCode = 0 (or omitting this entirely) and letting the event loop drain, unless there's a known hang that requires a hard exit.

Suggested change

process.exit(0);

process.exitCode = 0;

Copilot · 2026-03-03T15:16:18Z

@@ -0,0 +1,23 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>


The license header here says Copyright 2025, while the other newly added bootstrap commands/tasks in this PR use 2026. If these files are all being introduced together, consider aligning the year for consistency.

Suggested change

* Copyright 2025 Steven Roussey <sroussey@gmail.com>

* Copyright 2026 Steven Roussey <sroussey@gmail.com>

Copilot · 2026-03-03T15:16:18Z

+  try {
+    await pipeline.run(input);
+  } catch (e) {
+    const processedFactsRepo = globalServiceRegistry.get(PROCESSED_FACTS_REPOSITORY_TOKEN);
+    await processedFactsRepo.put({
+      cik: input.cik,
+      last_processed: todayYYYYdMMdDD(),
+      success: false,


fetchAndStoreCompanyFacts only writes to PROCESSED_FACTS_REPOSITORY_TOKEN on the error path. When this function is used by BootstrapCompanyFactsTask, inputs are passed without a date, so the success path relies on StoreCompanyFactsTask (which only updates the processed repo when input.date is set). Result: successfully bootstrapped CIKs never get marked processed (so they re-run every time), while failed CIKs do get marked (and will be skipped on subsequent bootstraps). Consider recording processed status in a finally (success + failure), using last_processed: input.date ?? todayYYYYdMMdDD() (or updating StoreCompanyFactsTask to always persist processed status).

Suggested change

try {

await pipeline.run(input);

} catch (e) {

const processedFactsRepo = globalServiceRegistry.get(PROCESSED_FACTS_REPOSITORY_TOKEN);

await processedFactsRepo.put({

cik: input.cik,

last_processed: todayYYYYdMMdDD(),

success: false,

let success = false;

try {

await pipeline.run(input);

success = true;

} catch (e) {

success = false;

} finally {

const processedFactsRepo = globalServiceRegistry.get(PROCESSED_FACTS_REPOSITORY_TOKEN);

await processedFactsRepo.put({

cik: input.cik,

last_processed: input.date ?? todayYYYYdMMdDD(),

success,

sroussey · 2026-03-03T15:19:54Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2026-03-03T15:20:03Z

@sroussey I've opened a new pull request, #64, to work on those changes. Once the pull request is ready, I'll request review from you.

sroussey requested a review from Copilot March 2, 2026 18:22

sroussey self-assigned this Mar 2, 2026

Copilot started reviewing on behalf of sroussey March 2, 2026 18:23 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

Copilot AI mentioned this pull request Mar 2, 2026

refactor: extract shared fetchAndStoreSubmission helper; fix invalid empty date in BootstrapSubmissionsTask #63

Merged

Copilot AI and others added 4 commits March 2, 2026 18:50

chore: update copyright year to 2026 in license headers for multiple …

a2c7190

…files

sroussey requested a review from Copilot March 3, 2026 15:06

Copilot started reviewing on behalf of sroussey March 3, 2026 15:06 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

Copilot AI mentioned this pull request Mar 3, 2026

fix: address review feedback on bootstrap tasks and CLI shutdown #64

Merged

sroussey and others added 2 commits March 3, 2026 05:20

Merge branch 'main' into bootstrap

4310e58

fix: address review feedback on bootstrap tasks and CLI shutdown (#64)

b18f904

sroussey merged commit 3be1835 into main Mar 3, 2026
1 of 2 checks passed

sroussey deleted the bootstrap branch March 3, 2026 16:23

	last_processed: todayYYYYdMMdDD(),
	last_processed: input.date ?? todayYYYYdMMdDD(),

	cik: unprocessedCiks,
	cik: unprocessedCiks,
	date: new Date().toISOString().slice(0, 10),

		console.error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);
		process.exit(1);

	console.error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);
	process.exit(1);
	throw new Error(`Invalid type "${type}". Must be submissions, companyfacts, or all.`);

	* Copyright 2025 Steven Roussey <sroussey@gmail.com>
	* Copyright 2026 Steven Roussey <sroussey@gmail.com>

Uh oh!

Conversation

sroussey commented Mar 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

sroussey commented Mar 2, 2026

Uh oh!

Copilot AI commented Mar 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

sroussey commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants