Skip to content

[recipes] Brain backup and export#193

Merged
justfinethanku merged 4 commits into
NateBJones-Projects:mainfrom
alanshurafa:contrib/alanshurafa/brain-backup
Jun 11, 2026
Merged

[recipes] Brain backup and export#193
justfinethanku merged 4 commits into
NateBJones-Projects:mainfrom
alanshurafa:contrib/alanshurafa/brain-backup

Conversation

@alanshurafa

Copy link
Copy Markdown
Collaborator

Summary

Node.js recipe that exports every Open Brain Supabase table to dated local JSON files with streaming pagination.

  • Paginates through PostgREST 1,000 rows at a time; streams rows to disk so large tables never load fully into memory
  • Dated filenames (<table>-YYYY-MM-DD.json) for simple daily retention
  • Prints a row-count and file-size summary per table
  • thoughts is required; optional companion tables (entities, edges, thought_entities, ingestion_jobs, ingestion_items) are backed up only if present. Stock Open Brain installs skip them cleanly instead of erroring.
  • No dependencies beyond Node 18+; config via .env.local

Why

Open Brain users accumulate thousands of thoughts over time but Supabase-side backups require plan-tier access or manual dumps. A zero-dependency Node recipe gives anyone on the free tier a scriptable backup path that runs from cron / Task Scheduler.

The optional-table handling lands specifically so this recipe works against every Open Brain install, not just ones that have also installed the entity-extraction or smart-ingest schemas. Missing companion tables return a PostgREST 404, which the script treats as "skip with a log line" rather than an error.

Part 11 of 12 in the OB1 Alpha Milestone consolidation.

Test plan

  • Run against a stock Open Brain project (only thoughts table) — verify thoughts backs up, optional tables log skipped (table not present)
  • Run against a project that has the optional tables — verify all exports succeed
  • Drop thoughts temporarily → verify the script throws Required table "thoughts" not found
  • Kill mid-run on a large thoughts export → verify no partial/corrupt JSON left behind (next run overwrites cleanly)
  • Verify dated filename rotates correctly across days
  • node --check backup-brain.mjs passes
  • metadata.json passes the gate schema

@github-actions github-actions Bot added the recipe Contribution: step-by-step recipe label Apr 18, 2026
@alanshurafa alanshurafa added area: recipes Review area: recipes review: needs-refresh Branch is stale, conflicted, or needs rebase before review alan-reviewed Reviewed by Alan Shurafa in Community Reviewer role labels May 20, 2026
@alanshurafa

Copy link
Copy Markdown
Collaborator Author

Conflicts with main. Before I rebase, should this stay as-is, be reshaped, or be dropped? I can refresh it quickly if you want to keep it.

Exports Supabase tables to JSON. Hardened from code review: pagination
that terminates correctly, PGRST205-only "table missing" detection,
atomic writes via tmp+rename, an AbortController fetch timeout, and
BOM-tolerant env parsing.
@alanshurafa alanshurafa force-pushed the contrib/alanshurafa/brain-backup branch from 5d67ad3 to 119ac66 Compare May 20, 2026 16:34
@alanshurafa

Copy link
Copy Markdown
Collaborator Author

Rebased onto main — no conflicts now. This updates the existing recipes/brain-backup/ rather than adding it.

For review context: main's copy of this recipe predates its code review. This PR is the reviewed version — terminating pagination (the loop on main checks a stale page-size value), PGRST205-only "table missing" handling, atomic writes via tmp+rename, an AbortController fetch timeout, and BOM-tolerant env parsing. An upgrade over what's on main.

@alanshurafa alanshurafa added review: ready-for-maintainer Community reviewer recommends maintainer review and removed review: needs-refresh Branch is stale, conflicted, or needs rebase before review labels May 20, 2026

@justfinethanku justfinethanku left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the recipe/code updates, added the missing community credit block, and confirmed the refreshed branch passes the visible required Actions checks. Approving for maintainer merge.

@justfinethanku justfinethanku merged commit 8fbc5e7 into NateBJones-Projects:main Jun 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

alan-reviewed Reviewed by Alan Shurafa in Community Reviewer role area: recipes Review area: recipes recipe Contribution: step-by-step recipe review: ready-for-maintainer Community reviewer recommends maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants