Skip to content

Add SQLite support, bring parity with dbos-transact-py#1288

Open
tobalo wants to merge 12 commits into
dbos-inc:mainfrom
tobalo:feature/sqlite-system-queues
Open

Add SQLite support, bring parity with dbos-transact-py#1288
tobalo wants to merge 12 commits into
dbos-inc:mainfrom
tobalo:feature/sqlite-system-queues

Conversation

@tobalo

@tobalo tobalo commented Jun 30, 2026

Copy link
Copy Markdown

This change adds SQLite-backed system database support for local development and tests.
When no system database URL is provided, DBOS defaults to a per-app SQLite file. Explicit
Postgres systemDatabaseUrl values are preserved and continue to use the existing
Postgres path.

SQLite support is additive. Postgres remains the recommended production system database.

Intent is to help with #1226

Brief implementation description

  • Added a better-sqlite3 system database adapter with SQLite URL handling, schema setup
    and reset, WAL, busy_timeout, foreign keys, and serialized logical client leasing.
  • Wired SQLite into config, executor/client initialization, and CLI schema/reset flows.
  • Translated DBOS system database SQL shapes into SQLite-compatible SQL, including casts,
    ANY(...), schema prefixes, CTE writes, UUID/time expressions, booleans, and
    RETURNING variants.
  • Disabled Postgres listen/notify behavior for SQLite.
  • Normalized SQLite row differences for queue booleans and workflow attributes without
    changing the explicit Postgres configuration path.
  • Added regression coverage for SQLite queues, fork workflow behavior, CTE writes, pool
    shutdown behavior, and explicit Postgres systemDatabaseUrl preservation.

Testing performed

  • node_modules/.bin/prettier --write src/sqlite_system_database.ts src/system_database.ts tests/sqlite_system_database.test.ts
  • npm run lint
  • npm run build
  • node_modules/.bin/jest tests/config.test.ts tests/sqlite_system_database.test.ts --runInBand
  • npx tsc --project tsconfig.json --noEmit
  • git diff --check

After the final cleanup to src/system_database.ts, the focused verification run was:

  • node_modules/.bin/prettier --write src/system_database.ts
  • npx tsc --project tsconfig.json --noEmit
  • git diff --check

Development note

Implemented with assistance from Codex 5.5 xhigh.

@kraftp kraftp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitting this! It's a substantial contribution, so review may take some time. I'm frankly a bit concerned by a large LLM-generated dialect translator, and while it's not necessarily a blocker, we'll have to be careful with it. Two things we need to see before looking at this further:

  1. Like in Python, we need all (or almost all) unit tests to run against both SQLite and Postgres.
  2. Instead of bringing in a large new dependency with many transitive dependencies, is it possible to build this on Node's built-in sqlite? This may mean gating it behind more recent versions of Node, we'd have to make sure that's possible without breaking DBOS on older Node versions (DBOS supports Node >20, SQLite was added in 22 and stabilized in 26). This may not be possible--I'm not very familiar with the Node SQLite API.

@tobalo

tobalo commented Jul 9, 2026

Copy link
Copy Markdown
Author

@kraftp I will research and learn further on the native node:sqlite drivers and refactor. The test suite did work via my local tests and pasts with existing test suite. I leaned heavily mirroring the python implementation.

Gating behind only recent versions of node is a good point, I didn't consider and will come back with.

@tobalo

tobalo commented Jul 11, 2026

Copy link
Copy Markdown
Author

Changes to depend solely on node:sqlite - test suite looks good on local run. If there's more expanded tests you want me to run for more stress/regression testing, let me know.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine maybe was a but presumptious to go ahead and bump the postgres:image but made sense in my head. The $PGDATA path change may not be optimal per the CI runners. Can't exactly verify, or perhaps should. May make sense to revert change on line 61

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants