Skip to content

feat: add server-internal database storage (skserver.sqlite)#2424

Open
dirkwa wants to merge 7 commits intoSignalK:masterfrom
dirkwa:feat-database-api-serverdb
Open

feat: add server-internal database storage (skserver.sqlite)#2424
dirkwa wants to merge 7 commits intoSignalK:masterfrom
dirkwa:feat-database-api-serverdb

Conversation

@dirkwa
Copy link
Contributor

@dirkwa dirkwa commented Mar 8, 2026

Summary

Builds on #2409 (Database API). Adds getServerDb() so the server itself can persist operational data to a dedicated skserver.sqlite file — separate from plugin databases.

Motivated by the discussion in #2369 (review comment): tracking device token "last seen" timestamps needs persistence, and writing high-frequency data to security.json isn't the right tool.

  • getServerDb() added as optional method on DatabaseProvider — existing community providers won't break
  • Both built-in providers (node:sqlite, better-sqlite3) implement it
  • Registry falls back to a built-in if the active community provider doesn't implement it
  • Not exposed to plugins — server-internal only, not on the DatabaseApi interface
  • Lazy: skserver.sqlite is only created when a consumer first calls getServerDb()
  • Returns the same PluginDb handle (migrate, query, run, transaction)

File layout

{configPath}/ 
├── skserver.sqlite ← server-internal (NEW) 
├── plugin-db/
│          ├── my-plugin.db ← per-plugin (existing)
│          └── ... 
├── security.json 
└── settings.json

What this enables (future PRs)

Use case Currently With getServerDb()
Token "last seen" Not tracked Persist on each authenticated request
Access request audit trail In-memory, lost on restart Persist to database

Tests

Three new tests covering the key scenarios:

  1. Registry fallback — registers a community provider without getServerDb(), verifies the registry falls back to the built-in provider and the server database works (migrate, insert, file exists)
  2. Provider close() releases the database — opens the server db, stops the registry, opens a fresh registry on the same file, verifies no locked-file error
  3. Type compatibility — a provider with only getPluginDb + close satisfies DatabaseProvider (no getServerDb required)

Tested

  • Node 20 (better-sqlite3 only), Node 22, Node 24 — all pass
  • Direct unit tests: migrate, insert, query, caching, file location, close
  • Registry fallback logic verified
  • Server startup: no skserver.sqlite created when no consumer calls it

Depends on: #2409

dirkwa added 7 commits March 9, 2026 09:01
Introduce a server-managed relational database API that gives each
plugin its own isolated SQLite database. The API supports migrations,
parameterized queries, transactions, and pluggable provider backends.

Two built-in providers ship with the server:
- _builtin: SQLite via better-sqlite3 (default)
- _builtin_nodesqlite: SQLite via node:sqlite (Node >= 22.5.0)

Plugins can register alternative providers (e.g. PostgreSQL) via
app.registerDatabaseProvider() and consumers access the API via
app.getDatabaseApi().getPluginDb(plugin.id).
On Node >= 22.5.0, use the built-in node:sqlite module as the default
database provider. This eliminates the native addon dependency for
most users. On Node 20 (e.g. Victron Cerbo), better-sqlite3 remains
the default since node:sqlite is not available.

Both providers use lazy require() with try/catch guards so the server
boots cleanly regardless of what is available. Plugin providers now
correctly override either built-in default.
Align with the naming convention proposed in SignalK#2417:
Provider = what plugins implement, Api = what plugins consume.
The registry interface follows the <D>ProviderRegistry pattern
consistent with WeatherProviderRegistry, RadarProviderRegistry, etc.
Add consumer-facing documentation (database_api.md) covering plugin
usage, migrations, queries, transactions, and provider management
REST endpoints. Add provider plugin documentation
(database_provider_plugins.md) with interface requirements, registration
example, and SQL portability notes.

Update JSDoc in databaseapi.ts to reflect node:sqlite preference order.
Extend the Database API with server-internal storage (skserver.sqlite)
for operational data like token activity and access request audit trails.

- Add optional getServerDb() to DatabaseProvider interface
- Implement in both built-in providers (node:sqlite, better-sqlite3)
- Wire up in registry with fallback to built-in if the active community
  provider doesn't implement it
- Not exposed to plugins — server-internal only
@dirkwa
Copy link
Contributor Author

dirkwa commented Mar 9, 2026

Further use cases:

In #2379:

  • Show when a offline tagged device was last seen
  • Optional: Store known devices cache (filebased atm)
  • Optional: Store Source priority logic (filebased atm)

In BLE Provider API https://discord.com/channels/1170433917761892493/1478948783630057596:

  • Show when offline tagged device was last seen
  • Store known devices

@dirkwa
Copy link
Contributor Author

dirkwa commented Mar 9, 2026

Also worth looking into:

@signalk/resources-provider

  • Stores waypoints, routes, notes, regions as one JSON file per resource
  • Listing resources =readdir→ loop through every file → readFile + JSON.parse + stat each one
  • No filtering at storage level — reads all, filters in JS
  • A database would give indexed queries, pagination, spatial filtering

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant