`sqldb/v2` as separate package #10175

ViktorT-11 · 2025-08-26T15:01:12Z

This PR addresses Phase 1 of the release plan of comment #9762 (comment) in #9762.

This PR extracts the non-specific lnd specific logic of the sqldb package into a new sqldb/v2 package, and introduces the extra functionality that sqldb/v2 adds.

ellemouton

Did a first pass, looking really good!

sqldb/v2/migrations.go

sqldb/v2/interfaces.go

sqldb/v2/postgres_test.go

ellemouton · 2025-09-04T06:36:22Z

go.mod

+replace github.com/golang-migrate/migrate/v4 => github.com/lightninglabs/migrate/v4 v4.18.2-9023d66a-fork-pr-2
+


ok afaict, the big thing we added in the fork was the post migration call-backs logic.

Something id just like some clarity on before we push forward here is:
in our own rolled migration strategy in LND today that takes care of code migrations, we know that the migration version table will only be updated once the code migration is complete. Is the same true for the post-migration callback?

I discussed this with guggero previously.
Currently, no that is not true unfortunately, as the version table will be updated if the SQL migration succeeds, while the code migration fails. But I'm planning to address that asap 🔥!

The current plan is to submit a new PR to our migrate library fork. This PR will add a separate tracking table for code migrations. With this change, the logic will ensure that if the db is initialized and the persisted code migration version is lower than the SQL migration version, the code migration will be re-run prior to any SQL migration.

I haven't gotten to updating the migrate fork yet though. Perhaps we should hold of with merging this one until that change has been merged to the migrate fork, so that we don't need to push a new PR that updates the dependency in lnd though.

Does that sound like a good plan to you @ellemouton?

sounds good!

In the upcoming commits, we will introduce a new sqldb module, sqldb version 2. The intention of the new sqldb module, is to make it generalizable so that it contains no `lnd` specific code, to ensure that it can be reused in other projects. This commit adds the base of the new module, but does not include any implementation yet, as that will be done in the upcoming commits.

This commit moves all non lnd-specific code of sqldb/v1 to the new sqldb/v2 module. Note however, that without additional changes, this package still needs to reference lnd, as references to the lnd `sqlc` package is required without further changes. Those changes will be introduced in the upcoming commits, to fully decouple the new sqldb/v2 module from lnd.

This commit introduces a new struct named `MigrationStream`, which defines a structure for migrations SQL migrations. The `MigrationStream` struct contains the SQL migrations which will be applied, as well as corresponding post-migration code migrations which will be executed afterwards. The struct also contains fields which define how the execution of the migrations are tracked. Importantly, it is also possible to define multiple different `MigrationStream`s which are executed, to for example define one `prod` and one `dev` migration stream.

This commit updates the `sqldb/v2` package to utilize the new `MigrationStream` type for executing migrations, instead of passing `[]MigrationConfig`'s directly.

This commit updates the definition of the `BaseDB` struct to decouple it from lnd`s `sqlc` package. We also introduce new fields to the struct to make it possible to track the database type used at runtime.

Drop the GetSchemaVersion and SetSchemaVersion methods for the PostgresStore and SqliteStore structs in the sqldb/v2 package.

In order to make it possible to replace `tapd`'s internal `sqldb` package with the new generic `sqldb/v2` package, we need to make sure that all features and functionality that currently exist in the `tapd` package are also present in the new `sqldb/v2` package. This commit adds such additional missing features to the `sqldb/v2` package.

Previously, the test db helper files were suffixed with "_test", which would indicate that the files specifically contained tests. However, these files actually contain helper functions to be used in tests, and are not tests themselves. To better reflect their purpose, the files have been renamed to instead be prefixed with "test_".

ViktorT-11 · 2025-09-04T11:35:53Z

Thanks a lot for the review @ellemouton 🔥🎉! Did one separate push to address the latest feedback (and add the contents of #10193) + one push to rebase this PR on master.

I'm not re-requesting a review of you just yet until we have decided on #10175 (comment) @ellemouton.

ViktorT-11 · 2025-09-04T18:08:53Z

Will address failing CI checks on the next push after the next review round.

ViktorT-11 · 2025-09-05T10:09:02Z

Based the PR on the now updated sqldb-v2 branch 🎉!

GustavoStingelin

Great work here! I'm excited to see this moving forward and to use it in btcwallet.
I’ve opened a PR with some related work already, but this looks more polished and complete.

I’ve added a few comments as well.

GustavoStingelin · 2025-09-15T18:47:37Z

sqldb/v2/migrations.go

+	// MigrateTableName is the name of the table used by golang-migrate to
+	// track the current schema version.
+	MigrateTableName string


What do you think about renaming this to MigrationTableName? Or, to avoid repeating the word “migration”, maybe TrackingTableName would work better. To me, MigrateTableName sounds more like the table that is going to be migrated, rather than the one used for tracking migrations.

GustavoStingelin · 2025-09-15T18:52:50Z

sqldb/v2/migrations.go

+	// Schemas is the embedded file system containing the SQL migration
+	// files.
+	Schemas embed.FS
+
+	// SQLFileDirectory is the directory containing the SQL migration files.
+	SQLFileDirectory string


Here we’re using two different names to describe the DDL migrations. One is Schemas, which might be confusing since it sounds like the literal database.schema relation. The other is SQLFileDirectory, which I think works well. What do you think about renaming it to something like DDLFiles and DDLFilesDirectory (or something along those lines) to make the intent clearer?

Or SQLFiles and SQLFilesDirectory as well.

GustavoStingelin · 2025-09-15T19:13:36Z

sqldb/v2/no_sqlite.go

+// SqliteStore is a database store implementation that uses a sqlite backend.
+type SqliteStore struct {
+	cfg *SqliteConfig
+
+	*BaseDB
+}


Maybe it could be clearer that it's a fake implementation.

GustavoStingelin · 2025-09-15T19:16:25Z

sqldb/v2/paginate.go

+	// maxSQLiteBatchSize is the maximum number of items that can be
+	// included in a batch query IN clause for SQLite. This was determined
+	// using the TestSQLSliceQueries test.
+	maxSQLiteBatchSize = 32766
+
+	// maxPostgresBatchSize is the maximum number of items that can be
+	// included in a batch query IN clause for Postgres. This was determined
+	// using the TestSQLSliceQueries test.
+	maxPostgresBatchSize = 65535


I think this value applies to all query parameters used in the query, not only for those used in the IN statement, so if this limit is reached when we have other parameters aside the IN, it will return an error.

GustavoStingelin · 2025-09-15T19:43:18Z

sqldb/v2/postgres.go

+	// Ensure the migration tracker table exists before running migrations.
+	// This table tracks migration progress and ensures compatibility with
+	// SQLC query generation. If the table is already created by an SQLC
+	// migration, this operation becomes a no-op.
+	migrationTrackerSQL := `
+	CREATE TABLE IF NOT EXISTS migration_tracker (
+		version INTEGER UNIQUE NOT NULL,
+		migration_time TIMESTAMP NOT NULL
+	);`


Is this different from the MigrationStream table name?

GustavoStingelin · 2025-09-15T20:20:54Z

sqldb/v2/postgres_fixture.go

+// TearDown stops the underlying docker container.
+func (f *TestPgFixture) TearDown(t testing.TB) {
+	err := f.pool.Purge(f.resource)
+	require.NoError(t, err, "Could not purge resource")
+}


This could be passed as a test Cleanup function inside the fixture creation

t.Cleanup(func() { fixture.TearDown(t) })

GustavoStingelin · 2025-09-15T20:27:19Z

sqldb/v2/sqlerrors.go

+	// ErrRetriesExceeded is returned when a transaction is retried more
+	// than the max allowed valued without a success.
+	ErrRetriesExceeded = errors.New("db tx retries exceeded")


Suggested change

// ErrRetriesExceeded is returned when a transaction is retried more

// than the max allowed valued without a success.

ErrRetriesExceeded = errors.New("db tx retries exceeded")

// ErrTxRetriesExceeded is returned when a transaction is retried more

// than the max allowed valued without a success.

ErrTxRetriesExceeded = errors.New("db tx retries exceeded")

Maybe ErrTxRetriesExceeded to avoid confusion with other types of retry.

GustavoStingelin · 2025-09-15T20:29:07Z

sqldb/v2/sqlerrors.go

+	// postgresErrMsgs are strings that signify retriable errors resulting
+	// from serialization failures.
+	postgresErrMsgs = []string{
+		"could not serialize access",
+		"current transaction is aborted",
+		"not enough elements in RWConflictPool",
+		"deadlock detected",
+		"commit unexpectedly resulted in rollback",
+	}


Maybe postgresRetriableErrMsgs to be clearer.

Suggested change

// postgresErrMsgs are strings that signify retriable errors resulting

// from serialization failures.

postgresErrMsgs = []string{

"could not serialize access",

"current transaction is aborted",

"not enough elements in RWConflictPool",

"deadlock detected",

"commit unexpectedly resulted in rollback",

}

// postgresRetriableErrMsgs are strings that signify retriable errors resulting

// from serialization failures.

postgresRetriableErrMsgs = []string{

"could not serialize access",

"current transaction is aborted",

"not enough elements in RWConflictPool",

"deadlock detected",

"commit unexpectedly resulted in rollback",

}

GustavoStingelin · 2025-09-15T20:33:40Z

sqldb/v2/sqlite.go

+// NewTestSqliteDB is a helper function that creates an SQLite database for
+// testing.
+func NewTestSqliteDB(t testing.TB, streams []MigrationStream) *SqliteStore {


I think we should separate the test helpers in another file from the functions used in "production".

GustavoStingelin · 2025-09-15T20:45:32Z

sqldb/v2/test_postgres.go

+	t.Cleanup(func() {
+		pgFixture.TearDown(t)
+	})
+


Maybe we can move this to DB creation, like done in SQLite.

lightninglabs-deploy · 2025-10-06T23:12:19Z

@ziggie1984: review reminder
@ViktorT-11, remember to re-request review from reviewers when ready

GustavoStingelin · 2025-10-08T02:09:01Z

Anything that I can do to help put energy into this?

ViktorT-11 · 2025-10-08T08:30:49Z

Hey @GustavoStingelin! Thank you so much for the review above 🙏!

Anything that I can do to help put energy into this?

Apologies, I should probably have informed you this in this PR earlier:
We're waiting for this PR in the dependent migrate fork to be merged before I'm going to start pushing for this PR again, including addressing your great feedback:
lightninglabs/migrate#3

As that PR will require updates of the implementation of this PR, it makes most sense that it is finalised and merged before changing this dependent PR.

In terms of areas where you can help out: Continuing to review this PR once the above is merged and I've addressed your feedback here would be very helpful. Additionally, if you want to, reviewing the above migrate PR would also be very helpful 🙏.

ViktorT-11 changed the title ~~2025 08 sqldb v2 separate package~~ sqldb/v2 as separate package Aug 26, 2025

ViktorT-11 changed the base branch from sqldb-v2 to master August 26, 2025 21:43

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch 3 times, most recently from 6b5e428 to 6899a39 Compare September 2, 2025 23:30

ViktorT-11 mentioned this pull request Sep 3, 2025

[sql-43] Add support for sqldb/v2 in litd lightninglabs/lightning-terminal#1085

Draft

ellemouton reviewed Sep 4, 2025

View reviewed changes

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch from 6899a39 to 7ca3000 Compare September 4, 2025 11:26

ViktorT-11 added 9 commits September 4, 2025 13:28

sqldb/v2: Use MigrationStream for migrations

9c1df3d

This commit updates the `sqldb/v2` package to utilize the new `MigrationStream` type for executing migrations, instead of passing `[]MigrationConfig`'s directly.

sqldb/v2: introduce sqldb/v2 BaseDB

a1d3359

This commit updates the definition of the `BaseDB` struct to decouple it from lnd`s `sqlc` package. We also introduce new fields to the struct to make it possible to track the database type used at runtime.

sqldb/v2: drop GetSchemaVersion & SetSchemaVersion

fafd451

Drop the GetSchemaVersion and SetSchemaVersion methods for the PostgresStore and SqliteStore structs in the sqldb/v2 package.

lnd: add sqldb/v2 dependency to lnd

233f1fb

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch from 7ca3000 to 233f1fb Compare September 4, 2025 11:32

ViktorT-11 requested a review from ziggie1984 September 4, 2025 11:36

ViktorT-11 changed the base branch from master to sqldb-v2 September 5, 2025 10:08

ViktorT-11 marked this pull request as ready for review September 5, 2025 10:09

GustavoStingelin suggested changes Sep 15, 2025

View reviewed changes

GustavoStingelin mentioned this pull request Sep 15, 2025

multi: add database test helpers btcsuite/btcwallet#1027

Draft

saubyk added this to the v0.21.0 milestone Sep 16, 2025

mohamedawnallah mentioned this pull request Oct 9, 2025

[1/10][wtxmgr]: sqlize blocks dimension table btcsuite/btcwallet#1101

Draft

7 tasks

		replace github.com/golang-migrate/migrate/v4 => github.com/lightninglabs/migrate/v4 v4.18.2-9023d66a-fork-pr-2

sqldb/v2 as separate package #10175

Are you sure you want to change the base?

sqldb/v2 as separate package #10175

Uh oh!

Conversation

ViktorT-11 commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorT-11 Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorT-11 commented Sep 4, 2025

Uh oh!

ViktorT-11 commented Sep 4, 2025

Uh oh!

ViktorT-11 commented Sep 5, 2025

Uh oh!

GustavoStingelin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightninglabs-deploy commented Oct 6, 2025

Uh oh!

GustavoStingelin commented Oct 8, 2025

Uh oh!

ViktorT-11 commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

`sqldb/v2` as separate package #10175

`sqldb/v2` as separate package #10175

ViktorT-11 commented Aug 26, 2025 •

edited

Loading

ViktorT-11 Sep 4, 2025 •

edited

Loading