Skip to content

sqldb/v2 as separate package #10175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: sqldb-v2
Choose a base branch
from
Open

sqldb/v2 as separate package #10175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
72f8639
sqldb/v2: add base for sqldb/v2 module
ViktorT-11 Aug 29, 2025
212186b
sqldb/v2: move all non lnd-specific v1 code to v2
ViktorT-11 Sep 1, 2025
1fcf9a0
sqldb/v2: introduce `MigrationStream`
ViktorT-11 Sep 1, 2025
9c1df3d
sqldb/v2: Use `MigrationStream` for migrations
ViktorT-11 Sep 2, 2025
a1d3359
sqldb/v2: introduce sqldb/v2 `BaseDB`
ViktorT-11 Sep 2, 2025
fafd451
sqldb/v2: drop GetSchemaVersion & SetSchemaVersion
ViktorT-11 Sep 2, 2025
88c1ffa
sqldb/v2: sync features with tapd's sqldb package
ViktorT-11 Sep 2, 2025
53d4708
sqldb/v2: rename test db helper files
ViktorT-11 Sep 1, 2025
233f1fb
lnd: add sqldb/v2 dependency to lnd
ViktorT-11 Sep 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 34 additions & 34 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ require (
github.com/gorilla/websocket v1.5.0
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0
github.com/jackc/pgx/v4 v4.18.3
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v0.0.0-20170405195558-28a68d0c24ad
Expand All @@ -40,6 +40,7 @@ require (
github.com/lightningnetwork/lnd/kvdb v1.4.16
github.com/lightningnetwork/lnd/queue v1.1.1
github.com/lightningnetwork/lnd/sqldb v1.0.10
github.com/lightningnetwork/lnd/sqldb/v2 v2.0.0
github.com/lightningnetwork/lnd/ticker v1.1.1
github.com/lightningnetwork/lnd/tlv v1.3.2
github.com/lightningnetwork/lnd/tor v1.1.6
Expand All @@ -52,22 +53,22 @@ require (
go.etcd.io/etcd/client/pkg/v3 v3.5.12
go.etcd.io/etcd/client/v3 v3.5.12
golang.org/x/crypto v0.37.0
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028
golang.org/x/sync v0.13.0
golang.org/x/sync v0.15.0
golang.org/x/term v0.31.0
golang.org/x/time v0.3.0
google.golang.org/grpc v1.59.0
google.golang.org/protobuf v1.33.0
golang.org/x/time v0.5.0
google.golang.org/grpc v1.64.1
google.golang.org/protobuf v1.34.2
gopkg.in/macaroon-bakery.v2 v2.0.1
gopkg.in/macaroon.v2 v2.0.0
pgregory.net/rapid v1.2.0
)

require (
dario.cat/mergo v1.0.1 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/NebulousLabs/fastrand v0.0.0-20181203155948-6fb6489aac4e // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
Expand All @@ -86,28 +87,26 @@ require (
github.com/decred/dcrd/crypto/blake256 v1.0.1 // indirect
github.com/decred/dcrd/lru v1.1.2 // indirect
github.com/docker/cli v28.1.1+incompatible // indirect
github.com/docker/docker v28.1.1+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/docker v28.3.3+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fergusstrange/embedded-postgres v1.25.0 // indirect
github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-viper/mapstructure/v2 v2.3.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang-migrate/migrate/v4 v4.17.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang-migrate/migrate/v4 v4.19.0 // indirect
github.com/golang/protobuf v1.5.4
github.com/golang/snappy v0.0.4 // indirect
github.com/google/btree v1.0.1 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.14.3 // indirect
github.com/jackc/pgerrcode v0.0.0-20240316143900-6e2875d9b438 // indirect
Expand All @@ -120,7 +119,7 @@ require (
github.com/jackc/puddle v1.3.0 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jonboulle/clockwork v0.2.2 // indirect
github.com/json-iterator/go v1.1.11 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/juju/loggo v0.0.0-20210728185423-eebad3a902c4 // indirect
github.com/juju/testing v0.0.0-20220203020004-a0ff61f03494 // indirect
github.com/klauspost/compress v1.17.9
Expand All @@ -132,12 +131,12 @@ require (
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/onsi/gomega v1.26.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/opencontainers/runc v1.1.14 // indirect
github.com/ory/dockertest/v3 v3.10.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
Expand All @@ -150,7 +149,7 @@ require (
github.com/rogpeppe/fastuuid v1.2.0 // indirect
github.com/russross/blackfriday/v2 v2.0.1 // indirect
github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
github.com/sirupsen/logrus v1.9.2 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/soheilhy/cmux v0.1.5 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/objx v0.5.2 // indirect
Expand All @@ -169,43 +168,40 @@ require (
go.etcd.io/etcd/raft/v3 v3.5.12 // indirect
go.etcd.io/etcd/server/v3 v3.5.12 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/otel v1.35.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.20.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.29.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.20.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/otel/sdk v1.35.0 // indirect
go.opentelemetry.io/otel/trace v1.35.0 // indirect
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/zap v1.17.0 // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/net v0.39.0 // indirect
golang.org/x/sys v0.32.0 // indirect
golang.org/x/sys v0.34.0 // indirect
golang.org/x/text v0.24.0 // indirect
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240513163218-0867130af1f8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8 // indirect
gopkg.in/errgo.v1 v1.0.1 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
modernc.org/libc v1.49.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/sqlite v1.29.10 // indirect
modernc.org/strutil v1.2.0 // indirect
modernc.org/token v1.1.0 // indirect
modernc.org/libc v1.66.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.38.2 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
)

// TODO(elle): remove once all the schemas and queries for the graph
// store have been included in a tagged sqldb version.
replace github.com/lightningnetwork/lnd/sqldb => ./sqldb

replace github.com/lightningnetwork/lnd/sqldb/v2 => ./sqldb/v2

// This replace is for https://github.com/advisories/GHSA-25xm-hr59-7c27
replace github.com/ulikunitz/xz => github.com/ulikunitz/xz v0.5.11

Expand All @@ -217,6 +213,10 @@ replace github.com/gogo/protobuf => github.com/gogo/protobuf v1.3.2
// allows us to specify that as an option.
replace google.golang.org/protobuf => github.com/lightninglabs/protobuf-go-hex-display v1.33.0-hex-display

// We are using a fork of the migration library with custom functionality that
// did not yet make it into the upstream repository.
replace github.com/golang-migrate/migrate/v4 => github.com/lightninglabs/migrate/v4 v4.18.2-9023d66a-fork-pr-2

Comment on lines +218 to +219
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok afaict, the big thing we added in the fork was the post migration call-backs logic.

Something id just like some clarity on before we push forward here is:
in our own rolled migration strategy in LND today that takes care of code migrations, we know that the migration version table will only be updated once the code migration is complete. Is the same true for the post-migration callback?

Copy link
Collaborator Author

@ViktorT-11 ViktorT-11 Sep 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discussed this with guggero previously.
Currently, no that is not true unfortunately, as the version table will be updated if the SQL migration succeeds, while the code migration fails. But I'm planning to address that asap 🔥!

The current plan is to submit a new PR to our migrate library fork. This PR will add a separate tracking table for code migrations. With this change, the logic will ensure that if the db is initialized and the persisted code migration version is lower than the SQL migration version, the code migration will be re-run prior to any SQL migration.

I haven't gotten to updating the migrate fork yet though. Perhaps we should hold of with merging this one until that change has been merged to the migrate fork, so that we don't need to push a new PR that updates the dependency in lnd though.

Does that sound like a good plan to you @ellemouton?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good!

// If you change this please also update docs/INSTALL.md and GO_VERSION in
// Makefile (then run `make lint` to see where else it needs to be updated as
// well).
Expand Down
Loading
Loading