Skip to content

wallet+db: normalize account address identity#1268

Open
yyforyongyu wants to merge 20 commits into
sql-walletfrom
fix-import-schema
Open

wallet+db: normalize account address identity#1268
yyforyongyu wants to merge 20 commits into
sql-walletfrom
fix-import-schema

Conversation

@yyforyongyu

@yyforyongyu yyforyongyu commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #1267.

Introduces normalized account and address identity as the base layer for the
runtime-store stack, with ADR coverage and separate SQL, kvdb, wallet, and
integration-test commits split for reviewability.

Highlights:

  • Add ADR 0013 for normalized account/address identity.
  • Normalize account, address, and UTXO identity so imported accounts and
    imported-xpub children do not fabricate wallet-derived account numbers.
  • Update SQL migrations, queries, sqlc output, and SQL adapters for the
    normalized schema.
  • Update kvdb adapters to expose the same identity semantics while preserving
    legacy waddrmgr behavior.
  • Update wallet account, address, PSBT, signer, tx-creation, and UTXO
    consumers.
  • Add regression coverage for account, address, and UTXO identity, including
    legacy kvdb mixed watch-only spendability.

Validation:

  • make fmt
  • make lint
  • go test -run '^$' ./wallet/internal/db/... ./wallet
  • go test ./wallet/internal/db/kvdb
  • Focused wallet and DB integration regression tests

Note: this PR follows the Task 291 review split, so branch-tip validation is
used instead of requiring every individual review commit to be independently
buildable.

@coveralls

coveralls commented Jun 15, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 28394691252

Warning

No base build found for commit 90eaacc on sql-wallet.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 54.582%

Details

  • Patch coverage: 463 uncovered changes across 22 files (1135 of 1598 lines covered, 71.03%).

Uncovered Changes

Top 10 Files by Coverage Impact Changed Covered %
wallet/internal/db/kvdb/addressstore.go 185 113 61.08%
wallet/internal/sql/pg/sqlc/addresses.sql.go 67 0 0.0%
wallet/internal/sql/sqlite/sqlc/addresses.sql.go 67 18 26.87%
wallet/internal/db/addresses_common.go 204 158 77.45%
wallet/internal/db/pg/addresses.go 241 213 88.38%
wallet/internal/db/sqlite/addresses.go 231 205 88.74%
wallet/internal/db/utxo_store_common.go 48 27 56.25%
wallet/tx_creator.go 101 82 81.19%
wallet/internal/db/accountstore_accountinfo.go 58 40 68.97%
wallet/address_manager.go 59 42 71.19%
Total (38 files) 1598 1135 71.03%

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 54848
Covered Lines: 29937
Line Coverage: 54.58%
Coverage Strength: 11083.69 hits per line

💛 - Coveralls

Comment thread wallet/internal/sql/pg/queries/accounts.sql Outdated
Comment thread wallet/internal/sql/pg/migrations/000005_accounts.up.sql Outdated
@yyforyongyu yyforyongyu self-assigned this Jun 15, 2026
@yyforyongyu yyforyongyu added this to the Introduce SQL store milestone Jun 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces the “normalized account and address identity” base layer, restructuring SQL persistence so account/address identity lives in parent tables (accounts, addresses) with subtype tables (derived_accounts, derived_addresses) and updating the Go store/runtime stack to consume the new shape (including removing legacy AccountOrigin / “imported bucket” assumptions).

Changes:

  • Normalizes SQL account/address schema (parent identity tables + derived subtype tables) and updates SQLite/Postgres queries, sqlc models, and migrations accordingly.
  • Updates wallet/internal/db domain types to make account numbers optional, remove AccountOrigin, and add read-time shape/corruption validation for UTXO/address/account joins.
  • Cuts wallet runtime logic/tests over to the new identity semantics (UTXO spendability rules, change source selection, address/account metadata mapping) and adjusts kvdb adapter behavior/tests.

Reviewed changes

Copilot reviewed 108 out of 108 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
wallet/utxo_manager.go Updates UTXO listing/get logic to use wallet sync height and new spendability semantics.
wallet/utxo_manager_test.go Updates UTXO tests for removed origin field; adds imported spendability test.
wallet/tx_creator.go Extends policy validation and source handling for value vs pointer coin sources; adjusts account filtering path.
wallet/test_helpers_test.go Adds helper for *uint32 test assertions.
wallet/signer_test.go Updates signer test fixtures for new address identity fields.
wallet/psbt_manager_test.go Updates PSBT tests to use derivation-path presence / imported flags and optional account number.
wallet/internal/sql/sqlite/sqlc/transactions.sql.go Regenerated sqlc output for rollback rewind and owned prevout queries.
wallet/internal/sql/sqlite/sqlc/models.go Updates sqlc models to match normalized identity schema.
wallet/internal/sql/sqlite/sqlc/db.go Updates prepared statements to match new/renamed queries.
wallet/internal/sql/sqlite/queries/utxos.sql Rewrites UTXO queries to join via normalized address/account identity and add shape constraints.
wallet/internal/sql/sqlite/queries/transactions.sql Adjusts owned-prevout queries and rollback sync-state rewind to sparse-block semantics.
wallet/internal/sql/sqlite/queries/addresses.sql Reworks address queries/inserts for normalized identity and adds raw-import listing query.
wallet/internal/sql/sqlite/migrations/000008_utxos.up.sql Updates wallet-ownership enforcement to use addresses.wallet_id directly.
wallet/internal/sql/sqlite/migrations/000006_addresses.up.sql Normalizes addresses schema; adds derived_addresses table, triggers, and indexes.
wallet/internal/sql/sqlite/migrations/000006_addresses.down.sql Updates down migration for new triggers/tables.
wallet/internal/sql/sqlite/migrations/000005_accounts.up.sql Normalizes accounts schema; adds derived_accounts table and identity immutability triggers.
wallet/internal/sql/sqlite/migrations/000005_accounts.down.sql Updates down migration for new triggers/tables.
wallet/internal/sql/pg/sqlc/transactions.sql.go Regenerated sqlc output for rollback rewind and owned prevout queries.
wallet/internal/sql/pg/sqlc/models.go Updates sqlc models to match normalized identity schema.
wallet/internal/sql/pg/sqlc/db.go Updates prepared statements to match new/renamed queries.
wallet/internal/sql/pg/queries/utxos.sql Rewrites UTXO queries to join via normalized address/account identity and add shape constraints.
wallet/internal/sql/pg/queries/transactions.sql Adjusts owned-prevout queries and rollback sync-state rewind to sparse-block semantics.
wallet/internal/sql/pg/queries/addresses.sql Reworks address queries/inserts for normalized identity and adds raw-import listing query.
wallet/internal/sql/pg/migrations/000008_utxos.up.sql Updates wallet-ownership enforcement to use addresses.wallet_id directly.
wallet/internal/sql/pg/migrations/000006_addresses.down.sql Updates down migration for new triggers/tables/functions.
wallet/internal/sql/pg/migrations/000005_accounts.up.sql Normalizes accounts schema; adds derived_accounts and identity immutability triggers/functions.
wallet/internal/sql/pg/migrations/000005_accounts.down.sql Updates down migration for new triggers/tables/functions.
wallet/internal/db/utxo_store_common.go Adds UTXO address/account shape metadata + validation helpers.
wallet/internal/db/test_helpers_test.go Adds helper for *uint32 assertions in db package tests.
wallet/internal/db/sqlite/utxostore_listutxos.go Adds read-time UTXO shape validation; removes origin enrichment.
wallet/internal/db/sqlite/utxostore_getutxo.go Adds read-time UTXO shape validation; removes origin enrichment.
wallet/internal/db/sqlite/txstore_rollback.go Updates rollback rewind call signature (computed rewind height).
wallet/internal/db/sqlite/block.go Adds post-insert block consistency check.
wallet/internal/db/sqlite/accountstore_renameaccount.go Updates numeric rename params handling for new account number semantics.
wallet/internal/db/sqlite/accountstore_helpers.go Updates account row mapping to new IsDerived shape fields.
wallet/internal/db/sqlite/accountstore_getaccount.go Updates numeric get params handling for optional account number.
wallet/internal/db/sqlite/accountstore_createimported.go Updates imported account creation/mapping to new schema.
wallet/internal/db/sqlite/accountstore_createderived.go Splits derived account insert from derived account number insert.
wallet/internal/db/pg/utxostore_listutxos.go Adds read-time UTXO shape validation; removes origin enrichment.
wallet/internal/db/pg/utxostore_getutxo.go Adds read-time UTXO shape validation; removes origin enrichment.
wallet/internal/db/pg/txstore_rollback.go Updates rollback rewind call signature (computed rewind height).
wallet/internal/db/pg/block.go Adds post-insert block consistency check.
wallet/internal/db/pg/accountstore_renameaccount.go Updates numeric rename params handling for new account number semantics.
wallet/internal/db/pg/accountstore_helpers.go Updates account row mapping to new IsDerived shape fields.
wallet/internal/db/pg/accountstore_getaccount.go Updates numeric get params handling for optional account number.
wallet/internal/db/pg/accountstore_createimported.go Updates imported account creation/mapping to new schema.
wallet/internal/db/pg/accountstore_createderived.go Splits derived account insert from derived account number insert.
wallet/internal/db/kvdb/walletstore.go Populates wallet-level IsWatchOnly in kvdb adapter.
wallet/internal/db/kvdb/walletstore_test.go Adds watch-only reporting coverage for kvdb wallet store.
wallet/internal/db/kvdb/utxostore.go Adjusts legacy UTXO account resolution to check all extracted script addresses; removes origin.
wallet/internal/db/kvdb/utxostore_test.go Updates kvdb UTXO tests for removed origin; adds multisig member scan coverage.
wallet/internal/db/kvdb/utxostore_balance_test.go Adds balance coverage for multisig member scan behavior.
wallet/internal/db/kvdb/txstore.go Adds addrStore nil-guard for credited tx creation; implements empty-label clearing for legacy labels bucket.
wallet/internal/db/kvdb/txstore_test.go Adds tests for missing addrStore credit path and empty-label clearing semantics.
wallet/internal/db/kvdb/addressstore_test.go Updates address store tests for optional account number and IsImported/HasDerivationPath.
wallet/internal/db/kvdb/accountstore_test.go Updates account store tests for optional account number and IsImported.
wallet/internal/db/kvdb/accountstore_helpers.go Updates balance attachment semantics for imported accounts with nil account number.
wallet/internal/db/kvdb/accountstore_getaccount.go Updates get-account logic for imported accounts exposing nil account number.
wallet/internal/db/kvdb/accountstore_fingerprints_test.go Updates fingerprint tests for optional account number.
wallet/internal/db/kvdb/accountstore_createimported.go Adds address-schema conversion validation and proper wallet address type mapping.
wallet/internal/db/itest/tx_store_corruption_test.go Adjusts rollback corruption expectation wording/behavior.
wallet/internal/db/itest/sqlite_identity_test.go Adds SQLite rowid-alias immutability tests for accounts/addresses.
wallet/internal/db/itest/fixtures_common_test.go Updates fixture account cases to use IsImported instead of origin enum.
wallet/internal/db/itest/accountstore_renameaccount_test.go Updates rename tests for optional account number and imported-xpub rename semantics.
wallet/internal/db/itest/accountstore_listaccounts_test.go Updates ordering/assertions for IsImported and optional account numbers.
wallet/internal/db/itest/accountstore_helpers_test.go Centralizes common test account names and adds helper to require account number.
wallet/internal/db/itest/accountstore_getaccount_test.go Updates get-account tests for optional account numbers and adds corruption exclusion coverage.
wallet/internal/db/itest/accountstore_createinvariants_test.go Adds immutability tests for account IDs and derived child rows; updates reserved alias test naming.
wallet/internal/db/itest/accountstore_createderived_test.go Updates derived account allocation tests for optional account number pointers.
wallet/internal/db/interface.go Updates reserved-name doc to reflect alias-only semantics (no stored SQL bucket row).
wallet/internal/db/data_types.go Removes AccountOrigin, makes account/address numbers optional, adds account/address identity fields and derivation-path flags, removes UTXO origin.
wallet/internal/db/accountstore_listaccounts_test.go Updates db-layer list accounts tests for optional account number pointers.
wallet/internal/db/accountstore_getaccount.go Updates imported-by-number rejection to use IsImported flag.
wallet/internal/db/accountstore_getaccount_test.go Updates db-layer get-account tests for optional account number pointers.
wallet/internal/db/accountstore_createimported_test.go Updates imported account creation tests for IsImported semantics.
wallet/internal/db/accountstore_createderived.go Adds derived-account number conversion helper and includes SQL account ID in returned info.
wallet/internal/db/accountstore_createderived_test.go Updates derived account creation tests for optional account number pointers.
wallet/internal/db/accountstore_accountinfo.go Refactors account info mapping around IsDerived + nullable account numbers; adds shape validation helpers.
wallet/internal/db/accountstore_accountinfo_test.go Updates account info mapping tests for IsDerived shape field.
wallet/address_manager.go Updates store-backed address mapping to use HasDerivationPath/optional account number and improves error message formatting.
wallet/address_manager_test.go Updates tests for new store address fields; adds imported-xpub unused-address test case.
wallet/address_info_test.go Adds coverage for store-backed address mapping vs imported-xpub/raw-import shape.
wallet/account_manager.go Updates derived/imported handling for optional account numbers and adds nil-key validation.
wallet/account_manager_test.go Updates tests for optional account number and adds nil extended pubkey validation test.
docs/developer/adr/README.md Adds ADR 0012 and ADR 0013 links.
docs/developer/adr/0013-normalized-account-address-identity.md New ADR documenting normalized identity schema and rationale.
docs/developer/adr/0012-wallet-level-watch-only-uniformity.md Updates ADR 0012 to reflect alias-only raw imports per ADR 0013.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread wallet/utxo_manager.go Outdated
Comment thread wallet/internal/sql/sqlite/queries/addresses.sql
Comment thread wallet/internal/sql/pg/queries/addresses.sql
@yyforyongyu yyforyongyu force-pushed the fix-import-schema branch 2 times, most recently from bbe49f1 to 38c86ce Compare June 15, 2026 14:04
@saubyk saubyk added this to lnd v0.22 Jun 15, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in lnd v0.22 Jun 15, 2026
Comment thread wallet/internal/db/pg/block.go
Comment thread wallet/internal/db/kvdb/walletstore.go
Comment thread wallet/internal/db/pg/txstore_rollback.go
Comment thread wallet/account_manager.go
Comment thread wallet/utxo_manager.go Outdated
Comment thread wallet/tx_creator.go
Comment thread wallet/internal/sql/pg/migrations/000006_addresses.up.sql Outdated
Comment thread wallet/internal/sql/pg/migrations/000006_addresses.up.sql Outdated
Comment thread wallet/internal/sql/sqlite/migrations/000006_addresses.up.sql Outdated
Comment thread wallet/internal/sql/sqlite/migrations/000006_addresses.up.sql Outdated
Comment thread wallet/internal/db/kvdb/accountstore_createimported.go Outdated
@yyforyongyu

Copy link
Copy Markdown
Collaborator Author

No issues found by gpt-5.5 🧭

Comment thread wallet/utxo_manager.go
@yyforyongyu yyforyongyu force-pushed the fix-import-schema branch 2 times, most recently from 677475b to 69ff2ee Compare June 17, 2026 15:23
Comment thread docs/developer/adr/0013-normalized-account-address-identity.md
Comment thread docs/developer/adr/0013-normalized-account-address-identity.md
Comment thread docs/developer/adr/0013-normalized-account-address-identity.md Outdated
Comment thread wallet/internal/db/test_helpers_test.go Outdated
Comment thread wallet/test_helpers_test.go Outdated
Comment thread wallet/internal/db/sqlite/accountstore_helpers.go Outdated
Comment thread wallet/internal/db/pg/accountstore_helpers.go Outdated
Comment thread wallet/internal/db/kvdb/accountstore_createimported.go Outdated
Comment thread wallet/internal/db/kvdb/addressstore.go
Comment thread wallet/internal/db/kvdb/txstore.go Outdated
@yyforyongyu

Copy link
Copy Markdown
Collaborator Author

@yyforyongyu do you think this is ready for review now?

Not yet, let me review first, will request when done!

@Abdulkbk

Copy link
Copy Markdown
Contributor

Not yet, let me review first, will request when done!

ping me too, started reviewing but paused because of the rebases.

@yyforyongyu

Copy link
Copy Markdown
Collaborator Author

No issues found by claude-opus-4-8[1m] 🧬

@yyforyongyu

Copy link
Copy Markdown
Collaborator Author

No issues found by claude-opus-4-8[1m] 🧩

@yyforyongyu

Copy link
Copy Markdown
Collaborator Author

No issues found by gpt-5.5 🧭

Record the move to a derived-marker/import-flag identity shape.
Add the account derived marker and import flag; route account reads/writes through it.
Atomic: account number becomes optional and every account consumer moves with the type change.
Add the address derived marker and import/path flags and consume them.
Additive schema for derived-address identity; no reads yet.
Transitional: write derived-address rows alongside the legacy shape until readers switch.
Read addresses through the derived-identity shape.
List addresses without requiring an owning account.
Stop emitting the imported-address scope from the wallet.
Drop the imported-address scope from the store.
Make utxos accountless under the normalized identity.
Key utxo spendability off derived identity.
Thread derived-address identity through the wallet.
Remove the now-unused account/address/utxo origin lookups, superseded by derived identity.
Delete unused origin columns and legacy account counters.
Atomic: the sqlc schema rename forces the SQL regen plus the address and account consumers into one unit -- legacy and new queries cannot coexist against the changed schema.
Utxo reads the re-aliased column; a separable refactor once the schema has settled.
Cover account identity invariants.
Cover address identity invariants.
Cover tx/utxo identity invariants.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants