Skip to content

Macaroon rotation breaks the previous default password migration, permanently locking the LND wallet #1112

New issue
New issue
Closed
#1150
Closed
Macaroon rotation breaks the previous default password migration, permanently locking the LND wallet#1112
#1150

Description

@AirHache
AirHache
opened on Sep 13, 2026

Updating to 2.4.x permanently locks the LND wallet on deployments whose walletunlock.json still holds the legacy default password. The new password is lost and the wallet cannot be unlocked again.

Reproduced twice on the same instance, eight hours apart, with a Proxmox snapshot restore in between — so this is deterministic, not a one-off.

Trigger condition

walletunlock.json contains "wallet_password":"hellorockstar", so MIGRATE_DEFAULT=1 in docker-initunlocklnd.sh.

Mechanism

lnd_unlock_entrypoint deletes every macaroon including macaroons.db, driven by LND_MACAROON_ROTATION_ID. initunlocklnd then runs the default-password migration against a wallet whose macaroon root key no longer exists.

The first POST /v1/changepassword opens the wallet — so the current password is valid — changes the passphrase, then fails with default root key not found, because the root key was deleted seconds earlier. The script treats this as a failed migration, exits 1, and never writes NEWPASS to walletunlock.json.

The second call, using WALLETPASS_BASE64_CURRENT, fails with invalid passphrase for master public key — consistent with the passphrase having already been changed by the first call. Unlocking manually with the value from the file, via lncli unlock --stdin, fails the same way.

The wallet passphrase is now 32 bytes of urandom that exist nowhere. The unlock file still holds the old, now-invalid value.

Incorrect assumption in the code

From docker-initunlocklnd.sh:

# INSTEAD of a plain unlock, not after it. The file is only rewritten
# AFTER changepassword succeeds, so a failed migration never strands
# the wallet: next start just retries.

That assumption does not hold. LND changes the wallet passphrase before touching the macaroon root key, so a failure on the second step leaves the passphrase already changed. And the rotation running immediately before guarantees that second step fails.

Impact

Wallet permanently locked. On-chain funds recoverable only from the seed; channels recoverable only via SCB with force closes. In our case a single channel had to be force-closed and the node rebuilt.

Possible fixes

Run the migration before the macaroon rotation; or persist NEWPASS to the unlock file before the call rather than after; or do not treat a root-key failure as a migration failure, since the passphrase change has already succeeded at that point.

Note for other operators

This is not a reason to skip the update — the vulnerability it fixes is far more serious. The condition is narrow, and taking a volume backup of lnd_bitcoin_datadir beforehand avoids any loss.

First occurrence — 07:30 UTC, LND 0.21.0-beta to 0.21.3-beta 
externalip=<redacted>.onion:9735 added to /data/lnd.conf
[lnd_unlock_entrypoint] Rotating macaroons (lnd-security-2026-09-v2), ALL existing macaroons are being invalidated
/data/admin.macaroon
/data/data/chain/bitcoin/mainnet/macaroons.db
/data/data/chain/bitcoin/mainnet/walletkit.macaroon
/data/data/chain/bitcoin/mainnet/invoices.macaroon
/data/data/chain/bitcoin/mainnet/router.macaroon
/data/data/chain/bitcoin/mainnet/signer.macaroon
/data/data/chain/bitcoin/mainnet/chainnotifier.macaroon
/data/readonly.macaroon
/data/invoice.macaroon
[lnd_unlock_entrypoint] Macaroons removed, lnd will regenerate them. Every client must be re-paired
[initunlocklnd] Waiting 2 seconds for lnd...
2026-09-13 07:30:34.704 [INF] LTND: Version Info rev=391889 version=0.21.3-beta commit=basedon-v0.21.3-beta-1-fresh-btcpay debuglevel=production logging=info
2026-09-13 07:30:34.721 [INF] LTND: Opening the main database, this might take a few minutes...
2026-09-13 07:30:40.527 [INF] LTND: Creating local graph and channel state DB instances
2026-09-13 07:30:40.532 [WRN] CHDB: DB version key missing, recovering from db_version=33
2026-09-13 07:30:40.532 [INF] CHDB: Checking for schema update: latest_version=35, db_version=33
2026-09-13 07:30:40.532 [INF] CHDB: Applying migration #35
2026-09-13 07:30:40.549 [INF] LTND: Database(s) now open (time_to_open=5.828767263s)!
2026-09-13 07:30:40.551 [INF] LTND: Waiting for wallet encryption password.
2026-09-13 07:30:42.392 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
[initunlocklnd] Wallet and Unlock files are present... parsing wallet password and unlocking lnd
2026-09-13 07:30:42.963 [INF] BTWL: Opened wallet
2026-09-13 07:30:45.156 [ERR] RPCS: [/lnrpc.WalletUnlocker/ChangePassword]: default root key not found
2026-09-13 07:30:45.755 [ERR] RPCS: [/lnrpc.WalletUnlocker/ChangePassword]: invalid passphrase for master public key
[initunlocklnd] WARNING: migration off the default password failed, lnd returned: {"code":2, "message":"invalid passphrase for master public key", "details":[]}
[initunlocklnd] Wallet is still locked, migration will be retried on next start
2026-09-13 07:30:46.135 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access
Second occurrence — 17:16 UTC, after restoring a pre-update snapshot 
alias=<redacted> added to /data/lnd.conf
LND_CHAIN=btc
LND_ENVIRONMENT=mainnet
Added bitcoin.active and bitcoin.mainnet to config file /data/lnd.conf
Waiting /root/.nbxplorer/btc_fully_synched to be created...
The chain is fully synched
externalip=<redacted>.onion:9735 added to /data/lnd.conf
[lnd_unlock_entrypoint] Rotating macaroons (lnd-security-2026-09-v2), ALL existing macaroons are being invalidated
/data/admin.macaroon
/data/data/chain/bitcoin/mainnet/macaroons.db
/data/data/chain/bitcoin/mainnet/walletkit.macaroon
/data/data/chain/bitcoin/mainnet/invoices.macaroon
/data/data/chain/bitcoin/mainnet/router.macaroon
/data/data/chain/bitcoin/mainnet/signer.macaroon
/data/data/chain/bitcoin/mainnet/chainnotifier.macaroon
/data/readonly.macaroon
/data/invoice.macaroon
[lnd_unlock_entrypoint] Macaroons removed, lnd will regenerate them. Every client must be re-paired
[initunlocklnd] Waiting 2 seconds for lnd...
2026-09-13 17:16:48.655 [INF] LTND: Version Info rev=391889 version=0.21.3-beta commit=basedon-v0.21.3-beta-1-fresh-btcpay debuglevel=production logging=info
2026-09-13 17:16:48.667 [INF] LTND: Opening the main database, this might take a few minutes...
2026-09-13 17:16:52.819 [INF] LTND: Creating local graph and channel state DB instances
2026-09-13 17:16:52.823 [WRN] CHDB: DB version key missing, recovering from db_version=33
2026-09-13 17:16:52.823 [INF] CHDB: Checking for schema update: latest_version=35, db_version=33
2026-09-13 17:16:52.823 [INF] CHDB: Applying migration #35
2026-09-13 17:16:52.843 [INF] LTND: Database(s) now open (time_to_open=4.175582196s)!
2026-09-13 17:16:52.845 [INF] LTND: Waiting for wallet encryption password.
2026-09-13 17:16:54.284 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
[initunlocklnd] Wallet and Unlock files are present... parsing wallet password and unlocking lnd
2026-09-13 17:16:54.829 [INF] BTWL: Opened wallet
2026-09-13 17:16:56.862 [ERR] RPCS: [/lnrpc.WalletUnlocker/ChangePassword]: default root key not found
2026-09-13 17:16:57.364 [ERR] RPCS: [/lnrpc.WalletUnlocker/ChangePassword]: invalid passphrase for master public key
[initunlocklnd] WARNING: migration off the default password failed, lnd returned: {"code":2, "message":"invalid passphrase for master public key", "details":[]}
[initunlocklnd] Wallet is still locked, migration will be retried on next start
Successful unlock on 0.21.0-beta the day before, for comparison 
2026-09-12 15:17:41.745 [INF] LTND: Version Info rev=ee13d2 version=0.21.0-beta commit=basedon-v0.21.0-beta-fresh-btcpay
2026-09-12 15:18:00.783 [INF] LTND: Waiting for wallet encryption password.
2026-09-12 15:18:01.834 [INF] BTWL: Opened wallet
2026-09-12 15:18:02.424 [INF] BTWL: The wallet has been unlocked without a time limit

cc @rockstardev

Activity

  1. AirHache
    changed the title [-]Macaroon rotation breaks the hellorockstar password migration, permanently locking the LND wallet[/-] [+]Macaroon rotation breaks the previous default password migration, permanently locking the LND wallet[/+] on Sep 13, 2026

  2. NicolasDorier commented on Sep 14, 2026

    @NicolasDorier
    NicolasDorier
    on Sep 14, 2026
    Member

    Removed macaroon rotation... we will need to communicate on it it is a critical failure from us.
    615664b

  3. NicolasDorier commented on Sep 14, 2026

    @NicolasDorier
    NicolasDorier
    on Sep 14, 2026
    Member

    @AirHache thank you for this report. We will work on an announcement as well as the recovery procedure.
    Luckily, given the channel backup and seed funds aren't lost.

    Everybody who:

    • updated between 2026-09-11T02:57:48.000Z (b4e8ce6) and 2026-09-14T04:24:57.000Z (615664b)
    • and had the last update pre-dating 2026-09-04T01:26:39.000Z (779274a)

    Got impacted by this.
    If they updated after 2026-09-04T01:26:39.000Z (779274a) but before 2026-09-11T02:57:48.000Z (b4e8ce6), then there is no problem as the password got changed before the macaroon was rotated.

  4. NicolasDorier commented on Sep 14, 2026

    @NicolasDorier
    NicolasDorier
    on Sep 14, 2026
    Member

    @AirHache also, can you DM me on X ?

  5. AirHache commented on Sep 14, 2026

    @AirHache
    AirHache
    on Sep 14, 2026
    Author

    Hi,
    DMed you this morning.

  6. rockstardev commented on Sep 16, 2026

    @rockstardev
    rockstardev
    on Sep 16, 2026
    Member

    We have a release candidate to fix this:
    btcpayserver/lnd#17 (comment)

    We'll likely need to yank current LND_MACAROON_ROTATION_ID code.

  7. NicolasDorier
    closed this as completedin #1150on Sep 21, 2026

  8. damanic commented on Oct 10, 2026

    @damanic
    damanic
    on Oct 10, 2026

    I just updated to 2.4.5 and have been hit with locked LND wallet - macaroon deleted!?

    [initunlocklnd] Password change or macaroon rotation failed: {"code":2,"message":"invalid passphrase for master public key","details":[]}
[initunlocklnd] Saved credentials were kept. Missing macaroon files or an inconsistent store require manual recovery.
2026-10-10 01:53:16.645 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access

    On further investigation it looks as though the macaroon assault occurred before the 2.4.5 update and I did not notice.

    @NicolasDorier

    We will work on an announcement as well as the recovery procedure.

    Can you point me to the recovery procedure please.

  9. NicolasDorier commented on Oct 10, 2026

    @NicolasDorier
    NicolasDorier
    on Oct 10, 2026
    Member

    @damanic it seems to be the problem yes. @rockstardev is there a recovery procedure? The user has the seed and the channel backups, so technically they should be able to recover the funds.

  10. AirHache commented on Oct 10, 2026

    @AirHache
    AirHache
    on Oct 10, 2026
    Author

    Hi,

    Standard lnd recovery guide is here: https://docs.lightning.engineering/lightning-network-tools/lnd/disaster-recovery

    I personally chose to restore the VM since I had backup and no activity on my btcpayserver.

  11. damanic commented on Oct 10, 2026

    @damanic
    damanic
    on Oct 10, 2026

    That guide says I need to set up a new node - really?

    Before I jump into it I would like to get the official BTCPAY dev advice considering it is a known issue with a BTCPAY version release that caused this 'disaster'.

  12. AirHache commented on Oct 10, 2026

    @AirHache
    AirHache
    on Oct 10, 2026
    Author

    Sure, they may propose a smoother approach.

    Anyway, if you have seed and channel.backup, you should be fine

  13. damanic commented on Oct 11, 2026

    @damanic
    damanic
    on Oct 11, 2026

    OK so my node is back up and running, remarkably no channels force closed.

    Here is how I resolved it on my docker deployment.

    All commands are presented for example only, do not copy and paste blindly. Check where your data resides, container names etc and adjust accordingly. THIS WORKED FOR ME, VERIFY IT FOR YOUR SELF.

    • Stop LND and back up the volume
      D=/var/lib/docker/volumes/generated_lnd_bitcoin_datadir/_data
  M=$D/data/chain/bitcoin/mainnet
  docker stop btcpayserver_lnd_bitcoin
  mkdir -p /root/lnd-rescue && tar czf /root/lnd-rescue/lnd-full.tgz -C $D .
  cp $M/walletunlock.json $M/channel.backup /root/lnd-rescue/

    • Get the seed; BTCPay keeps it in walletunlock.json , and you can get it from BTCPAY backend Server Settings-> Services -> LND Seed Backup

    • Create a new wallet password and save it before using it

      openssl rand -base64 24 > /root/lnd-rescue/NEW_PASSWORD
    • Move the locked wallet and old macaroon DB aside.
     mv $M/wallet.db $M/macaroons.db /root/lnd-rescue/
    • Start LND without the BTCPay init script so it can't auto-create or auto-unlock a wallet; --use-aliases is needed because lnd.conf listens on the hostname lnd_bitcoin
      . /etc/profile.d/btcpay-env.sh && cd "$BTCPAY_BASE_DIRECTORY/btcpayserver-docker"
  docker-compose -f "$BTCPAY_DOCKER_COMPOSE" run -d --no-deps --use-aliases --name lnd_rescue --entrypoint /bin/sh lnd_bitcoin -c 'ln -sfn /data /root/.lnd; exec lnd'
    • Recreate the wallet from the seed; same keys as before, new password. Enter the new password, answer y to existing seed, paste the 24 words, leave the seed passphrase empty, recovery window 2500 (default)
      docker exec -it lnd_rescue lncli create
    • Confirm it's the same node; the pubkey must match your old one, which proves the seed is right
      docker exec lnd_rescue lncli --macaroonpath=/data/admin.macaroon getinfo
    • Wait for recovery_finished: true; LND is finding your on-chain funds. Could take hours.
      docker exec lnd_rescue lncli --macaroonpath=/data/admin.macaroon getrecoveryinfo
    • Stop the rescue container cleanly so it never runs alongside the BTCPay container
      docker stop -t 60 lnd_rescue && docker rm lnd_rescue
    • Write the new password into walletunlock.json
      jq --arg p "$(cat /root/lnd-rescue/NEW_PASSWORD)" '.wallet_password=$p' /root/lnd-rescue/walletunlock.json > $M/walletunlock.json
    • Start the BTCPay container to bring LND back under BTCPay
      docker start btcpayserver_lnd_bitcoin
    • Never restore the old channel.db from the backup tarball, it's out of date once the node is running again.

  14. NicolasDorier commented on Oct 11, 2026

    @NicolasDorier
    NicolasDorier
    on Oct 11, 2026
    Member

    @damanic thank you for documenting this. I am sorry about this bug which could really have made people lose their money... we got lucky @AirHache catched it fast enough so not many people fell in this issue. We will communicate on that in case some people like you didn't see that their nodes were offline.

  15. damanic commented on Oct 11, 2026

    @damanic
    damanic
    on Oct 11, 2026

    @NicolasDorier
    How does BTCPAY communicate issues like this outside of github issues?

    I think this comment should have moved quickly to a public announcement post with steps to resolve.

    Perhaps there should also be an option in BTCPAY for users to enable 'blog' notifications for 'announcements' or other categories. Once enabled they are revealed through the 'bell' icon in the same way as new release notifications.

  16. rockstardev commented on Oct 12, 2026

    @rockstardev
    rockstardev
    on Oct 12, 2026
    Member

    Thanks @damanic for writing this up, it is the right approach. I reproduced the locked wallet on regtest with the LND image BTCPay Server ships and tested your recovery end to end: the channel came back with its balance, the on-chain funds were recovered, and payments worked both ways.

    One addition for anyone following it: move the old macaroon files aside too (/data/*.macaroon and /data/data/chain/bitcoin/mainnet/*.macaroon), together with wallet.db and macaroons.db. LND only writes new macaroon files when none exist, and the old ones fail against the rebuilt wallet with verification failed: signature mismatch, so BTCPay Server cannot connect to LND. Your node may not have needed this if those files were already gone.

    We are adding a step-by-step guide to the docs, based on your steps: btcpayserver/btcpayserver-doc#1649

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions