litd + tapd fails to start: `could not start macaroon service: unable to derive a shared secret with LND: rpc error: code = DeadlineExceeded desc = context deadline exceeded`

[bufo24]

When starting litd, with integrated lnd and tapd. Both litd and tapd fails to start with this error:

2026-08-20 14:41:19.485 [ERR] STAT: could not start the lit sub-server: could not start Lit: could not start litd sub-servers: could not start macaroon service: unable to derive a shared secret with LND: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-08-20 14:41:19.485 [DBG] STAT: Setting the lit sub-server as errored: could not start Lit: could not start litd sub-servers: could not start macaroon service: unable to derive a shared secret with LND: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-08-20 14:41:19.394 [ERR] STAT: could not start the taproot-assets sub-server: received critical error from sub-server (taproot-assets), shutting down: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [DBG] STAT: Setting the taproot-assets sub-server as errored: received critical error from sub-server (taproot-assets), shutting down: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [ERR] GRDN: Error in custodian: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [ERR] GRDN: Aborting main custodian event loop: main event loop: rpc error: code = Canceled desc = context canceled

System information

I'm using litd v0.17.2, but it also happend on v0.17.0. Running on a neutrino backend.

Full startup logs of a filtered selections of services:
Let me know if additional logs are needed.

2026-08-20 14:46:19.732 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:45:19.730 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:44:19.730 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:43:19.735 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:42:19.731 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:42:02.223 [INF] INVC: New invoice subscription client: id=1
2026-08-20 14:41:19.730 [DBG] HLCK: Health check: chain backend invoking success callback
2026-08-20 14:41:19.485 [ERR] STAT: could not start the lit sub-server: could not start Lit: could not start litd sub-servers: could not start macaroon service: unable to derive a shared secret with LND: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-08-20 14:41:19.485 [DBG] STAT: Setting the lit sub-server as errored: could not start Lit: could not start litd sub-servers: could not start macaroon service: unable to derive a shared secret with LND: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-08-20 14:41:19.394 [ERR] STAT: could not start the taproot-assets sub-server: received critical error from sub-server (taproot-assets), shutting down: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [DBG] STAT: Setting the taproot-assets sub-server as errored: received critical error from sub-server (taproot-assets), shutting down: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [ERR] GRDN: Error in custodian: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:41:19.394 [ERR] GRDN: Aborting main custodian event loop: main event loop: rpc error: code = Canceled desc = context canceled
2026-08-20 14:40:49.481 [INF] AMBX: Authmailbox cleanup loop started, interval=24h0m0s
2026-08-20 14:40:49.481 [INF] LITD: Starting LiT macaroon service
2026-08-20 14:40:49.481 [DBG] STAT: Setting the taproot-assets sub-server as running
2026-08-20 14:40:49.481 [INF] AMBX: Starting authmailbox RPC Server
2026-08-20 14:40:49.481 [INF] TRPC: Starting Taproot Assets RPC Server
2026-08-20 14:40:49.481 [INF] TCHN: Starting AuxSweeper
2026-08-20 14:40:49.481 [INF] TCHN: Starting aux chan closer
2026-08-20 14:40:49.481 [INF] TCHN: Starting aux invoice manager
2026-08-20 14:40:49.481 [INF] TCHN: Starting aux traffic shaper
2026-08-20 14:40:49.481 [INF] TCHN: Starting FundingController
2026-08-20 14:40:49.481 [INF] TCHN: Starting aux leaf signer
2026-08-20 14:40:49.481 [INF] SUCO: Supply verifier manager started successfully
2026-08-20 14:40:49.477 [INF] RFQS: Starting RFQ manager main event loop
2026-08-20 14:40:49.477 [INF] SUCO: Initializing supply verifier state machines
2026-08-20 14:40:49.477 [INF] SUCO: Starting supply verifier manager
2026-08-20 14:40:49.477 [INF] RFQS: Starting subsystem: negotiator
2026-08-20 14:40:49.471 [INF] RFQS: Initializing RFQ subsystems
2026-08-20 14:40:49.471 [INF] UNIV: Obtained 0 roots from remote Universe server
2026-08-20 14:40:49.471 [INF] UNIV: Fetching 0 roots
2026-08-20 14:40:49.471 [INF] TSVR: Connecting to Universe server at universe.lightning.finance:10029
2026-08-20 14:40:49.471 [INF] UNIV: Attempting to sync universe: host=universe.lightning.finance:10029, sync_type=full, ids=([]universe.Identifier) <nil>
2026-08-20 14:40:49.471 [INF] UNIV: Syncing Universe state with server=universe.lightning.finance:10029
}
}
addr: (net.Addr) <nil>
addrStr: (string) (len=32) ""universe.lightning.finance:10029"",
ID: (int64) 0,
(universe.ServerAddr) {
2026-08-20 14:40:49.462 [INF] UNIV: Adding new Universe server to Federation, addrs=([]universe.ServerAddr) (len=1 cap=1) {
2026-08-20 14:40:49.430 [INF] TSVR: Connecting to Universe server at universe.lightning.finance:10029
2026-08-20 14:40:49.423 [INF] UNIV: Starting FederationEnvoy
2026-08-20 14:40:49.423 [INF] FRTR: No pending parcels to resume
2026-08-20 14:40:49.418 [INF] FRTR: Starting ChainPorter
2026-08-20 14:40:49.395 [INF] GRDN: New block at height 963319
2026-08-20 14:40:49.395 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:49.394 [INF] GRDN: Loading wallet transactions starting at block height 0
2026-08-20 14:40:49.394 [INF] GRDN: Resuming 0 pending inbound asset events
2026-08-20 14:40:49.390 [INF] GRDN: Fetching last v2 address event height
2026-08-20 14:40:49.369 [INF] GRDN: Starting re-org watcher
2026-08-20 14:40:49.361 [INF] GRDN: Loading pending inbound asset events
2026-08-20 14:40:49.361 [INF] GRDN: Gardener for ChainPlanter now active!
2026-08-20 14:40:49.361 [INF] GRDN: Starting asset custodian
2026-08-20 14:40:49.361 [INF] GRDN: Retrieved 0 non-finalized batches from DB
2026-08-20 14:40:49.357 [INF] GRDN: Starting ChainPlanter
2026-08-20 14:40:49.341 [INF] TRPC: Validating RPC requests based on macaroon at: /data/.tapd/data/mainnet/admin.macaroon
2026-08-20 14:40:49.341 [INF] TSVR: Version: 0.8.1-alpha commit=v0.8.1, build=production, logging=default, debuglevel=info, active_network=mainnet
2026-08-20 14:40:49.340 [INF] SSVR: Configuring universe.lightning.finance:10029 as initial Universe federation server
})
RootNodePageCacheSize: (uint64) 10240
SyncerCachePreAllocSize: (uint64) 100000,
SyncerCacheEnabled: (bool) false,
LeavesPerUniverse: (uint64) 50,
LeavesNumCachedUniverses: (uint64) 2000,
MaxProofCacheSize: (string) (len=4) ""32MB"",
2026-08-20 14:40:49.340 [DBG] SSVR: multiverse_cache=(*tapdb.MultiverseCacheConfig)(0xc0007adf40)({
2026-08-20 14:40:49.340 [INF] TADB: Database version after migration: 60
2026-08-20 14:40:49.339 [DBG] TADB: Finished 60/u asset_transfers_superseded (read 29.892µs, ran 56.22793ms)
2026-08-20 14:40:49.296 [DBG] TADB: Read and execute 60/u asset_transfers_superseded
2026-08-20 14:40:49.282 [DBG] TADB: Start buffering 60/u asset_transfers_superseded
2026-08-20 14:40:49.280 [INF] TADB: Attempting to apply migration(s) (current_db_version=59, latest_migration_version=60)
2026-08-20 14:40:49.254 [INF] TADB: Using SQL database 'postgres://xxx
2026-08-20 14:40:49.254 [INF] SSVR: Opening postgres database at: postgres://xxx
2026-08-20 14:40:49.254 [DBG] STAT: Setting the lnd sub-server as running
2026-08-20 14:40:49.247 [INF] LITD: Baking internal super macaroon
2026-08-20 14:40:49.247 [INF] NTFN: Cancelling epoch notification, epoch_id=10
2026-08-20 14:40:49.247 [INF] LITD: Full lnd client connected
2026-08-20 14:40:49.247 [INF] LNDC: Chain notifier RPC is ready
2026-08-20 14:40:49.247 [INF] LNDC: RegisterBlockEpochNtfn returned a height. Success!
2026-08-20 14:40:49.247 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:49.229 [INF] LNDC: Trying to make RegisterBlockEpochNtfn and receive the current height...
2026-08-20 14:40:49.229 [INF] LNDC: Waiting for chain notifier RPC to be ready
2026-08-20 14:40:49.229 [INF] LNDC: lnd is now fully synced to its chain backend
2026-08-20 14:40:49.228 [INF] LNWL: Web API returning 1012 sat/kvb for min relay feerate
2026-08-20 14:40:49.161 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:49.161 [INF] CHIO: BlockbeatDispatcher is starting with 1 consumer queues
2026-08-20 14:40:43.986 [INF] CHFT: Adding 159 channels to event store
2026-08-20 14:40:41.133 [INF] CHFT: ChannelEventStore starting...
2026-08-20 14:40:37.634 [INF] GRDB: Finished populating in-memory channel graph (took 655.07535ms, num_node_features=13385, num_nodes=13385, num_channels=83252)
2026-08-20 14:40:37.044 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:37.044 [INF] NANN: Channel Status Manager starting
2026-08-20 14:40:37.036 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:37.035 [INF] INVC: InvoiceRegistry starting...
2026-08-20 14:40:37.014 [INF] SWPR: Registered sweep request at block 963319: out_point=f5f3f1c6da7e03cee299c005fc45b8e203114f4b75265c567d32418e5910415b:1, witness_type=CommitmentTimeLock, amount=0.13947104 BTC, deadline=965699, state=Init, params=(startingFeeRate={false 0}, immediate=false, exclusive_group=none, budget=0.06973552 BTC, deadline=none)
2026-08-20 14:40:37.013 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:37.011 [INF] SWPR: Registered sweep request at block 963319: out_point=f5f3f1c6da7e03cee299c005fc45b8e203114f4b75265c567d32418e5910415b:0, witness_type=CommitmentAnchor, amount=0.00000330 BTC, deadline=964327, state=Init, params=(startingFeeRate={false 0}, immediate=false, exclusive_group=1052177652354646017, budget=0.00000330 BTC, deadline=none)
2026-08-20 14:40:37.003 [INF] CRTR: Channel Router starting
2026-08-20 14:40:36.978 [INF] GRDB: Populating in-memory channel graph, this might take a while...
2026-08-20 14:40:30.799 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:30.799 [INF] INVC: HtlcModificationInterceptor starting...
2026-08-20 14:40:24.386 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:20.572 [INF] BRAR: Starting contract observer, watching for breaches.
2026-08-20 14:40:20.572 [INF] FNDG: Funding manager starting
2026-08-20 14:40:20.152 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:20.152 [INF] BRAR: Breach arbiter starting
2026-08-20 14:40:19.767 [INF] BTCN: Registering block subscription: id=4
2026-08-20 14:40:19.730 [INF] UTXN: UTXO nursery starting
2026-08-20 14:40:19.730 [INF] LNWL: Web API returning 1012 sat/kvb for min relay feerate
2026-08-20 14:40:19.730 [INF] SWPR: Sweeper starting
2026-08-20 14:40:19.730 [INF] SWPR: TxPublisher starting...
2026-08-20 14:40:19.730 [INF] NTFN: Cancelling epoch notification, epoch_id=2
2026-08-20 14:40:19.730 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:19.729 [INF] PRNF: PeerNotifier starting
2026-08-20 14:40:19.729 [INF] CHNF: ChannelNotifier starting
2026-08-20 14:40:19.729 [INF] NTFN: New block epoch subscription
2026-08-20 14:40:19.716 [INF] NTFN: neutrino notifier starting...
2026-08-20 14:40:19.716 [INF] LNWL: SigPool starting
2026-08-20 14:40:19.716 [DBG] HLCK: Monitoring: chain backend
2026-08-20 14:40:19.716 [WRN] HLCK: check: tls configured with 0 attempts, skipping it
2026-08-20 14:40:19.716 [WRN] HLCK: check: disk space configured with 0 attempts, skipping it
2026-08-20 14:40:12.372 [INF] LTND: Chain backend is fully synced! rev=6b90de end_height=963319
2026-08-20 14:40:12.364 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=true
2026-08-20 14:40:12.172 [INF] BTWL: Finished rescan for 2430 addresses (synced to block 00000000000000000000d4d225070c76cce2148e82173fafd12a0436b6102393, height 963319)
2026-08-20 14:40:12.172 [INF] BTWL: Done catching up block hashes
2026-08-20 14:40:11.947 [INF] BTCN: Registering block subscription: id=3
2026-08-20 14:40:11.905 [INF] BTWL: Catching up block hashes to height 963319, this might take a while
2026-08-20 14:40:11.893 [INF] BTCN: Canceling block subscription: id=2
2026-08-20 14:40:11.893 [INF] BTWL: Started rescan from block 00000000000000000000d4d225070c76cce2148e82173fafd12a0436b6102393 (height 963319) for 2430 addrs, 8 outpoints"
2026-08-20 14:40:11.352 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:10.336 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:09.314 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:08.298 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:07.278 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:06.235 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:05.219 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:04.200 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:03.141 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:02.005 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:40:00.838 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:59.510 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:58.494 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:57.478 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:56.449 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:55.433 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:54.418 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:53.404 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:52.384 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:51.365 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:50.348 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:49.331 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:48.313 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:47.298 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:46.282 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:45.268 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:44.247 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:43.231 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:42.215 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:41.200 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:40.183 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:39.169 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:38.154 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:37.135 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:36.119 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:35.104 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:34.089 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:33.075 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:32.289 [INF] LNDC: Waiting for lnd to be fully synced to its chain backend, this might take a while
2026-08-20 14:39:32.289 [INF] LNDC: Using network mainnet
2026-08-20 14:39:32.289 [INF] LNDC: lnd version: v0.21.2-beta, build tags 'litd,autopilotrpc,signrpc,walletrpc,chainrpc,invoicesrpc,watchtowerrpc,neutrinorpc,monitoring,peersrpc,kvdb_postgres,kvdb_etcd,kvdb_sqlite'
2026-08-20 14:39:32.060 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:31.046 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:30.031 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:29.018 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:28.003 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:26.989 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:25.974 [DBG] LTND: Syncing to block chain rev=6b90de best_block_time=""2026-08-20 14:25:05 +0000 UTC"" is_synced=false
2026-08-20 14:39:25.960 [INF] LTND: Waiting for chain backend to finish sync rev=6b90de start_height=963319
2026-08-20 14:39:25.945 [INF] LNDC: Wallet state of lnd is now: Lnd RPC server is ready for requests
2026-08-20 14:39:25.945 [INF] LTND: We're not running within systemd or the service type is not 'notify'
2026-08-20 14:39:25.942 [INF] RRPC: Making macaroons for Router RPC Server at: /data/.lnd/data/chain/bitcoin/mainnet/router.macaroon
2026-08-20 14:39:25.938 [INF] NTFR: Baking macaroons for ChainNotifier RPC Server at: /data/.lnd/data/chain/bitcoin/mainnet/chainnotifier.macaroon
2026-08-20 14:39:25.931 [INF] IRPC: Baking macaroons for invoices RPC Server at: /data/.lnd/data/chain/bitcoin/mainnet/invoices.macaroon
2026-08-20 14:39:25.928 [INF] WLKT: Baking macaroons for WalletKit RPC Server at: /data/.lnd/data/chain/bitcoin/mainnet/walletkit.macaroon
2026-08-20 14:39:25.923 [INF] SGNR: Making macaroons for Signer RPC Server at: /data/.lnd/data/chain/bitcoin/mainnet/signer.macaroon
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: DevRPC
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: WatchtowerClientRPC
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: WatchtowerRPC
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: RouterRPC
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: NeutrinoKitRPC
2026-08-20 14:39:25.923 [DBG] LTND: Populating dependencies for sub RPC server: PeersRPC
2026-08-20 14:39:25.922 [DBG] LTND: Populating dependencies for sub RPC server: InvoicesRPC
2026-08-20 14:39:25.922 [DBG] LTND: Populating dependencies for sub RPC server: ChainRPC
2026-08-20 14:39:25.922 [DBG] LTND: Populating dependencies for sub RPC server: AutopilotRPC
2026-08-20 14:39:25.922 [DBG] LTND: Populating dependencies for sub RPC server: WalletKitRPC
2026-08-20 14:39:25.922 [DBG] LTND: Populating dependencies for sub RPC server: SignRPC
2026-08-20 14:39:25.917 [INF] ATPL: Instantiating autopilot with active=false, max_channels=5, allocation=0.600000, min_chan_size=20000, max_chan_size=16777215, private=false, min_confs=1, conf_target=3
2026-08-20 14:39:25.917 [INF] CHIO: Registered queue=1 with 3 blockbeat consumers
2026-08-20 14:39:25.917 [INF] LTND: Channel backup proxy channel notifier starting
2026-08-20 14:39:21.330 [INF] ACSM: Access Manager initialized
2026-08-20 14:39:08.848 [INF] CRTR: Loaded 0 mission control entries
2026-08-20 14:39:04.263 [INF] BTCN: Registering block subscription: id=2
2026-08-20 14:39:03.162 [INF] CHRE: LightningWallet opened
2026-08-20 14:39:02.424 [INF] BTWL: The wallet has been unlocked without a time limit
2026-08-20 14:39:01.677 [INF] LNWL: Web API fee estimator using update timeout of 14m26s
2026-08-20 14:39:01.602 [INF] LNDC: Wallet state of lnd is now: Wallet is unlocked
2026-08-20 14:39:01.602 [INF] LNDC: Waiting for lnd to unlock
2026-08-20 14:39:01.600 [INF] LNDC: Connected to lnd
2026-08-20 14:39:01.600 [INF] LNDC: Creating lnd connection to 127.0.0.1:10009
2026-08-20 14:39:01.600 [INF] LITD: Connecting full lnd client
2026-08-20 14:39:01.585 [INF] SQL2: Database version after migration: 6
2026-08-20 14:39:01.583 [INF] SQL2: Attempting to apply migration(s) (current_db_version=6, latest_migration_version=6)
2026-08-20 14:39:01.568 [INF] LNWL: Starting Web API fee estimator...
2026-08-20 14:39:01.568 [INF] CHRE: Using external fee estimator https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json: cached=true: min update timeout=5m0s, max update timeout=20m0s
2026-08-20 14:39:01.556 [INF] SQL2: Using SQL database 'postgres://xxx
2026-08-20 14:39:01.556 [INF] LITD: Basic lnd client connected
2026-08-20 14:39:01.556 [INF] LITD: Connecting basic lnd client
2026-08-20 14:39:01.499 [INF] BTWL: Opened wallet
2026-08-20 14:38:47.100 [INF] BTCN: Fully caught up with cfheaders at height 963319, waiting at tip for new blocks
2026-08-20 14:38:47.096 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=963, checkpoints=963
2026-08-20 14:38:47.096 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=963319, hash=4a9ff0778e21113f9460700529379201279548cb2a78a5aa275eced1604217d2
'----------------------------------------------------------
Web interface disabled
LiT Version 0.17.2-alpha-docker commit=v0.17.2-alpha-docker commit_hash=6b90de557550bc16c98d51cd031c764250f3e399
LND Version 0.21.2-beta
LND Alias xxx
LND Node status locked
LND Operating mode integrated
Lightning Terminal (LiT) by Lightning Labs
'----------------------------------------------------------
2026-08-20 14:38:46.506 [INF] BTCN: Starting cfheaders sync for filter_type=0
2026-08-20 14:38:46.506 [INF] BTCN: Starting cfheaders sync from (block_height=963319, block_hash=00000000000000000000d4d225070c76cce2148e82173fafd12a0436b6102393) to (block_height=963319, block_hash=00000000000000000000d4d225070c76cce2148e82173fafd12a0436b6102393)
2026-08-20 14:38:46.498 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 963319...
2026-08-20 14:38:46.465 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2026-08-20 14:38:46.465 [INF] LTND: We're not running within systemd or the service type is not 'notify'
2026-08-20 14:38:46.460 [INF] BTCN: Broadcaster now active
2026-08-20 14:38:46.460 [INF] BTCN: Registering block subscription: id=1
2026-08-20 14:38:46.460 [INF] BTCN: Loaded 2033 addresses from file '/data/.lnd/data/chain/bitcoin/mainnet/peers.json'
2026-08-20 14:38:46.448 [INF] BTCN: Block manager header state reset: block tip=963319, filter tip=963319
2026-08-20 14:38:46.432 [INF] BTCN: Headers import completed: processed 0 headers (block and filter) (added: 0, skipped: 0) from height 0 to 899999 in 17.123204605s (0.00 headers/sec, 0.00% new)
2026-08-20 14:38:29.860 [INF] BTCN: Chain continuity validation successful: import data (0-899999) connects properly with target chain
2026-08-20 14:38:29.309 [INF] BTCN: Using default write batch size of 65536 headers per region
2026-08-20 14:38:29.271 [INF] LTND: Database(s) now open (time_to_open=71.1658ms)!
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000015_chain_params' (version 18) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000014_payments_no_fail_reason_index' (version 17) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000013_payments_index_improvements' (version 16) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000012_drop_redundant_invoice_indexes' (version 15) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration 'kv_payments_migration' (version 14) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000011_payment_duplicates' (version 13) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000010_payments' (version 12) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000009_graph_v2' (version 11) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration 'kv_graph_migration' (version 10) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000008_graph' (version 9) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000007_invoice_add_settled_index' (version 8) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration 'kv_invoice_migration' (version 7) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000006_invoice_migration' (version 6) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000005_migration_tracker' (version 5) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000004_invoice_expiry_fix' (version 4) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000003_invoice_events' (version 3) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000002_amp_invoices' (version 2) as it has already been applied
2026-08-20 14:38:29.262 [INF] SQLD: Skipping migration '000001_invoices' (version 1) as it has already been applied
2026-08-20 14:38:29.260 [DBG] CHDB: Skipping optional migration: name=gc_decayed_log as it has already been applied
2026-08-20 14:38:29.260 [INF] CHDB: Checking for optional update: name=gc_decayed_log
2026-08-20 14:38:29.260 [DBG] CHDB: Skipping optional migration: name=prune_revocation_log as it is not specified in the config
2026-08-20 14:38:29.260 [INF] CHDB: Checking for optional update: name=prune_revocation_log
2026-08-20 14:38:29.260 [INF] CHDB: Applying 2 optional migrations
2026-08-20 14:38:29.257 [INF] CHDB: Checking for schema update: latest_version=35, db_version=35
2026-08-20 14:38:29.240 [INF] LTND: Using remote postgres database! Creating graph and channel state DB instances
2026-08-20 14:38:29.221 [INF] SQLD: Using SQL database 'postgres://xxx'
2026-08-20 14:38:29.200 [INF] LTND: Opening the main database, this might take a few minutes...
2026-08-20 14:38:29.173 [INF] LTND: TLS certificate is expired or outdated, generating a new one
2026-08-20 14:38:29.171 [INF] LTND: Network Info rev=6b90de active_chain=Bitcoin network=mainnet
2026-08-20 14:38:29.160 [INF] LITD: Listening for http_tls on: [::]:8443
2026-08-20 14:38:29.159 [DBG] STAT: Setting the taproot-assets sub-server as enabled
2026-08-20 14:38:29.159 [DBG] STAT: Setting the accounts sub-server as enabled
2026-08-20 14:38:29.158 [INF] LITD: LiT version: 0.17.2-alpha-docker commit=v0.17.2-alpha-docker commit_hash=6b90de557550bc16c98d51cd031c764250f3e399

[ziggie1984]

Thanks for the detailed logs, they were enough to pin this down. Short version: this is a startup race between tapd's custodian and litd's macaroon service over lnd's wallet DB, where both sides carry a hardcoded 30s deadline and both expire at the same instant. It is not a new bug, it is a latent race that your node has now grown into.

Timeline

Reordering your log chronologically:

14:40:49.394  GRDN: Loading wallet transactions starting at block height 0
14:40:49.481  STAT: Setting the taproot-assets sub-server as running
14:40:49.481  LITD: Starting LiT macaroon service
14:41:19.394  GRDN: Aborting main custodian event loop: ... context canceled   (+30.000s)
14:41:19.485  STAT: could not start macaroon service: ... DeadlineExceeded     (+30.004s)

Two independent 30s timeouts, to the millisecond. lnd itself is fine throughout, health checks keep running until 14:46 and a new invoice subscription lands at 14:42:02. So nothing crashed, two RPCs simply stalled for exactly the length of their timeouts.

Note also that the custodian error reads context canceled rather than DeadlineExceeded only because fn.ContextGuard.WithCtxQuitCustomTimeout implements its timeout as context.WithCancel plus a timer. It is a timeout, not a shutdown, and it fired 91ms before litd's error, so it is not fallout from the macaroon failure.

Mechanism

Both stalled calls hit the same btcwallet kvdb namespace with opposite lock modes.

1. tapd's custodian calls ListTransactions(lastDetectHeight, -1). lastDetectHeight is derived only from pending inbound asset events, and your log says Resuming 0 pending inbound asset events, so it is 0 and the scan covers genesis to tip. In btcwallet, Wallet.GetTransactions wraps the entire RangeTransactions walk in a single walletdb.View.

2. In lnd's postgres kvdb, View maps to a read-only tx which takes RLock on a per-namespace sync.RWMutex, held for the whole walk (kvdb/sqlbase/db.go:210-216, kvdb/sqlbase/readwrite_tx.go:32-43).

3. litd's macaroon service calls signrpc.DeriveSharedKey → keychain.BtcWalletKeyRing.ECDH → DerivePrivKey, which opens a walletdb.Update (keychain/btcwallet.go:287) and therefore takes the full Lock on that same mutex. Go's RWMutex is writer-preferring, so the write parks behind the scan.

4. That global lock is on by default for the wallet DB:

   // lncfg/db.go
   ChannelDBWithGlobalLock: false,
   // Default to true to maintain safe single-writer
   // behavior until the wallet subsystem is upgraded to
   // a native sql schema.
   WalletDBWithGlobalLock: true,

   It is wired into NSWalletDB only, which is exactly the namespace in play here.

5. litd starts the two concurrently by construction: StartIntegratedServers (terminal.go:851) is immediately followed by startInternalSubServers (terminal.go:855), 87ms apart in your log.

6. Both deadlines are 30s. tapd's is tapgarden.DefaultTimeout; litd's comes from lndclient's defaultRPCTimeout, because litd never sets RPCTimeout on the MacaroonServiceConfig and inherits it by accident.

One correction to a natural first guess: the tapd schema migration to version 60 visible in your log is not involved. It ran in 58ms and finished before the custodian scan even started, and no lnd migration ran at all this startup (every SQLD: line is "already been applied"). Those timings are actually useful in the other direction, they show your postgres is fast. The 30s is not DB slowness, it is round-trip count: kvdb-over-SQL turns a full-history wallet scan into many thousands of sequential queries, each individually quick.

Why it is not a regression

Every link in the chain predates v0.17:

Piece	Since
custodian's full-history ListTransactions under a 30s WithCtxQuit	5e2a32d0, 2022-09-16, unchanged since
litd's DeriveSharedKey macaroon DB unlock	50cfd3b8, 2022-03-30
StartIntegratedServers then startInternalSubServers ordering	f4ca4651, 2023-03-07
wallet DB global lock	hardcoded true in lnd 0.19, configurable but still default true in 0.20/0.21

The failure was also always fatal on both sides. In tapd v0.7.0 the same error went to reportErr → cfg.ErrChan, documented as the channel for critical errors.

So the race is old. What is new is that your wallet scan now takes longer than 30s. Candidates on your node: 2430 wallet addresses, 159 channels, native SQL enabled (db.use-native-sql), and four consumers on one postgres during that window (lnd kvdb, lnd native SQL, litd's own SQL2 store, and tapd's TADB).

One question that would help: did litd v0.16.x start cleanly on this same node and same postgres? Nothing in the code changed between v0.16 and v0.17 here, so if v0.16 worked it points at the extra concurrent load from lnd 0.21 + tapd 0.8; if it never worked with tapd enabled, the version correlation is coincidence and it is purely node size.

Stopgaps

Two options that change no storage-layer semantics:

1. Move postgres closer to litd (same host, ideally a unix socket). The scan's cost is essentially keys touched x round-trip latency, and lnd's kv-over-SQL layer issues one round trip per key (SELECT key, value FROM walletdb_kv WHERE ... AND key>$1 ORDER BY key LIMIT 1, kvdb/sqlbase/readwrite_cursor.go:93-96). Going from a network hop to a local socket is a 10-20x reduction on the whole scan. This is likely also behind the multi-second gaps between trivial lnd startup steps elsewhere in your log.
2. Temporarily set taproot-assets-mode=disable. Removes the long scan entirely, so the macaroon service's write is uncontended and litd starts. You lose tapd until a real fix lands, but nothing about the database's guarantees changes.

What not to do: please do not set db.postgres.walletdb-with-global-lock=false. See the note at the bottom of this comment.

Proper fixes

Three ingredients have to line up for this to fire: (A) a long-held RLock on the wallet namespace, (B) a deadlined, non-retrying Lock on it, (C) the two overlapping. Worth being explicit about which fix kills which, because only some of them close it for good.

1. litd, and this is the one that closes it. Two changes:

   • Kill (C): split the macaroon-service block out of startInternalSubServers and run it before StartIntegratedServers (terminal.go:851 / :855). There is no ordering dependency in either direction: that block needs only cfg.LitDir, cfg.Network, perms.RequiredPermissions, g.lndClient.LndServices and cfg.MacaroonPath, and perms.RequiredPermissions is a static package-level map (perms/permissions.go:10) rather than something sub-server registration populates. tapd's start takes createDefaultMacaroons as a plain bool, not the service itself.
   • Kill (B): retry DeriveSharedKey with backoff instead of erroring the lit sub-server permanently, and set RPCTimeout explicitly rather than inheriting 30s by omission. The reorder only protects this one pair of calls; the retry protects against wallet-lock contention from any source, including ones not enumerated here. Startup unlock is a natural place for a retry loop, there is no correctness cost to trying again.

2. tapd, the deeper fix but not sufficient on its own. Bound the custodian's startup startHeight. As it stands, any node with no pending inbound transfers rescans the full chain on every startup, which is wasted work independent of this bug. Done properly this wants a persisted "custodian scanned to height H" watermark: the SubscribeTransactions stream is deliberately started before the scan and covers new transactions, so the scan exists only to catch up on what arrived while tapd was down, which is exactly what a watermark expresses. AddrBook.LastEventHeightByVersion (already fetched two lines above, currently used only for the mailbox start height) is a reasonable interim bound.

   Note this attacks (A) only partially: a node with a pending inbound event from months ago still scans from that height, and a node offline a long time still has a wide range. Also, once bounded, that call should move to WithCtxQuitNoTimeout, since catch-up work is inherently unbounded, but only once bounded, because on its own that change lengthens the RLock hold and makes starvation worse.

3. lnd. The wallet global lock is known tech debt and goes away when the wallet subsystem moves to native SQL.

Related: #1124 was the same failure class, slow lnd → DeriveSharedKey misses its window → lit sub-server errored with no retry.

---

Edited to correct one point. The original version of this comment offered lnd's indexOffset/maxTransactions as an alternative way to bound the tapd scan. That is wrong and would make things worse. ListTransactionDetails calls GetTransactions(start, stop, accountFilter, nil) for the full range first and only then slices txDetails[indexOffset:end] in memory (lnwallet/btcwallet/btcwallet.go:1483-1560), so those two params reduce no DB work at all and N paginated calls would mean N full scans, each taking RLock. startHeight is the only parameter that touches the database. The fix ordering above was also re-ranked in the same edit: litd first, since that is what actually makes the failure impossible.

Second edit: withdrawing a workaround. An earlier version of this comment suggested db.postgres.walletdb-with-global-lock=false as a stopgap, described as a trade rather than a free win. That was too generous and I am withdrawing it.

That lock is a Go sync.RWMutex, not a postgres lock. With a single lnd process owning walletdb_kv it serialises every wallet transaction in-process, so postgres serialisation failures (SQLSTATE 40001) essentially cannot arise. Disabling it makes them possible, and the layer underneath is not retry-safe: lnd retries such conflicts up to DefaultNumTxRetries = 50 (kvdb/sqlbase/db.go:27) and calls reset() before each attempt, but btcwallet reaches the database through the legacy package-level helpers, which hard-code a no-op reset:

func View(db DB, f func(tx ReadTx) error) error   { return db.View(f, func() {}) }
func Update(db DB, f func(tx ReadWriteTx) error) error { return db.Update(f, func() {}) }

walletdb@v1.5.1/interface.go:259-274, used by all 39 walletdb.Update and 41 walletdb.View call sites in wallet/, wtxmgr/ and waddrmgr/. And the function at the centre of this issue is exactly the pattern that breaks under retry, accumulating into state declared outside the transaction:

var res GetTransactionsResult
err := walletdb.View(w.db, func(dbtx walletdb.ReadTx) error {
	...
		res.MinedTransactions = append(res.MinedTransactions, Block{...})   // wallet.go:2600

A retry re-runs the closure without clearing res and duplicates entries. Postgres SERIALIZABLE read-only transactions can still abort with 40001 under SSI, since DEFERRABLE is not set.

So the flag does not trade concurrency for safety; it moves the wallet from a regime where conflicts are impossible into one where they are possible and the retry path has incorrect reset semantics. I have confirmed one instance and have not audited the other 38 Update sites, which is itself a reason not to suggest it on a node with live channels. The in-tree comment calling it a temporary measure "until the wallet subsystem is upgraded to a native sql schema" reads, on this evidence, as load-bearing for correctness rather than caution about throughput. Filing the reset-semantics problem separately.

[ziggie1984]

To confirm the diagnosis (and work out which of two fixes you actually need), could you get us three numbers? None of these require a config change or a restart.

Short version of what we think is happening: tapd's startup scan asks lnd for your entire wallet transaction history, and on a postgres backend lnd's key-value layer issues one database round trip per key rather than one bulk query. So the scan's duration is roughly keys × round-trip latency. It holds a read lock on the wallet DB the whole time, which is what starves litd's macaroon service. The three numbers below tell us which half of that product is the problem.

1. How long the scan actually takes

This runs the exact same RPC the custodian uses, with tapd and litd completely out of the picture:

time lncli listchaintxns --start_height 0 --end_height -1

(However you normally run lncli against this node — you may need your usual --rpcserver / --macaroonpath / --tlscertpath flags.)

Please tell us how long it takes, or whether it never finishes at all. That distinction matters: slow and stuck have different causes and different fixes.

Two notes:

• Run it when nothing important is in flight. It takes the same wallet read lock for its full duration, so it can briefly stall wallet operations — the same effect you're already seeing at startup.
• Don't bother with --max_transactions to make it cheaper; lnd applies that after the full scan, so it won't speed anything up.

2. How many keys it is walking

Against your lnd database:

SELECT count(*) FROM walletdb_kv;

3. Where the database lives, and how far away

Is your postgres on the same host as litd, in another container on the same machine, elsewhere on your LAN, or a remote/managed instance? And is the connection string a TCP host or a unix socket?

Then, for the actual latency:

psql "<your lnd DSN>"
\timing on
SELECT 1;
SELECT 1;

and report the milliseconds it prints (the second one; the first includes warm-up).

Roughly, count from (2) × latency from (3) should land close to the time from (1). If it does, the diagnosis is confirmed and we know exactly which factor to attack — and moving the database closer may well be a large, immediate win for you regardless of what we change in the code.

If those numbers don't line up

Then our analysis is wrong somewhere and a goroutine dump is the next step. Add to lit.conf:

lnd.pprof.profile=9736

Restart, and once you see could not start macaroon service: unable to derive a shared secret with LND in the log, run this twice, about 10 seconds apart:

curl -s 'http://localhost:9736/debug/pprof/goroutine?debug=2' > goroutines-1.txt
sleep 10
curl -s 'http://localhost:9736/debug/pprof/goroutine?debug=2' > goroutines-2.txt

You don't need to race the 30 second window — the scan isn't cancellable server-side, so it keeps running after the error appears. Capturing twice is the point: if the stack under GetTransactions has moved between the two files the scan is progressing and merely slow; if it's identical and parked, something is genuinely stuck. These are stack traces only, no keys or credentials, but do skim them before attaching if you prefer.

(If you're on the Docker image you'll need to publish port 9736 or docker exec into the container to run the curl.)

---

Also still open from my earlier comment: did litd v0.16.x start cleanly on this same node and the same postgres? Nothing in the relevant code changed between v0.16 and v0.17, so that answer tells us whether this is a genuine threshold you've newly crossed or whether the version correlation is coincidence.

[ziggie1984]

@bufo24 could you inspect your db so we can understand what is causing this longer startup see here for more information: #1381 (comment)