litd Shows SERVER_ACTIVE but Commands Unresponsive, Channel State Mismatch After Restart

[shuishihang]

This issue is occurring across multiple nodes in my network. When checking node status via litcli status, it shows normal status:
"state": "SERVER_ACTIVE"

However, the terminal becomes completely unresponsive to any commands until timeout occurs. Service only resumes after restarting litd.

Channel State Mismatch:

Most affected nodes actually have their asset channels force-closed at the peer level
After restart, these channels still appear as "online" in listchannels output
The online status persists across multiple restarts

Some nodes: Channels eventually transition to "force-closed" status after waiting + restarting
Majority of nodes: Channels remain persistently "online" in listchannels until I manually force-close them

I have identified logs from 4 nodes that experienced this issue. Logs 3 and 4 document the most frequently occurring cases where forced channel closures were triggered.

lnd4.log.gz
lnd3.log.gz
lnd2.log.gz
lnd.log.gz

[ViktorT-11]

Thanks for the report @shuishihang,

This needs further investigation, but here's some initial findings:
I think the main problem here is that taproot-assets never starts fully in any of your logs. The exact reason why taproot-assets never starts starts is still unclear to me. While your litcli status command reports "state": "SERVER_ACTIVE", that's only the report for the status server which responds to your litcli status command, and not for litd itself.

After looking through your logs, I never see any STAT: logline which reports that any sub-server is started, such as for example "STAT: Setting the lit sub-server as running" which would be expected when litd has fully started.
I only see "STAT:" loglines which reports that the taproot-assets sub-server has failed to start, and that the lit sub-server has failed to start due to lnd not fully starting.
As lnd will use taproot-assets as the aux-signer and taproot-assets never fully starts, I think that's the main issue that's causing lnd to not start correctly. The rest of the sub-servers will therefore not start correctly either, as they're reliant on lnd.

Regarding the force-closure issue:
I can't find any loglines which cover the time when the channel(s) were actually force closed, do you know if these logs should cover that timespan? I do however find cases of ChannelArbitrators, being created (such as "Creating ChannelArbitrators for 1 closing channels" for lnd4.log), but never fully resolving. I'm pretty sure the reason they never fully resolve is due to the above issue that lnd needs taproot-assets to be started as the aux-signer though.
That explains your experienced discrepancy in which "status" the channel has in the listchannels output.

We will investigate the taproot-assets startup issue further, as I'm pretty confident that's your main issue here!

[shuishihang]

Thank you for your reply. I would like to know what situations could cause a channel to be force-closed when I did not manually trigger a channel close. Multiple of my nodes frequently experience force closures. Each of my nodes has both a BTC channel and an asset channel. Force closures rarely happen on the BTC channels; most of the force closures occur on the asset channels. In addition, it seems that many of the nodes did not experience a sudden power outage.

[ziggie1984]

In the provided logs there are no force closes happening, so you might share a bigger log period where you are sure a Force-Close happened. Does the listchannels command in your case show that the channel is not active tho, meaning that it shows up in the listchannels output but is tagged as inactive there ?

[shuishihang]

My node's funded channel appears as active in listchannels, but the peer node is already listed in pendingchannels or closedchannels. Below, I've collected the full logs of several nodes where this issue occurred

https://cdn.file.microlinktoken.com/box/Downloads.zip

[shuishihang]

My LITD version has been updated to 0.16.1-alpha, but the issue of forced closures in asset channels still persists and has not been fully resolved.

I have collected logs from the asset channel opener's side over a certain period, during which approximately two asset channels were forcibly closed. Their channel points (chan_points) are:
https://cdn.file.microlinktoken.com/box/lnd.log.gz
7b0ca01c2499cc116ce8907c64ab0f99b08fbe1f012d2505581c4f78ff4aca6e:0
7d1801b64dd2c4cbefca99b732e4a7589e051de3b132ce84c4a65fa70866806e:0
This issue has existed for a long time, persisting across multiple versions and recurring repeatedly without ever being fundamentally resolved.

In my test scenario, I have deployed 4 main server nodes to open asset channels and perform periodic asset push operations to a total of 1500 peer nodes using the keysend method. However, these main nodes themselves frequently experience forced closures at regular intervals, which has become the most common and critical issue in the current setup. During implementation, I utilized 10 goroutines per server node to concurrently execute keysend calls for distributing assets to different nodes. Could these architectural factors and concurrency patterns be contributing to the systemic forced closures observed across my asset channels?

Additionally, the BTC capacity in my asset channel is set to 20,000 sats.

[jtobin]

@shuishihang these are the same symptoms reported in lightninglabs/taproot-assets#1645, yeah? I.e. the thing we presumed fixed the issue there did not actually fix the issue?

[shuishihang]

yes

[shuishihang]

I found another issue. After updating to the new version, some of the forcibly closed asset channels were indeed closed properly, but there are still many asset channels in the waiting_close_channels state that have not been handled. Below is the information returned by my node calling lncli pendingchannels, along with the corresponding logs from the node.

Node1.txt
https://cdn.file.microlinktoken.com/box/Node1.log.gz
Node2.txt
https://cdn.file.microlinktoken.com/box/Node2.log.gz
Node3.txt
https://cdn.file.microlinktoken.com/box/Node3.log.gz