CLN isn't tracking unilateral output correctly?

[grubles]

Now using v25.05rc1 but this has been happening since before that was tagged.

A relatively long-standing channel had errored with invalid revocation. Now the CLN node seems stuck waiting for an output to be confirmed. The problem is that output has some 12k confirms already.

 "state_changes": [                                                                                                                                                                                   
            {                                                                                                                                                                                                            
               "timestamp": "2024-07-27T00:55:10.767Z",                                                                                                                                                                  
               "old_state": "CHANNELD_AWAITING_LOCKIN",                                                                                                                                                                  
               "new_state": "CHANNELD_NORMAL",                                                                                                                                                                           
               "cause": "remote",                                                                                                                                                                                        
               "message": "Lockin complete"                                                                                                                                                                              
            },                                                                                                                                                                                                           
            {                                                                                                                                                                                                            
               "timestamp": "2025-03-21T21:18:08.717Z",                                                                                                                                                                  
               "old_state": "CHANNELD_NORMAL",                                                                                                                                                                           
               "new_state": "AWAITING_UNILATERAL",                                                                                                                                                                       
               "cause": "protocol",   
               "message": "channeld: received ERROR channel b7b3c2ac79c3f7643ad80c9b23635b1d59019c872906f0315d3b63e8f6078107: invalid revocation"
            },                                   

Ok the channel errored and it looks like CLN is trying to do a unilateral close.

  "status": [                                                                                        
            "ONCHAIN:Tracking our own unilateral close",                                                    
            "ONCHAIN:1 outputs unresolved: waiting confirmation that we spent DELAYED_OUTPUT_TO_US (onchaintxid:0) using OUR_DELAYED_RETURN_TO_WALLET"
         ],                 

The problem is the onchaintxid transaction has a ton of confs already and, strangely, CLN seems aware to some degree?

2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: billboard: 1 outputs unresolved: waiting confirmation that we spent DELAYED_OUTPUT_TO_US (onchaintxid:0) using OUR_DELAYED_RETURN_TO_WALLET                                                                                                                         
2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: Got new message WIRE_ONCHAIND_DEPTH                                                             
2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 12002                                  
2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: OUR_UNILATERAL/ANCHOR_TO_US->SELF depth 12002                                                   
2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: OUR_UNILATERAL/ANCHOR_TO_THEM->SELF depth 12002                                                 
2025-06-11T17:56:03.286Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 12002                                                 

[rustyrussell]

We're trying to spend onchaintxid:0 (to return it to the wallet). Has that tx indeed been spent? If not, are we trying to broadcast a tx which spends it? Is that failing for some reason? We often lowball these transactions, but right now that shouldn't matter...

[grubles]

It's unclear to me what is occurring. CLN has been trying to RBF transactions related to that channel close for multiple months now. Though at some point I was able to use the funds from that closed channel to open an entirely new one. I'm attaching the info.log contents for that channel so you can see what I mean.

infologs.txt

[rustyrussell]

See-also: #8503

[grubles]

Hmm no outputs in listfunds are unconfirmed.

But listpeerchannels has one, even though the output actually has ~23k confirmations and has since been spent.

$ lightning-cli listpeerchannels | grep unresolved

            "ONCHAIN:1 outputs unresolved: waiting confirmation that we spent DELAYED_OUTPUT_TO_US (tx with 23000 confirmations) using OUR_DELAYED_RETURN_TO_WALLET"

Also the info logs I attached above I think should show how CLN has been RBF'ing that output for months now.

[rustyrussell]

In both cases we missed a spend of something we care about...

[grubles]

Rescanning with --rescan=100000 seems to have fixed it?

2025-09-30T13:16:48.774Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: OUR_UNILATERAL/OUR_HTLC reached reasonable depth 27751                                                                  
2025-09-30T13:16:48.774Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-chan#14575: onchain_failed_our_htlc                                                                                                          
2025-09-30T13:16:48.774Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-chan#14575: HTLC id 80656 failonion = (nil), failmsg = 0x310e3d08, preimage = (nil)                                                          
2025-09-30T13:16:48.774Z DEBUG   0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-onchaind-chan#14575: billboard: All outputs resolved: waiting 0 more blocks before forgetting channel                                        
2025-09-30T13:16:48.774Z INFO    0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1-chan#14575: onchaind complete, forgetting peer