[bug]: payment_failure_reason > 0 when payment status = SUCCEEDED

[rsafier]

Background

Appears since 0.18.0 some payments now occasional will have payment_failure_reason not equal to FAILURE_REASON_NONE despite payment being in SUCCEEDED status, and having preimage recorded.

Your environment

• lnd 0.18.0-beta
• Linux ip-100-0-1-117 6.5.0-1020-aws 22.04.1-Ubuntu SMP Wed May 1 16:10:50 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• bitcoind backend

Steps to reproduce (only works on 0.18.0 and daily-testing-only dockerimage, 0.17.5 does not show this behavior)

• Make network Alice->Bob->Carol
• Have HTLC interceptor delay all traffic by 30s on Bob
• Have Carol pay invoice on Alice with TimeoutSeconds set at 15s on the SendPaymentV2 request.
• Original blocking payment will return SUCCEEDED and FAILURE_REASON_NONE
• TrackPayment again and will return SUCCEEDED and FAILURE_REASON_TIMEOUT

C# unit test of behavior

Expected behaviour

if payment status equals SUCCEEDED then payment_failure_reason should be reported back as FAILURE_REASON_NONE as in prior LND versions.

Actual behaviour

Some payments in SUCCEEDED status also have non-zero payment_failure_reason set

[rsafier]

I also now am noticing same behavior with some successful payments having a payment_failure_reason of Canceled

[guggero]

I think there have been some fixes for this. Since you have a unit test handy, can you please test if this is still the case on 0.18.5 (and also current master)?

[rsafier]

0.18.5 is still generating these in our production environment. Will try some tests on master and report back.

[rsafier]

0.19.0rc1 tested continues to generate this behavior in testing.

Seeing ~0.5% on average of our successful production payments being marked like this.

[yyforyongyu]

I think what's happening here is, that the payment is timed out (FAILURE_REASON_TIMEOUT) or canceled FAILURE_REASON_CANCELED while it still has inflight HTLCs, at this stage the payment status would be inflight. Later on the htlcs got settled, then the status will be succeeded. This is expected, as if the payment still has inflight HTLCs, we cannot say the payment is finished, so relying on the status field is the most reliable.

We probably could rename payment_failure_reason to sth like payment_memo, as it's just a field to record the possible failure reasons, but it doesn't determine the payment's final state.

Also FAILURE_REASON_CANCELED can be an issue as it means the user canceled the payment via RPC/cli, but it doesn't mean the payment won't get settled. Do you know how many payments are being marked like this?

[rsafier]

I think what's happening here is, that the payment is timed out (FAILURE_REASON_TIMEOUT) or canceled FAILURE_REASON_CANCELED while it still has inflight HTLCs, at this stage the payment status would be inflight. Later on the htlcs got settled, then the status will be succeeded. This is expected, as if the payment still has inflight HTLCs, we cannot say the payment is finished, so relying on the status field is the most reliable.

We probably could rename payment_failure_reason to sth like payment_memo, as it's just a field to record the possible failure reasons, but it doesn't determine the payment's final state.

Also FAILURE_REASON_CANCELED can be an issue as it means the user canceled the payment via RPC/cli, but it doesn't mean the payment won't get settled. Do you know how many payments are being marked like this?

We have already refactored make sure we only use the payment status and not source reason field for any success/fail logic.

Looks like the FAILURE_REASON_CANCELED only shows up post 0.18.3 deployment in the data.
Tens of thousands of records in this pattern at this point. Overall average time to resolve is 1-2s, the timeout marked ones tend to be in the 100-1000s range, and the cancels look to be always quicker than NONE reason successful payments. This seems consistent, other than I don't believe we have been utilizing the cancel functionality via API, at least explicitly.

[ziggie1984]

Given your description we know the underlying issue. We are going to fix this while refactoring the code for the upcoming SQL support (LND 20). We want to refactor the payment state on the way so there will be a fix in 20.

FYI: The payment failure reason "canceled" was only introduced in LND 18.3, before every payment would be considered timed-out rather than making an explicit distinction between both. But we only set the "Canceled Reason" when the Payment flow is explicitly canceled so you might look into the scenarios where this might happen for you.