Attributable failures (feature 36/37)

[joostjager]

Failure attribution is important to properly penalize nodes after a payment failure occurs. The goal of the penalty is to give the next attempt a better chance at succeeding. In the happy failure flow, the sender is able to determine the origin of the failure and penalizes a single node or pair of nodes.

Unfortunately it is possible for nodes on the route to hide themselves. If they return random data as the failure message, the sender won't know where the failure happened.

This PR proposes a new failure message format that lets each node commit to the failure message. If one of the nodes corrupts the failure message, the sender will be able to identify that node.

For more information, see https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-October/003723.html.

LND implementation: lightningnetwork/lnd#7139

LDK implementation: lightningdevkit/rust-lightning#3611

Eclair implementation: ACINQ/eclair#2519

[thomash-acinq]

I've started implementing it in eclair, do you have some test vectors so we can check that we are compatible?
The design seems good to me, but as I've said previously, I think keeping hop payloads and hmacs for 8 nodes only (instead of 27) is enough for almost all use cases and would give us huge size savings.

[joostjager]

I don't have test vectors yet, but I can produce them. Will add them to this PR when ready.

Capping the max hops at a lower number is fine to me, but do you have a scenario in mind where this would really make the difference? Or is it to more generally that everything above 8 is wasteful?

[joostjager]

@thomash-acinq added a happy fat error test vector.

[joostjager]

I think this big gap in the bits has emerged here because of tentative spec changes that may or may not make it. Not sure why that is necessary. I thought for unofficial extensions, the custom range is supposed to be used?

I can see that with unofficial features deployed in the wild, it is easier to keep the same bit when something becomes official. But not sure if that is worth creating the gap here? An alternative is to deploy unofficial features in the custom range first, and then later recognize both the official and unofficial bit. Slightly more code, but this feature list remains clean.

[joostjager]

Added fat error signaling to the PR.

[thomash-acinq]

I've spent a lot of time trying to make the test vector pass and I've finally found what was wrong:
In the spec you write that the hmac covers

• failure_len, failuremsg, pad_len and pad.

• The first y+1 payloads in payloads. For example, hmac_0_2 would cover all three payloads.

• y downstream hmacs that correspond to downstream node positions relative to x. For example, hmac_0_2 would cover hmac_1_1 and hmac_2_0.

implying that we need to concatenate them in that order. But in your code you follow a different order:

// Include payloads including our own.
_, _ = hash.Write(payloads[:(NumMaxHops-position)*payloadLen])

// Include downstream hmacs.
var hmacsIdx = position + NumMaxHops
for j := 0; j < NumMaxHops-position-1; j++ {
	_, _ = hash.Write(
		hmacs[hmacsIdx*sha256.Size : (hmacsIdx+1)*sha256.Size],
	)

	hmacsIdx += NumMaxHops - j - 1
}

// Include message.
_, _ = hash.Write(message)

I think the order message + hop payloads + hmacs is more intuitive as it matches the order of the fields in the packet.

[joostjager]

Oh great catch! Will produce a new vector.

[joostjager]

@thomash-acinq updated vector

[joostjager]

Updated LND implementation with sender-picked fat error structure parameters: lightningnetwork/lnd#7139

[thomash-acinq]

This proposal brings us two things:

• It fixes a flaw in the failure message design that allows any node to opt out.
• It tells senders which nodes are fast and which ones are slow.

The second thing is not really related to failures. If we think it is valuable, we should add it to the success case too.

[thomash-acinq]

We could clarify how the hold time is computed exactly. Do we start counting when receiving update_add_htlc, commitment_signed or revoke_and_ack? It should probably be revoke_and_ack because that's when we can start relaying.

[tnull]

We discussed this on multiple occasions offline, but while we're at it, it would be good to introduce the notion of 'acceptable delay threshold' to the spec. For privacy and HTLC-batching/efficiency reasons, forwarding nodes still want to introduce some artificial forwarding delay, and scoring/penalization should really only start above a certain threshold deemed acceptable by the community.

[joostjager]

I think it is up to implementations to decide how to compute hold time exactly, and make it so that it is in the best interest of their users. In the end, the rational thing to do is look at the sender logic for penalizing latency, and optimize for minimal penalty?

[joostjager]

@thomash-acinq carried out an interop test between LDK and Eclair and it passed 🎉 A milestone. Will proceed with updating this PR and get it ready for merge.

[thomash-acinq]

Clarify that we use the return packet before the xor.

[thomash-acinq]

"covers" is a bit ambiguous, make it explicit that these things are concatenated in this order. Same below for hold times and hmacs. For hold times, you can say that we take the first (y + 1) * 4 bytes of htlc_hold_times.

[thomash-acinq]

It would be nice to add a new test vector where the failing node returns random garbage and we can still attribute the failure.

[joostjager]

@GeorgeTsagk suggested the exact same thing today. In LDK, I've added a test that mutates every byte for every node, and asserts that it is properly attributed. Basically an exhaustive test to make sure it always works.

Will also add a random garbage test vector.

[joostjager]

I am not sure what that vector should look like though. The transformation of the failure packet is already covered in the happy flow test vector. Nodes can't read the data returned by their upstream peer, and the transformation isn't affected by that data being random. Testing that attribution works is just a matter of asserting that the decrypt process picks up the node that returned the random data.

Open to ideas for extending the test vectors!

[GeorgeTsagk]

I think we can make a vector where an intermediate hop simply drops all data (as if they don't understand it) and then the next upstream hop re-creates the attr data.

[joostjager]

But is this testing anything new? That next upstream hop will just recreate attribution data as if it were the failure source, and from there it will flow upstream again.

[joostjager]

Regarding hold time reporting, selection pressure in pathfinding, and its potential negative effect on privacy: can't senders opt-in to privacy-preserving random delays via the forward onion? That way there is no loss of precision and nodes can still serve both privacy-valueing users as well as the ones that aim for the absolute fastest routes.

[t-bast]

can't senders opt-in to privacy-preserving random delays via the forward onion?

That could be an interesting trade-off. The sender would include a TLV in the onion to ask relaying nodes to add a random delay? We'd go from delays/batching decided by the intermediate nodes to something that is driven by the senders instead. At first glance, I think it would work.

The downside would be that it's off by default (because most app developers will care more about UX than privacy), while I believe that Matt would like basic privacy features to be on by default at the network level.

I personally think using 100ms precision would be a good trade-off: batch/delays of this order of magnitude still allow for sub-second payment latency for paths that aren't too long, which I believe is good enough in terms of payment experience while offering some privacy if done at the network level.

I'm more in favor of always having 100ms precision (instead of 1ms precision), but if there is a larger consensus for 1ms precision, I'd be ok with it! Not sure how to move forward though: should we just have a small survey among implementers?

[tnull]

Regarding hold time reporting, selection pressure in pathfinding, and its potential negative effect on privacy: can't senders opt-in to privacy-preserving random delays via the forward onion? That way there is no loss of precision and nodes can still serve both privacy-valueing users as well as the ones that aim for the absolute fastest routes.

Well, I don't think this is a good idea as "privacy loves company" and having senders signal "hey, look at me, I want to be more private - I have something to hide" might actually have them stand out more overall.

The preferred outcome would really be that we agree on a reasonable delay threshold that we encourage for all forwarding nodes. If all just do a little, it might introduce sufficient noise to throw on-path and AS-level attackers off.

I'm def. +1 for 100ms buckets, which should be mostly sufficient to cover at least 1-2 RTTs of the handshake, hence introducing at least some degree of uncertainty on the attacker's side.

[joostjager]

There are users who don't like privacy delays. They patch their nodes to remove the delay. And surely there will also be users who don't like coarse grained hold times.

Do we really want to force this trade off onto them via the protocol without a way to opt-out, and rule out possibly undiscovered use cases that require ultra-fast payment settlement?

Picking a constant also doesn't look particularly attractive to me.

[t-bast]

Do we really want to force this trade off onto them via the protocol without a way to opt-out, and rule out possibly undiscovered use cases that require ultra-fast payment settlement?

If you require ultra-fast settlement, there are always cases where multi-hop payments will fail you: shouldn't you just open a channel to your destination instead?

[joostjager]

I still believe in a future of the lightning network where failures aren't tolerated at all, and all routing nodes deliver excellent service. Then it's just a matter of measuring latency and going for the sub-10 ms ones.

[joostjager]

Hold time filtering to avoid users doing things we think they shouldn't. OP_RETURN vibes 😂

[t-bast]

I still believe in a future of the lightning network where failures aren't tolerated at all, and all routing nodes deliver excellent service.

I agree, but that's orthogonal to payment latency? I honestly don't understand why it's important to aim for minimal latency: is it for performance's sake, to boast about numbers? A payment that takes 500ms to complete is almost indistinguishable from being instantaneous for our brains, so why does it matter that it takes 50ms instead of 500ms if you have to sacrifice privacy for it?

[joostjager]

For me it is two-fold:

• I don't want to rule out potential future uses cases that somehow do require minimal latency.

• It is unclear to me how much an arbitrary 100 ms granularity is going to help with privacy today and also in a future in which we are stuck with this hard-coded value in the protocol. Yes it probably helps, but it may not be substantial.

Furthermore I think that an instant instant payment with no observable latency at all is just cool too indeed.

[GeorgeTsagk]

Are we talking about introducing actual time delays here, or just bucketing the reported value of the hold time?

If we're talking about further delaying the actual forwarding of the HTLC then I'm very much against it. Keep in mind this is done per payment attempt so if I need to try out 20 different routes this will accumulate quite fast, significantly killing ux.

Seems like the concern is that the accurate reporting of hold times does not directly ruin the privacy for the nodes reporting the hold times, but will eventually lead to intense competition around latency, contributing to further centralization in the network?

If we're talking about hold times leaking information about the routing nodes (fingerprinting?) then this is already too noisy to produce consistent results. Every node has different hardware and internet connectivity configuration, I wouldn't expect all LNDs or LDKs to have some entropy in their reports, and there exist better ways to fingerprint anyway.

I don't think we should pollute the protocol with this kind of safeguard/filter against more precise reports. If we assume that a user is willing to get their hands dirty to somehow signal lower hold times, they'll find a way. Also let's not underestimate the power of what implementations default to.

By having implementations default to scoring all hold times <=100ms the same, or reporting buckets of 100ms by default, most of the concerns seem to be eliminated. Of course, if we assume the majority of the network to be running custom software then what's even the point of this discussion?

[DerEwige]

I’m very much against the intentional slowing down of HTLC resolves.

Every milli second a HTLC is unresolved it poses a risk to the node runner.
One of the common causes of force closures are stuck HTLCs.

A HTLC gets stuck when a node goes offline after the HTLC passed that node.
So the failure or success can not propagate backwards through it.
The chances of a HTLC getting stuck is proportional to the time the HTLC needs to resolve.

So if you intentionally slow down the HTLC resolve speed, you also increase the number of force closures.

I would consider the intentional hold of a HTLC of more than 10ms as an jamming attack.

[t-bast]

A HTLC gets stuck when a node goes offline after the HTLC passed that node.
So the failure or success can not propagate backwards through it.
The chances of a HTLC getting stuck is proportional to the time the HTLC needs to resolve.
So if you intentionally slow down the HTLC resolve speed, you also increase the number of force closures.

That's not at all a valid conclusion. HTLCs get stuck and channels force-close because of bugs, not because nodes go temporarily offline or introduce batching/delays. That's completely unrelated.

I 100% agree with you that all bugs that lead to stuck HTLCs or force-closed channels must be fixed and should be the highest priority for all implementations. But that has absolutely nothing to do with whether or not we should introduce delays or measure relay latency in buckets of 100ms.

[DerEwige]

That's not at all a valid conclusion. HTLCs get stuck and channels force-close because of bugs, not because nodes go temporarily offline or introduce batching/delays. That's completely unrelated.

I 100% agree with you that all bugs that lead to stuck HTLCs or force-closed channels must be fixed and should be the highest priority for all implementations. But that has absolutely nothing to do with whether or not we should introduce delays or measure relay latency in buckets of 100ms.

I do agree, that it is usually a bug or sometimes negligence, that is the root cause of most force-closes.

I also totally agree that there should be an easy way to measure relay latency.
Currently the only way to measure relay latency is by aggregating a lot of data (100’000+ payments) and calculate the time for each channel from that data.

In fact I do exactly this to generate the list you can find here on the “node speeds” tab
https://docs.google.com/spreadsheets/d/1gn2TY3zDoPdk46yYQfRhGb2_YDepQEn7l3qtHWCxov4
And I would love to have simpler and more accurate way to do this. And 100ms accuracy in the reported time would also be good enough.

But saying that adding a relay delay would not have an impact of the number of force-closes is - in my opinion - wrong.

If a channel with active HTLC goes offline and does not come online again before the HTLC reaches its timeout, this will lead to a force-close.
The chance of a channel going offline with an active HTLC on it increases with the time a HTLC lingers on the channel.

I send out more than 100’000 HTLCs each day.
0.1s extra per hop would add hours of exposure each day.