Skip to content

Commit 47ff1da

Browse files
committed
fix(network-fabric): propagate ledger info failures instead of rescanning from genesis
Signed-off-by: AkramBitar <akram@il.ibm.com>
1 parent cc7383b commit 47ff1da

9 files changed

Lines changed: 789 additions & 39 deletions

File tree

docs/services/network-fabric.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,58 @@ sequenceDiagram
294294
- LRU cache for recent transactions
295295
- Automatic retry on connection failures
296296

297+
#### Choosing the Starting Block
298+
299+
The scan resumes at the peer's current ledger height, read via `GetLedgerInfo`
300+
([`delivery.go`](../../token/services/network/fabric/finality/delivery.go)). Because that
301+
one RPC decides the starting block, a failure is retried with an exponentially growing
302+
delay — 7 attempts spanning ~31.5s by default, aborting early if the context is cancelled.
303+
Both are configurable via `token.finality.delivery.ledgerInfoAttempts` and
304+
`ledgerInfoRetryDelay` (see [Finality Configuration](#finality-configuration)).
305+
306+
That budget is deliberately long. Nothing retries `ScanBlock`: FSC's
307+
`events.ListenerManager` calls it once from a goroutine that only logs the result, so an
308+
error escaping the retry loop leaves the channel with **no block-based finality until the
309+
process restarts**. The retries therefore have to outlast a peer restart, not merely a
310+
dropped packet.
311+
312+
If the height is still unavailable after the last attempt, `ScanBlock` returns the error
313+
rather than defaulting to block 0. Starting at genesis would rescan the entire chain and
314+
replay finality notifications for every historical transaction, while the caller would have
315+
no way to tell a transient RPC failure from a genuinely fresh chain. Block 0 is used only
316+
when no ledger is configured at all.
317+
318+
The doubling has a ceiling of 30s, so raising `ledgerInfoAttempts` lengthens the budget
319+
without letting a single pause grow without bound. A `ledgerInfoRetryDelay` larger than the
320+
ceiling is honoured as configured — the cap limits growth, it does not shorten the delay you
321+
asked for. The backoff itself is `utils.RetryRunner`, shared with the rest of the SDK rather
322+
than reimplemented here.
323+
324+
The returned error is classifiable with `errors.Is`, so a caller does not have to match on
325+
its message. `ErrLedgerHeightUnavailable` is the single test for "the starting block could
326+
not be resolved, so no scan started" — it accompanies every such failure, including a
327+
cancelled one:
328+
329+
| Sentinel | Meaning |
330+
| --- | --- |
331+
| `finality.ErrLedgerHeightUnavailable` | the height could not be read, so no scan started |
332+
| `finality.ErrNoLedgerInfo` | **some** attempt saw the ledger return neither info nor an error — a driver contract violation |
333+
| `context.Canceled` / `context.DeadlineExceeded` | a wait between attempts was cut short, or the scan itself was cancelled |
334+
335+
Every attempt's failure is reported, so `ErrNoLedgerInfo` is present whenever the contract
336+
was violated at least once, even intermittently. The context error is not a discriminator on
337+
its own: the same context governs the scan, so it does not say which phase ended — pair it
338+
with `ErrLedgerHeightUnavailable` to tell a cancelled height read from a cancelled scan.
339+
340+
The underlying ledger errors are preserved in every case. An error that does not match
341+
`ErrLedgerHeightUnavailable` comes from the block scan itself rather than from resolving the
342+
starting block.
343+
344+
There is no in-tree caller that inspects these sentinels yet — today the sole consumer logs
345+
the error and stops. Classifying the failure is what a future caller needs to react
346+
(fall back to query-based finality, retry, or restart the manager) rather than a description
347+
of current behaviour.
348+
297349
### Notification Mode
298350

299351
Uses asynchronous event notifications from the FSC layer:
@@ -434,8 +486,15 @@ token:
434486
blockProcessParallelism: 10 # Parallel block processors
435487
lruSize: 30 # Cache size for recent transactions
436488
listenerTimeout: 10s # Timeout for listener notifications
489+
ledgerInfoAttempts: 7 # Attempts at reading the starting ledger height
490+
ledgerInfoRetryDelay: 500ms # First retry pause; doubles each attempt (~31.5s total)
437491
```
438492
493+
`ledgerInfoAttempts` and `ledgerInfoRetryDelay` bound the ledger-height read that decides
494+
where a block scan starts — see [Choosing the Starting Block](#choosing-the-starting-block).
495+
Non-positive values are ignored in favour of the defaults: zero attempts would refuse every
496+
scan, and a non-positive delay would busy loop.
497+
439498
### Endorsement Configuration
440499

441500
```yaml

token/services/network/fabric/config/config.go

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,19 +19,35 @@ type ListenerManagerConfig interface {
1919
DeliveryListenerTimeout() time.Duration
2020
DeliveryLRUSize() int
2121
DeliveryLRUBuffer() int
22+
DeliveryLedgerInfoAttempts() int
23+
DeliveryLedgerInfoRetryDelay() time.Duration
2224
}
2325

2426
const (
25-
DeliveryMapperParallelism = "token.finality.delivery.mapperParallelism"
26-
DeliveryBlockProcessParallelism = "token.finality.delivery.blockProcessParallelism"
27-
DeliveryLRUSize = "token.finality.delivery.lruSize"
28-
DeliveryLRUBuffer = "token.finality.delivery.lruBuffer"
29-
DeliveryListenerTimeout = "token.finality.delivery.listenerTimeout"
27+
DeliveryMapperParallelism = "token.finality.delivery.mapperParallelism"
28+
DeliveryBlockProcessParallelism = "token.finality.delivery.blockProcessParallelism"
29+
DeliveryLRUSize = "token.finality.delivery.lruSize"
30+
DeliveryLRUBuffer = "token.finality.delivery.lruBuffer"
31+
DeliveryListenerTimeout = "token.finality.delivery.listenerTimeout"
32+
// DeliveryLedgerInfoAttempts bounds how many times the current ledger height is
33+
// read before the block scan is refused. See finality.Delivery.
34+
DeliveryLedgerInfoAttempts = "token.finality.delivery.ledgerInfoAttempts"
35+
// DeliveryLedgerInfoRetryDelay is the pause before the first retry of that read;
36+
// it doubles on each further attempt, up to a ceiling that keeps a large
37+
// attempt budget from growing the pause without bound. See finality.Delivery.
38+
DeliveryLedgerInfoRetryDelay = "token.finality.delivery.ledgerInfoRetryDelay"
3039
DefaultDeliveryMapperParallelism = 10
3140
DefaultDeliveryBlockProcessParallelism = 10
3241
DefaultDeliveryLRUSize = 30
3342
DefaultDeliveryLRUBuffer = 15
3443
DefaultDeliveryListenerTimeout = 10 * time.Second
44+
// DefaultDeliveryLedgerInfoAttempts and DefaultDeliveryLedgerInfoRetryDelay
45+
// span ~31.5s of retries (0.5s + 1s + 2s + 4s + 8s + 16s). The budget has to
46+
// outlast a peer restart rather than a dropped packet: nothing retries the
47+
// block scan, so a height that stays unreadable costs the channel its
48+
// block-based finality until the process restarts.
49+
DefaultDeliveryLedgerInfoAttempts = 7
50+
DefaultDeliveryLedgerInfoRetryDelay = 500 * time.Millisecond
3551
)
3652

3753
type ManagerType string
@@ -88,6 +104,30 @@ func (c *serviceListenerManagerConfig) DeliveryListenerTimeout() time.Duration {
88104
return DefaultDeliveryListenerTimeout
89105
}
90106

107+
func (c *serviceListenerManagerConfig) DeliveryLedgerInfoAttempts() int {
108+
if v := c.c.GetInt(DeliveryLedgerInfoAttempts); v > 0 {
109+
return v
110+
}
111+
112+
return DefaultDeliveryLedgerInfoAttempts
113+
}
114+
115+
func (c *serviceListenerManagerConfig) DeliveryLedgerInfoRetryDelay() time.Duration {
116+
if v := c.c.GetDuration(DeliveryLedgerInfoRetryDelay); v > 0 {
117+
return v
118+
}
119+
120+
return DefaultDeliveryLedgerInfoRetryDelay
121+
}
122+
91123
func (c *serviceListenerManagerConfig) String() string {
92-
return fmt.Sprintf("Delivery [mapperParalellism: %d, lru: (%d, %d), listenerTimeout: %v]", c.DeliveryMapperParallelism(), c.DeliveryLRUSize(), c.DeliveryLRUBuffer(), c.DeliveryListenerTimeout())
124+
return fmt.Sprintf(
125+
"Delivery [mapperParalellism: %d, lru: (%d, %d), listenerTimeout: %v, ledgerInfo: (%d, %v)]",
126+
c.DeliveryMapperParallelism(),
127+
c.DeliveryLRUSize(),
128+
c.DeliveryLRUBuffer(),
129+
c.DeliveryListenerTimeout(),
130+
c.DeliveryLedgerInfoAttempts(),
131+
c.DeliveryLedgerInfoRetryDelay(),
132+
)
93133
}
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
/*
2+
Copyright IBM Corp. All Rights Reserved.
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
*/
6+
7+
package config_test
8+
9+
import (
10+
"testing"
11+
"time"
12+
13+
"github.com/LFDT-Panurus/panurus/token/services/network/fabric/config"
14+
"github.com/stretchr/testify/assert"
15+
)
16+
17+
// mapConfigService is a driver.ConfigService backed by two maps: whatever the test
18+
// puts in them is "set", everything else reads as a zero value, which is what an
19+
// absent YAML key looks like.
20+
type mapConfigService struct {
21+
ints map[string]int
22+
durations map[string]time.Duration
23+
}
24+
25+
func (c *mapConfigService) GetInt(key string) int { return c.ints[key] }
26+
func (c *mapConfigService) GetDuration(key string) time.Duration { return c.durations[key] }
27+
func (c *mapConfigService) GetString(string) string { return "" }
28+
func (c *mapConfigService) GetBool(string) bool { return false }
29+
func (c *mapConfigService) GetStringSlice(string) []string { return nil }
30+
func (c *mapConfigService) IsSet(string) bool { return false }
31+
func (c *mapConfigService) UnmarshalKey(string, any) error { return nil }
32+
func (c *mapConfigService) ConfigFileUsed() string { return "" }
33+
func (c *mapConfigService) GetPath(string) string { return "" }
34+
func (c *mapConfigService) TranslatePath(path string) string { return path }
35+
36+
// TestLedgerInfoRetryDefaults pins the defaults an unconfigured deployment gets.
37+
// The attempt budget matters beyond taste: nothing retries the block scan, so a
38+
// height that stays unreadable costs the channel its block-based finality until
39+
// the process restarts, and the budget has to outlast a peer restart.
40+
func TestLedgerInfoRetryDefaults(t *testing.T) {
41+
c := config.NewListenerManagerConfig(&mapConfigService{})
42+
43+
assert.Equal(t, config.DefaultDeliveryLedgerInfoAttempts, c.DeliveryLedgerInfoAttempts())
44+
assert.Equal(t, config.DefaultDeliveryLedgerInfoRetryDelay, c.DeliveryLedgerInfoRetryDelay())
45+
assert.GreaterOrEqual(t, totalRetryWait(c.DeliveryLedgerInfoAttempts(), c.DeliveryLedgerInfoRetryDelay()), 20*time.Second,
46+
"the default budget must outlast a peer restart, not just a dropped packet")
47+
}
48+
49+
// TestLedgerInfoRetryReadsConfiguredValues covers the point of the exercise: an
50+
// operator can shorten or lengthen the budget without a rebuild.
51+
func TestLedgerInfoRetryReadsConfiguredValues(t *testing.T) {
52+
c := config.NewListenerManagerConfig(&mapConfigService{
53+
ints: map[string]int{config.DeliveryLedgerInfoAttempts: 12},
54+
durations: map[string]time.Duration{config.DeliveryLedgerInfoRetryDelay: 250 * time.Millisecond},
55+
})
56+
57+
assert.Equal(t, 12, c.DeliveryLedgerInfoAttempts())
58+
assert.Equal(t, 250*time.Millisecond, c.DeliveryLedgerInfoRetryDelay())
59+
}
60+
61+
// TestLedgerInfoRetryRejectsNonPositiveValues covers the values a hand-edited YAML
62+
// can hold: 0 attempts would refuse every scan and a negative delay would busy
63+
// loop, so both fall back to the default rather than being honoured.
64+
func TestLedgerInfoRetryRejectsNonPositiveValues(t *testing.T) {
65+
for _, attempts := range []int{0, -1} {
66+
c := config.NewListenerManagerConfig(&mapConfigService{ints: map[string]int{config.DeliveryLedgerInfoAttempts: attempts}})
67+
assert.Equal(t, config.DefaultDeliveryLedgerInfoAttempts, c.DeliveryLedgerInfoAttempts(), "attempts %d must not be honoured", attempts)
68+
}
69+
70+
for _, delay := range []time.Duration{0, -time.Second} {
71+
c := config.NewListenerManagerConfig(&mapConfigService{durations: map[string]time.Duration{config.DeliveryLedgerInfoRetryDelay: delay}})
72+
assert.Equal(t, config.DefaultDeliveryLedgerInfoRetryDelay, c.DeliveryLedgerInfoRetryDelay(), "delay %v must not be honoured", delay)
73+
}
74+
}
75+
76+
// TestStringReportsLedgerInfoBudget keeps the startup log line informative: the
77+
// budget is otherwise invisible to an operator diagnosing a missing finality.
78+
func TestStringReportsLedgerInfoBudget(t *testing.T) {
79+
s := config.NewListenerManagerConfig(&mapConfigService{}).String()
80+
81+
assert.Contains(t, s, "ledgerInfo:")
82+
}
83+
84+
// totalRetryWait sums the doubling schedule ledgerHeight sleeps through: one wait
85+
// less than the number of attempts, each twice the previous.
86+
func totalRetryWait(attempts int, first time.Duration) time.Duration {
87+
var total time.Duration
88+
for i, delay := 0, first; i < attempts-1; i, delay = i+1, delay*2 {
89+
total += delay
90+
}
91+
92+
return total
93+
}

0 commit comments

Comments
 (0)