You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fourth upstream issue filed: HIT_PENDING no-deadline hang in the shared
tiering manager, Codex-reviewed. All four known crash/stall defects are
now filed; #49820 and #49829 remain unfixed upstream.
Signed-off-by: nilig <nili.ifergan@gmail.com>
Maps the three filed upstream issues to the two internal defect catalogs
3
+
Maps the four filed upstream issues to the two internal defect catalogs
4
4
(`p2p-pd-defects.md`, `p2p-lookup-hangs.md`) and states what remains after
5
-
each upstream fix lands. All three issues are OPEN as of 2026-07-25.
5
+
each upstream fix lands. All four issues are OPEN as of 2026-07-25.
6
6
7
-
## The three issues (distinct bugs, distinct layers)
7
+
## The four issues (distinct bugs, distinct layers)
8
8
9
9
| issue | what it is | layer / file | crash or stall | catalog | fix PR |
10
10
|---|---|---|---|---|---|
11
11
|#49635| late KV-offload store after request finalization -> `KeyError` (`_req_state` deleted at `tiering/manager.py:721`, indexed unguarded at `:542` via `scheduler.py:1045 _build_store_jobs`) | OffloadingConnector store path -- NOT P2P-specific | crash | neither catalog |#49671 (open) |
12
12
|#49809| reconnect to a reaped peer trips `AssertionError: ZmqConnection already exists`; dead conn never released | P2P control transport (`tiering/p2p/control/zmq.py`) | crash | pd-defects Defect 2 |#49823 (open) |
13
13
|#49820| symmetric producer accepts a fetch it cannot serve, never sends `TransferDone(success=False)` -> consumer deferred full `_LOAD_TIMEOUT_S=30s`| P2P session (`tiering/p2p/session/*`) | stall | new (post-Liran residual) | none |
14
+
|#49829|`TieringOffloadingManager.lookup()` returns `HIT_PENDING` unconditionally, no deadline, no downgrade-to-MISS | shared tiering manager (`tiering/manager.py`) -- NOT P2P-specific | stall | lookup-hangs Defect 3 | none |
14
15
15
-
Distinguishing axis: #49635is OffloadingConnector-general (fires even
16
-
aggregated / no P2P); #49809 and #49820 are P2P-tier-specific but different
17
-
sub-layers (control transport vs session).
16
+
Distinguishing axis: #49635and #49829 are OffloadingConnector-general (fire
17
+
even aggregated / no P2P, different files from each other); #49809 and #49820
18
+
are P2P-tier-specific but different sub-layers (control transport vs session).
18
19
19
-
## Coverage: are those three enough?
20
+
## Coverage: are those four enough?
20
21
21
-
No. After #49635 (->#49671), #49809 (->#49823), #49820 (->future fix), and
22
-
Liran's #48021 all land:
22
+
Yes, this is now the complete known set. After #49635 (->#49671), #49809
23
+
(->#49823), #49820 (->future fix), #49829 (->future fix), and Liran's #48021
24
+
all land:
23
25
24
26
| defect | covered by | patch still needed? |
25
27
|---|---|---|
@@ -28,7 +30,7 @@ Liran's #48021 all land:
28
30
|#49820 symmetric-fetch 30s stall |#49820 fix (TBD) | no, once written |
29
31
| lookup-hangs Defect 1 (duplicate-fetch teardown) | Liran's one-fetch contract in #48021 (`dupfetch=0` on a version-matched engine) | no |
30
32
| lookup-hangs Defect 2 (pop-on-read MISS livelock) | Liran's `register_lookup` retains resolved HIT+MISS until fetch/finish (no pop-on-read) = the sticky-MISS. VERIFIED in `client.py`@145a460c| no (`defect12` Part 2 redundant) |
31
-
| lookup-hangs Defect 3 (`HIT_PENDING` write-in-flight, no deadline in UPSTREAM `tiering/manager.py`) |nothing | YES -- `defect3_fix_pending-wait-deadline.diff`, or file upstream|
33
+
| lookup-hangs Defect 3 = #49829(`HIT_PENDING` write-in-flight, no deadline in `tiering/manager.py`) |#49829 fix (TBD) | no, once written -- `defect3_fix_pending-wait-deadline.diff` validates the direction (deterministic repro: `repro_defect3.py`)|
32
34
33
35
Softer gap: `defect12` Part 1 (lookup-phase deadline) is NOT in Liran's branch
34
36
-- only `_LOAD_TIMEOUT_S=30` on the fetch phase, no deadline on a stranded
@@ -46,14 +48,16 @@ Two catalogs both start at "Defect 1", causing collisions:
46
48
47
49
## Net
48
50
49
-
Two crashes fixed (#49635, #49809), the 30s stall fixed (#49820), and
50
-
lookup-hangs Defect 1+2 covered by #48021. The remaining real bug is
51
-
lookup-hangs Defect 3 -- an upstream `tiering/manager.py` hang untouched by any
52
-
P2P issue.
51
+
Two crashes fixed (#49635 -> #49671, #49809 -> #49823), and lookup-hangs
52
+
Defect 1+2 covered by #48021. Two stalls remain unfixed: #49820 (P2P session,
53
+
symmetric fetch) and #49829 (shared tiering manager, `HIT_PENDING`
54
+
write-in-flight -- lookup-hangs Defect 3). Both have a documented mechanism, a
55
+
deterministic repro, and a validated local patch; neither has an upstream fix
56
+
yet.
53
57
54
58
VERIFIED 2026-07-25 against `main`@70009fb9: `TieringManager.lookup()` still
55
59
returns `LookupResult.HIT_PENDING` bare, with no deadline. Upstream added only
56
60
observability around it (`secondary_lookup_start_time`, `sync_lookup_delay`
57
-
accumulation) -- no bound, no downgrade-to-MISS. So Defect 3 is LIVE in main;
58
-
`defect3_fix_pending-wait-deadline.diff` is still required. Not filed upstream
59
-
-> candidate for a companion issue to #49809/#49820.
61
+
accumulation) -- no bound, no downgrade-to-MISS. So Defect 3 was LIVE in main;
62
+
filed as **#49829** (Codex-reviewed, ASCII-only, cc @orozery@ronensc).
63
+
`defect3_fix_pending-wait-deadline.diff` is still required until it lands.
0 commit comments