Skip to content

Commit cb13133

Browse files
committed
cute/profiler: compact decoder catches event_idx overflow via anchor divergence
When a caller's event_idx >= max_events_per_unit, the overflowing store lands in the next unit's slot 0 -- silently clobbering its anchor. The clobbered value looks like a packed event record, not a globaltimer timestamp, so the decoder's reconstructed timestamps for that unit are garbage. CTAs in the same launch start within microseconds, so all real anchors should agree to within ~1s. Detect divergence from the median and raise a clear error pointing the user at the slot-allocation bug. This caught an off-by-one in the agent_space gemm slot layout.
1 parent 86d027a commit cb13133

1 file changed

Lines changed: 17 additions & 1 deletion

File tree

  • transformer_nuggets/cute/profiler

transformer_nuggets/cute/profiler/host.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -330,7 +330,23 @@ def _decode_events_compact(buf: ProfileBuf, tag_table: TagTable, tid_base: int)
330330
anchors = grid[:, 0]
331331
records = grid[:, 1:]
332332

333-
valid = (records != 0) & (anchors != 0).unsqueeze(1)
333+
# CTAs in the same launch start within microseconds, so all real anchors
334+
# should agree to within a small window. An anchor diverging from the
335+
# median by >1s almost certainly means a caller event_idx overflowed and
336+
# clobbered the next unit's anchor slot.
337+
valid_anchor = anchors != 0
338+
if valid_anchor.any():
339+
median = anchors[valid_anchor].median()
340+
suspect = valid_anchor & ((anchors - median).abs() > 1_000_000_000)
341+
if suspect.any():
342+
bad = suspect.nonzero(as_tuple=True)[0].tolist()
343+
raise RuntimeError(
344+
f"Compact-mode anchors for units {bad} diverge from the median by >1s; "
345+
"likely an event_idx >= max_events_per_unit overflow clobbered the next "
346+
"unit's anchor. Bump max_events_per_unit or fix caller slot allocation."
347+
)
348+
349+
valid = (records != 0) & valid_anchor.unsqueeze(1)
334350
if not valid.any():
335351
return []
336352

0 commit comments

Comments
 (0)