conformance: read the archive, and the loss it admits to

[juan-malbeclabs]

--pcap refused pcapng with open source: Unknown magic a0d0d0a, which
is its Section Header Block read as a legacy pcap file header. pcapng is
the format the recorder archives, so the validator could not read the
archive it exists to judge, and every replay went through a conversion
first.

The conversion works — nanosecond timestamps survive — but it strips the
per-packet epb_dropcount option, and that option is the recorder's own
admission of what it failed to write. It is the only field in an
archive that separates capture loss from publisher loss, so a converted
replay can charge the recorder's drops to the publisher: the one error
keeping the bytes was meant to prevent. The only guard was reading
capture_drop_total out of the segment manifest by hand, which nothing
enforced.

Reading the format

input/pcapng.go parses the blocks: both byte orders, if_tsresol and
if_tsoffset, multiple sections, the three packet-bearing block types,
and every other block skipped by length. gopacket's NgReader was not
used because it discards per-packet options — its own comment says so —
which is exactly the field that matters here. Both formats go under the
one --pcap flag, chosen from the file's magic.

Verified on real data: the committed 2001-packet market-by-price capture,
converted to pcapng, replays to a byte-identical JSON report.

What the rule set does with it

epb_dropcount reaches the engine per datagram, ahead of the frame whose
windows it taints, and taints every instance on every port — the recorder
drops at its interface, so what it lost was never parsed and its channel,
source and destination port are all unknowable. That is the direction
taintPortWide already takes: the cost is a rule grading unverifiable
instead of pass, never a violation the publisher did not commit.

• New capture_loss reason, distinct from loss because the owner is
  different — loss sends an operator to the network, capture_loss
  sends them to the recorder. Emit re-owns a loss-explained finding at
  the one point every finding passes through.
• MBP.DELTA.PERINSTR_DENSITY now downgrades under an admitted drop. It
  is otherwise reported even on a channel with a frame gap, deliberately
  — at that layer a publisher's skip and a lost datagram look identical —
  but an admitted drop is not a judgement call. This is the rule the
  misattribution amplified through: one lost datagram breaks the
  per-instrument chain of every instrument it carried, and a segment
  admitting 663 drops earned 238 findings on this rule alone, of 316
  must violations where a clean control segment produced zero.
• Tier-1 structural rules are untouched, as they are by any other loss.
  Injecting drops into the committed capture leaves its 619
  FRAME.LENGTH_CONSISTENCY and 6 MSG.SNAPSHOT_FLAG_MATCHES_PORT
  violations where they were, and its 38 clean groups as passes.

The total lands on stderr after the end-of-run findings and in the JSON
report's capture_drops, the only place a one-shot CI replay can carry
it. The exit code is deliberately unchanged: a lossy segment is still
worth replaying and the violations it confirms are real — but exit 0 over
a capture that admits loss is not the same claim as exit 0 over one that
does not.

Two boundaries are stated in the README rather than papered over:
transport_loss_total still counts a capture-owned gap, because at the
point the frame-seq hole is seen nothing can say which side lost that
datagram; and a live socket admits nothing, since the kernel's overflow
accounting is not wired into the multicast source.

Tests

input/pcapng_test.go assembles files block by block — a writer library
could not produce the fixture, since none of them emit the option under
test. Covers both byte orders, the drop count, drops carried across
skipped packets and past the last datagram, if_tsresol/if_tsoffset,
unknown blocks, section restarts, and the malformed cases that must fail
rather than read as complete. engine/capture_loss_test.go pins the
downgrade, the control that keeps an unadmitted gap a violation, the
taint's era scope, and Tier-1 immunity. run_test.go replays a pcapng
end to end and checks capture_drops in the report.

Closes #62

[ben-dz]

Three confirmed defects in the new paths. A capture torn at a block-header boundary reads as complete and exits 0 with an empty read_error. The capture-loss taint never expires inside an era, so one admitted drop silences MBP.DELTA.PERINSTR_DENSITY for the rest of the segment and relabels later network loss as capture_loss. A snaplen-truncated packet is decoded as publisher bytes, producing false MUST violations — the same misattribution epb_dropcount was threaded in to prevent. The parser itself held up under 6.5M fuzz executions.

Findings not anchored to the current diff:

• tools/conformance/prometheus/alerts/conformance.yml:48 — Minor — A capture_loss alert tells the operator to investigate the feed, which is the opposite of why the reason was split out. capture_loss joined this alert's reason=~ set but the shared description still ends "Check feed health and instrument definitions", and rule_tests/conformance_test.yml:116 now pins that text as expected output. Branch the description on $labels.reason.

[ben-dz]

Blocking — A capture file torn part-way through a block reads as a complete one, so the run exits 0 with an empty read_error over a segment it only partly read. io.ReadFull returns bare io.EOF when it reads zero bytes, readBlock passes it through, and input/pcap.go:96 turns any io.EOF into a clean end-of-file — reachable whenever the file ends exactly at the start of a block body (verified: 8 bytes of a trailing EPB, and inside a second Section Header Block before the byte-order magic). The comment at :138-141 asserts the opposite property. Fix: past the 8-byte type+length the block is committed, so map io.EOF to io.ErrUnexpectedEOF on both later ReadFulls (:156 too).

[juan-malbeclabs]

Fixed in 017aa58. Both reads past the 8-byte type/length pair now map io.EOF onto io.ErrUnexpectedEOF — :156 as you note, and the body read at :182 — while the header read at :145 is untouched, since that one is the genuine end of file. The comment now describes what the code does rather than what it intended. TestPcapngFileEndingAtABlockBoundaryErrors covers both shapes you verified (a trailing EPB cut to its header, and a second SHB cut before its byte-order magic); it fails on main with the file read as complete.

[ben-dz]

Blocking — The capture-loss taint has no window: nothing clears captureDirty short of an era advance, so one admitted drop suppresses genuine publisher violations for the rest of the segment. Verified: one ObserveCaptureLoss(1), then 500 contiguous frames with contiguous per-instrument sequences, then a real skip 501→552 on a gapless frame series — graded unverifiable/capture_loss, zero violations. Emit's relabelling has the same lifetime: a network gap 260 frames later reports capture_loss, pointing an operator at the recorder for loss the wire caused. MBP.DELTA.PERINSTR_DENSITY was previously gated on no taint at all, so this stickiness is new for it. A drop can only explain the first gap after it — clear the flag once the port's frame series is contiguous again.

[juan-malbeclabs]

Fixed in 017aa58, and the principle you state is the one I implemented — but at two granularities rather than one, because ending it on frame contiguity alone reintroduces the findings this gate exists to prevent.

The label. captureDirty is recomputed per frame and spent on the next frame that advances the series, so a network gap 260 frames later reports loss. dirtyWindow is deliberately not spent with it: that flag answers "could a gap explain this?", which stays true for the era by design (taintOn), while this one answers the narrower "was the recorder's admitted drop what that gap was?".

The density rule. "Clear once the port's frame series is contiguous again" is not a sufficient bound for it. An instrument that updates once every few hundred frames shows its broken chain long after the frame series has recovered, so the drop's own gap is charged to the publisher — most of the 238 findings would come back. I ran that variant against the new test to be sure: TestTheExcuseWaitsForTheInstrumentThatWasBroken fails under it with Per-Instrument Seq jumped 1 -> 3 graded a Violation, exactly as it fails under the sticky flag.

So the density rule's bound is that instrument's own next dense step, which is proof its chain survived the drop. mbpDensityStatus consults the instrument rather than the port window and spends the excuse as it grants it, which also restores the invariant state.go already claimed — that captureDirty gates nothing on its own and only names the owner.

One thing your repro surfaced. Both comparisons are on an epoch counter stamped on each frame at intake, not on the engine's current value. The reorder buffer separates arrival from classification: an admission landing while a frame is buffered belongs to the frame behind it, and comparing against the live counter attributed it to the frame ahead — which arrived before the drop and cannot be what it explains. Without the stamp, TestCaptureLossNamesTheOwnerOfTheLoss breaks under a window-1 engine for that reason alone.

Three tests, all failing on main: the segment is not silenced, the label does not outlive the drop, and the excuse waits for the instrument that was broken and is spent once.

[ben-dz]

Blocking — A capture recorded with a snap length shorter than the feed's frames replays as short datagrams that get charged to the publisher. The EPB declares both captured_len and original_len; origLen is parsed into ci.Length and never read again. Verified: EPBs declaring captured_len 20 bytes below original_len replay to FRAME.SEQ_DUP_DIVERGENT must-violations and exit 1, with capture_drops: 0 and no read error — the same recorder-artifact-charged-to-the-feed error epb_dropcount was threaded in to prevent, from a field this parser already reads. Fix: capLen < origLen means the block does not hold the packet; fail the read or treat it as capture-owned rather than handing the bytes to wire.Decode.

[juan-malbeclabs]

Fixed in 017aa58, taken as capture-owned. One note on where: the check is in PcapSource.Next rather than in readEnhancedPacket, because a legacy pcap declares both lengths too — tcpdump -s 96 produces exactly this defect in either format, and the pcapng-only fix would have left it. A short packet is counted as capture-owned loss and never reaches wire.Decode, so the sequence number it carried goes missing from the series with the taint that follows. SnaplenTruncated() tallies them apart from the admitted drops and stderr names them separately: the remedy is re-recording with a snap length that holds the feed, which is not what an operator does about a dropped datagram. Tests in both formats, both failing on main with the truncated packet yielded as a datagram.

[ben-dz]

Minor — Drops admitted after the last mapped datagram reach capture_drops but never taint anything, so a report can show non-zero drops over a run whose end-of-run findings were graded as if the capture were clean. reportCaptureLoss runs after Flush/EndRun, and flushOpenSnaps (engine/gate.go:1213) emits SNAP.BEGIN_ORDER_END_GROUPING as a Violation unless open.dirty — which only an intra-group snapshot-port gap sets, and at EOF no later frame can show one. Hand the residual to the engine before eng.Flush().

[juan-malbeclabs]

Fixed in 017aa58. PendingDrops goes to the engine before eng.Flush(), as you say. That alone did not reach flushOpenSnaps, though, and it is worth recording why: the gate is the group's own dirty, and ObserveCaptureLoss cannot set it, because the reorder buffer may not have opened the group yet when its loss is admitted — open.dirty is written during classification, which happens inside Flush. So the group records the epoch it opened at and flushOpenSnaps compares it against the run's admission count at the point of judgement (captureLossSince). MBP's end-of-run path needed nothing: flushOpenMBPSnaps already grades every unclosed group unverifiable. Pinned twice — at the engine, and end to end through Run, since the ordering you asked for lives there and nothing else can see it.

[juan-malbeclabs]

All five findings are addressed in 017aa58 — three blocking, both minors, and the unanchored alert-description one. Each has a test that fails without its fix; replies are on the individual threads.

Two places where I did something other than the literal suggestion, both explained inline:

• The taint window. Bounding it on frame-series contiguity alone would charge the drop’s own gap to any instrument that updates less often than every frame, so MBP.DELTA.PERINSTR_DENSITY gets the narrower bound — that instrument’s next dense step. I verified the frame-contiguity variant fails the new test in the same way the sticky flag does.
• The snaplen check lives in PcapSource.Next rather than in readEnhancedPacket, because a legacy pcap declares both lengths too and has the identical defect.

Also worth flagging from working on it: the epoch both bounds compare is stamped on each frame at intake, not read live, because the reorder buffer separates arrival from classification.

Local gates: go test ./..., go vet ./..., golangci-lint run (0 issues), and promtool check rules + test rules.

[ben-dz]

Approved, one new Minor: the snaplen guard keys on declared length alone, so a writer that reports the wire length including the FCS loses every packet and the run passes having checked nothing.

[ben-dz]

Minor — This guard discards any packet where captured_len < original_len, which is true not only for a snap-length cut but whenever the writer reports a wire length including bytes never handed to userspace — the Ethernet FCS being the standard case, which pcapng acknowledges with if_fcslen. Verified: EPBs declaring original_len 4 bytes high with every byte present lose all three packets, and the run exits 0 with an empty report while the warning blames a snap length that was fine. udp.Length - 8 versus len(udp.Payload) separates the two exactly from what is already parsed; discarding on an incomplete UDP payload drops the false positives and still catches every truncation that costs the decoder bytes.

[juan-malbeclabs]

Verified against a real recorder archive, and it works — plus one gap worth deciding on deliberately.

Replayed a five-minute segment written by dz-recorder on a DoubleZero recorder host (live Kalshi perps market-by-price, nanosecond pcapng). This build reads it: read_error empty, 1,860 MBP.SNAP.GROUP_STRUCTURE passes and the reconstruction rule graded throughout. The deployed v0.2.1 dies on the same file with open source: Unknown magic a0d0d0a, which is what #62 was about.

The gap: .pcapng.zst is refused with Unknown magic fd2fb528 — the zstd frame magic. That matters more than it looks, because the recorder's default is compression = "zstd", so every object it publishes is zstd-framed and every archived segment on disk and in any future object store is .pcapng.zst. As it stands the caller has to decompress first, which is fine as a contract but is worth stating rather than discovering: today the error names a magic and leaves the reader to work out that the file needs zstd -d.

If decompress-first is the intended contract, a message naming zstd and saying so would close it. If not, the frame is cheap to detect on the same magic you are already switching on.

One unrelated observation while I had both builds side by side: this branch still shows the MBP.SNAP.RECONSTRUCTED_BOOK_MATCHES_SNAPSHOT violations that #64 removes — 14 on my segment, going to 0 with that PR's build. Expected, since they are independent changes, but it means judging this reader's output on a busy MBP capture will show findings that are not the reader's fault.