fix(bridge): send Record timestamps at nanosecond resolution - #310
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## jazzy #310 +/- ##
==========================================
+ Coverage 40.35% 40.48% +0.14%
==========================================
Files 82 82
Lines 5076 5094 +18
==========================================
+ Hits 2048 2062 +14
- Misses 3028 3032 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Aug 9, 2026
The Bridge read only `msg.header.stamp.sec` and dropped `.nanosec`, then packed it as the ingest protocol's plain integer time. Every Record's timestamp was rounded to the second, so a Measurement polling faster than 1 Hz produced Records that were indistinguishable in time — measured at 5 Hz: 8 Records, 3 distinct timestamps. The wire format already supported the fix. Fluent Forward has an EventTime extension (ext type 0x00, 4 bytes seconds + 4 nanoseconds), and the vendored Vector 0.57.0 parses it with all nine digits intact — verified by feeding its fluent source both frame shapes. Only the Bridge needed to change: Record now carries seconds and nanoseconds, and the Forwarder packs EventTime. Adds `epoch_nanos` as a new time_format, and makes it the default. `double` cannot represent nanoseconds — a float64 has ~15-16 significant digits and current epoch seconds spend 10 of them, so it tops out near microseconds. That is IEEE 754, not an implementation limit. `double` and `iso8601` stay available. Exactness also matters beyond precision: it is what makes the timestamp usable as part of a Record identity, which #309 needs. The schema follows: `date` becomes bigint in both init.sql files, `to_timestamp(date)` becomes `to_timestamp(date / 1e9)` in the dashboard, and params writing to those tables drop their explicit `time_format: "double"`. Console-only demos keep theirs. Regression cover: e2e_params.yaml's memory Measurement now polls at 5 Hz so several Records land inside one second, and verify_zero_loss.py asserts a nanosecond remainder exists, that some second holds 2+ Records (so it cannot pass vacuously), and that no two Records share an instant. At 1 Hz a return to whole-second stamps would not collide and would pass unnoticed. Closes #308 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A5JwZEZrxEtYJdQUfsZRVo Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Minipada
force-pushed
the
feature/308-record-timestamp-nanoseconds
branch
from
August 10, 2026 07:14
fffa347 to
accad12
Compare
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #308
The Bridge read only
msg.header.stamp.secand dropped.nanosec, then packed it as the ingest protocol's plain integer time. Every Record's timestamp was rounded to the second.Before / after
Same
MemoryMeasurement atpolling_interval: 200(5 Hz), console Destination:The wire format already supported it
Fluent Forward has an EventTime extension (msgpack ext type
0x00, 4 bytes seconds + 4 nanoseconds). I fed both frame shapes to the vendored Vector 0.57.0fluentsource by hand — same source, same config:So only the Bridge changed:
Recordcarries seconds and nanoseconds, and the Forwarder packs EventTime.New default
time_format: epoch_nanosdoublecannot represent nanoseconds — a float64 has ~15-16 significant digits and current epoch seconds already spend 10 of them, so it tops out near microseconds. That is IEEE 754, not an implementation limit.epoch_nanosis an exact i64;doubleandiso8601remain available.Exactness matters beyond precision: it is what makes the timestamp usable as part of a Record identity, which #309 needs. Keying on the old rounded
datewould have merged genuinely distinct Records above 1 Hz — turning a visible over-counting bug into silent data loss.Schema follows:
datebecomesbigintin bothinit.sqlfiles,to_timestamp(date)becomesto_timestamp(date / 1e9)in the Grafana dashboard, and params writing to those tables drop their explicittime_format: "double". Console-only demos keep theirs.migration.mddocuments the column change and theALTER TABLEfor anyone already trackingjazzy.Testing
origin/jazzyto confirm the delta was mine. 5 new: EventTime bytes for a sub-second stamp; two stamps one nanosecond apart producing different frames; a whole-second stamp still using EventTime; everytime_formatstring parsing; andepoch_nanosrendering with noto_float(anywhere in the VRL.e2e_params.yaml'smemoryMeasurement now polls at 5 Hz specifically so several Records land inside one wall-clock second — at 1 Hz a return to whole-second stamps would not collide and would pass unnoticed.check_timestamp_resolutionasserts a nanosecond remainder exists, that some second holds 2+ Records (so it cannot pass vacuously), and that no two Records share an instant.bigintschema (viapodman exec, so no host port needed): good 5 Hz data passes; whole-second data, colliding stamps, 1 Hz-only data and an empty table all fail.A trap worth flagging for review
Adding
timestamp_nanosbetweentimestamp_secsandpayloadsilently broke an aggregate initialisation inforwarder_test.cpp. It still compiled, becausenlohmann::jsonhas an implicit conversion touint32_t— then threw at runtime and took the whole test binary down withterminate called without an active exception, surfacing as a crash in an unrelated backpressure test. The pristine-baseline run is what proved it was mine. Worth knowing if anyone adds fields toRecordlater.Not run locally
The full
tools/e2e/scripts/run.sh— a rootless-Docker Postgres holds port 5432 on this machine and the harness binds its stores on--network host. CI gates it.🤖 Generated with Claude Code
https://claude.ai/code/session_01A5JwZEZrxEtYJdQUfsZRVo