Skip to content

fix(bridge): send Record timestamps at nanosecond resolution - #310

Merged
Minipada merged 1 commit into
jazzyfrom
feature/308-record-timestamp-nanoseconds
Aug 10, 2026
Merged

fix(bridge): send Record timestamps at nanosecond resolution#310
Minipada merged 1 commit into
jazzyfrom
feature/308-record-timestamp-nanoseconds

Conversation

@Minipada

@Minipada Minipada commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Closes #308

The Bridge read only msg.header.stamp.sec and dropped .nanosec, then packed it as the ingest protocol's plain integer time. Every Record's timestamp was rounded to the second.

Before / after

Same Memory Measurement at polling_interval: 200 (5 Hz), console Destination:

# before
date=1786275388.0   timestamp=2026-08-09T11:36:28Z   used=98.0783
date=1786275388.0   timestamp=2026-08-09T11:36:28Z   used=98.1031
date=1786275389.0   timestamp=2026-08-09T11:36:29Z   used=98.1441
...
distinct date values: 3 out of 8

# after
date=1786279421087842889   timestamp=2026-08-09T12:43:41.087842889Z   used=90.4075
date=1786279421287875768   timestamp=2026-08-09T12:43:41.287875768Z   used=90.3758
date=1786279421487844760   timestamp=2026-08-09T12:43:41.487844760Z   used=90.3774
...
distinct date values: 8 out of 8, exactly 200 ms apart

The wire format already supported it

Fluent Forward has an EventTime extension (msgpack ext type 0x00, 4 bytes seconds + 4 nanoseconds). I fed both frame shapes to the vendored Vector 0.57.0 fluent source by hand — same source, same config:

integer seconds  ->  "timestamp":"2026-08-09T11:36:28Z"
EventTime ext    ->  "timestamp":"2026-08-09T11:36:28.123456789Z"

So only the Bridge changed: Record carries seconds and nanoseconds, and the Forwarder packs EventTime.

New default time_format: epoch_nanos

double cannot represent nanoseconds — a float64 has ~15-16 significant digits and current epoch seconds already spend 10 of them, so it tops out near microseconds. That is IEEE 754, not an implementation limit. epoch_nanos is an exact i64; double and iso8601 remain available.

Exactness matters beyond precision: it is what makes the timestamp usable as part of a Record identity, which #309 needs. Keying on the old rounded date would have merged genuinely distinct Records above 1 Hz — turning a visible over-counting bug into silent data loss.

Schema follows: date becomes bigint in both init.sql files, to_timestamp(date) becomes to_timestamp(date / 1e9) in the Grafana dashboard, and params writing to those tables drop their explicit time_format: "double". Console-only demos keep theirs. migration.md documents the column change and the ALTER TABLE for anyone already tracking jazzy.

Testing

  • 58 unit tests pass, against a 53-test baseline I ran on pristine origin/jazzy to confirm the delta was mine. 5 new: EventTime bytes for a sub-second stamp; two stamps one nanosecond apart producing different frames; a whole-second stamp still using EventTime; every time_format string parsing; and epoch_nanos rendering with no to_float( anywhere in the VRL.
  • Regression guard in CI: e2e_params.yaml's memory Measurement now polls at 5 Hz specifically so several Records land inside one wall-clock second — at 1 Hz a return to whole-second stamps would not collide and would pass unnoticed. check_timestamp_resolution asserts a nanosecond remainder exists, that some second holds 2+ Records (so it cannot pass vacuously), and that no two Records share an instant.
  • That check was exercised against a real Postgres with the new bigint schema (via podman exec, so no host port needed): good 5 Hz data passes; whole-second data, colliding stamps, 1 Hz-only data and an empty table all fail.

A trap worth flagging for review

Adding timestamp_nanos between timestamp_secs and payload silently broke an aggregate initialisation in forwarder_test.cpp. It still compiled, because nlohmann::json has an implicit conversion to uint32_t — then threw at runtime and took the whole test binary down with terminate called without an active exception, surfacing as a crash in an unrelated backpressure test. The pristine-baseline run is what proved it was mine. Worth knowing if anyone adds fields to Record later.

Not run locally

The full tools/e2e/scripts/run.sh — a rootless-Docker Postgres holds port 5432 on this machine and the harness binds its stores on --network host. CI gates it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A5JwZEZrxEtYJdQUfsZRVo

@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.48%. Comparing base (8c00136) to head (accad12).

Files with missing lines Patch % Lines
dc_bridge/src/bridge_node.cpp 0.00% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            jazzy     #310      +/-   ##
==========================================
+ Coverage   40.35%   40.48%   +0.14%     
==========================================
  Files          82       82              
  Lines        5076     5094      +18     
==========================================
+ Hits         2048     2062      +14     
- Misses       3028     3032       +4     
Flag Coverage Δ
cpp-jazzy 40.48% <66.67%> (+0.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The Bridge read only `msg.header.stamp.sec` and dropped `.nanosec`, then packed
it as the ingest protocol's plain integer time. Every Record's timestamp was
rounded to the second, so a Measurement polling faster than 1 Hz produced
Records that were indistinguishable in time — measured at 5 Hz: 8 Records,
3 distinct timestamps.

The wire format already supported the fix. Fluent Forward has an EventTime
extension (ext type 0x00, 4 bytes seconds + 4 nanoseconds), and the vendored
Vector 0.57.0 parses it with all nine digits intact — verified by feeding its
fluent source both frame shapes. Only the Bridge needed to change: Record now
carries seconds and nanoseconds, and the Forwarder packs EventTime.

Adds `epoch_nanos` as a new time_format, and makes it the default. `double`
cannot represent nanoseconds — a float64 has ~15-16 significant digits and
current epoch seconds spend 10 of them, so it tops out near microseconds. That
is IEEE 754, not an implementation limit. `double` and `iso8601` stay
available. Exactness also matters beyond precision: it is what makes the
timestamp usable as part of a Record identity, which #309 needs.

The schema follows: `date` becomes bigint in both init.sql files,
`to_timestamp(date)` becomes `to_timestamp(date / 1e9)` in the dashboard, and
params writing to those tables drop their explicit `time_format: "double"`.
Console-only demos keep theirs.

Regression cover: e2e_params.yaml's memory Measurement now polls at 5 Hz so
several Records land inside one second, and verify_zero_loss.py asserts a
nanosecond remainder exists, that some second holds 2+ Records (so it cannot
pass vacuously), and that no two Records share an instant. At 1 Hz a return to
whole-second stamps would not collide and would pass unnoticed.

Closes #308

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A5JwZEZrxEtYJdQUfsZRVo
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
@Minipada
Minipada force-pushed the feature/308-record-timestamp-nanoseconds branch from fffa347 to accad12 Compare August 10, 2026 07:14
@Minipada
Minipada merged commit aff253d into jazzy Aug 10, 2026
8 checks passed
@Minipada
Minipada deleted the feature/308-record-timestamp-nanoseconds branch August 16, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant