Add StaticTimestamped generator that uses timestamps from a file to send logs by TheSafo · Pull Request #1687 · DataDog/lading

TheSafo · 2026-01-07T22:32:22Z

What does this PR do?

Adds a new "static" generator that parses timestamps from a log file, and chunks each second worth of logs into a single block.

Motivation

This allows you to e.g. download an hour worth of logs, parse their timestamps, and then (in conjunction w/ prior block-based throttling) send 1 block per second to get a "realistic" load pattern as opposed to generating X bytes/second.

Related issues

Follow up of #1683

TheSafo · 2026-01-07T22:32:51Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 739872f122

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1c39bb28f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75583e6697

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40e6e498ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lading_payload/src/static_timestamped.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b5501b16b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-23T17:55:04Z

lading_payload/src/static_timestamped.rs

+            let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
+                .map_err(|_| Error::Timestamp(line.to_string()))?;
+            let body = remainder
+                .strip_prefix(|c: char| c.is_ascii_whitespace())
+                .unwrap_or(remainder);
+            (
+                Utc.from_utc_datetime(&ts).timestamp(),


Reject or resolve %Z zones instead of forcing them to UTC

In parse_line_with_format, any format that uses %Z can fall through to NaiveDateTime::parse_and_remainder and then be converted with Utc.from_utc_datetime(...). Chrono's %Z parser ignores the zone token rather than producing an offset, so inputs like PST/PDT are accepted but treated as UTC. That breaks real logs that cross a DST abbreviation change or mix named zones: a one-second transition can become a 3601-second gap, or the fall-back transition can be rejected as out-of-order.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-03-23T17:55:04Z

lading_payload/src/static_timestamped.rs

+                Some(s) if s == line_sec => {
+                    // This line is in the same second as the current block add to it.
+                    lines.push(payload);


Stream oversized seconds instead of buffering the entire group

read_next_block() stores every line from the current second in memory before to_bytes() ever sees max_bytes. If a single hot second contains far more data than maximum_block_bytes (for example, hundreds of MB of small log lines in one second), startup still materializes the whole second into Vec<Vec<u8>> and only then truncates it, which can cause very large transient allocations or OOM during cache construction.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b48880d37

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-23T18:04:59Z

lading_payload/src/static_timestamped.rs

+            let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
+                .map_err(|_| Error::Timestamp(line.to_string()))?;


Support chrono formats that omit a full date-time

parse_line_with_format() only falls back to DateTime<FixedOffset> and NaiveDateTime, so formats that do not describe a complete date-time (for example common syslog prefixes like %b %e %H:%M:%S or time-only %H:%M:%S) are rejected as Timestamp errors even though timestamp_format is documented as a general chrono format. In practice this makes the new generator unusable for many real log files unless they already include a full date, which is a surprising limitation for replaying existing logs.

Useful? React with 👍 / 👎.

This was referenced Jan 7, 2026

Update generator throttler to support block-based strategy #1675

Merged

Modify throttle errors to not assume capacity issue #1679

Merged

Add flush_every config option to BufWriter generators #1683

Merged

TheSafo mentioned this pull request Jan 7, 2026

Add deserialization tests for BytesThrottleConfig and LoadProfile #1684

Merged

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 78768d6 to b3de2e0 Compare January 7, 2026 22:59

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 78c5594 to fa19c37 Compare January 15, 2026 21:34

TheSafo force-pushed the jsaf/flush-every-n branch from 8071239 to ff8a57f Compare January 15, 2026 21:34

TheSafo mentioned this pull request Jan 15, 2026

Resize block buffer after allocation if significant wasted space #1674

Merged

TheSafo force-pushed the jsaf/staic-timestamped branch from fa19c37 to 7e1a732 Compare January 15, 2026 21:35

TheSafo force-pushed the jsaf/flush-every-n branch 2 times, most recently from 1be6e74 to 30363c3 Compare January 21, 2026 15:48

TheSafo force-pushed the jsaf/staic-timestamped branch from 7e1a732 to 1098a47 Compare January 21, 2026 15:48

TheSafo force-pushed the jsaf/flush-every-n branch from 30363c3 to 4aba943 Compare January 21, 2026 18:03

TheSafo force-pushed the jsaf/staic-timestamped branch from 1098a47 to 02c10d3 Compare January 21, 2026 18:03

TheSafo force-pushed the jsaf/flush-every-n branch from 4aba943 to b413062 Compare February 4, 2026 19:38

TheSafo force-pushed the jsaf/staic-timestamped branch from 02c10d3 to 9aa2aee Compare February 4, 2026 19:38

TheSafo force-pushed the jsaf/flush-every-n branch from b413062 to 85dfb43 Compare February 9, 2026 15:24

TheSafo force-pushed the jsaf/staic-timestamped branch from 9aa2aee to 49cdb8f Compare February 9, 2026 15:24

TheSafo force-pushed the jsaf/flush-every-n branch from 85dfb43 to fac862b Compare March 5, 2026 15:52

TheSafo force-pushed the jsaf/staic-timestamped branch from 49cdb8f to 9e8ecc9 Compare March 5, 2026 15:52

TheSafo force-pushed the jsaf/flush-every-n branch from fac862b to c5eee9b Compare March 5, 2026 15:59

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 6bc7260 to b173aed Compare March 5, 2026 16:21

TheSafo force-pushed the jsaf/flush-every-n branch from 8e1b11c to 84e040f Compare March 5, 2026 16:38

TheSafo force-pushed the jsaf/staic-timestamped branch from b173aed to 9e7b813 Compare March 5, 2026 16:38

TheSafo changed the title ~~static-timestamped~~ Add generator to parse timestamps from a log-file and send at specified rate Mar 5, 2026

TheSafo force-pushed the jsaf/staic-timestamped branch from 9e7b813 to 72668f1 Compare March 5, 2026 17:35

TheSafo changed the base branch from jsaf/flush-every-n to graphite-base/1687 March 12, 2026 15:50

TheSafo force-pushed the jsaf/staic-timestamped branch from 72668f1 to 7be29f5 Compare March 18, 2026 16:48

TheSafo force-pushed the graphite-base/1687 branch from 84e040f to 021b37b Compare March 18, 2026 16:48

TheSafo changed the base branch from graphite-base/1687 to main March 18, 2026 16:48

TheSafo changed the title ~~Add generator to parse timestamps from a log-file and send at specified rate~~ Add static generator that uses timestamps from a file to send logs at "realistic" rate Mar 18, 2026

TheSafo force-pushed the jsaf/staic-timestamped branch from 7be29f5 to 739872f Compare March 18, 2026 16:58

TheSafo marked this pull request as ready for review March 18, 2026 16:59

TheSafo requested a review from a team as a code owner March 18, 2026 16:59

TheSafo changed the title ~~Add static generator that uses timestamps from a file to send logs at "realistic" rate~~ Add StaticTimestamped generator that uses timestamps from a file to send logs Mar 18, 2026

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from efc7cc8 to f1c39bb Compare March 18, 2026 23:05

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch 2 times, most recently from 6c49d5b to 75583e6 Compare March 18, 2026 23:14

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch from 75583e6 to 40e6e49 Compare March 23, 2026 17:08

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

lading_payload/src/static_timestamped.rs Show resolved Hide resolved

lading_payload/src/static_timestamped.rs Outdated Show resolved Hide resolved

TheSafo force-pushed the jsaf/staic-timestamped branch from 40e6e49 to 7b5501b Compare March 23, 2026 17:43

fix span label and error context in StaticTimestamped; update CHANGELOG

2b48880

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

TheSafo force-pushed the jsaf/staic-timestamped branch from 7b5501b to 2b48880 Compare March 23, 2026 17:57

chatgpt-codex-connector bot reviewed Mar 23, 2026

View reviewed changes

		let (ts, remainder) = NaiveDateTime::parse_and_remainder(line, timestamp_format)
		.map_err(\|_\| Error::Timestamp(line.to_string()))?;

Conversation

TheSafo commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Related issues

Uh oh!

TheSafo commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

TheSafo commented Jan 7, 2026 •

edited

Loading

TheSafo commented Jan 7, 2026 •

edited

Loading