Skip to content

Add ability to tune per-log batching behavior #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lukevalenta merged 14 commits into from
Jun 5, 2025
Merged

Add ability to tune per-log batching behavior #43

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e9d76c0
Specify sequence interval in milliseconds instead of seconds
lukevalenta May 22, 2025
0f92d2a
Allow skipping checkpoint creation when there are no entries to sequence
lukevalenta May 22, 2025
a16ee94
Add batcher parameters to log configuration, and allow setting zero b…
lukevalenta May 22, 2025
859df92
Add option to disable deduplication checks
lukevalenta May 27, 2025
a053648
Add ability to set a maximum timeout for skipped entries
lukevalenta Jun 2, 2025
f721b0c
Add breakdown of full vs partial tiles to metrics
lukevalenta Jun 3, 2025
c913c38
Fix comment about dropped channels
lukevalenta May 22, 2025
cd3f536
Address @cjpatton's comments
lukevalenta Jun 5, 2025
458002e
Address @rozbb's comments and use const generics for serde default fns
lukevalenta Jun 5, 2025
5e6d030
Rename leftover_pending to leftover_dedup for clarity and consistency
lukevalenta Jun 5, 2025
3940b8d
Clarify that leftover_timestamps_next_slot is reduce modulo tile width
lukevalenta Jun 5, 2025
0500548
Set minimum value in schema for sequence_skip_threshold_millis
lukevalenta Jun 5, 2025
93ea29a
Fix typo in enable_dedup description
lukevalenta Jun 5, 2025
2736e1c
Set minimum value of zero for max_sequence_skips
lukevalenta Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions crates/ct_worker/src/ctlog.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,15 +707,21 @@ async fn sequence_entries<L: LogEntryTrait>(
// If the data tile is full, stage it.
if n % u64::from(TlogTile::FULL_WIDTH) == 0 {
stage_data_tile(n, &mut edge_tiles, &mut tile_uploads, &data_tile);
metrics.seq_data_tile_size.observe(data_tile.len().as_f64());
metrics
.seq_data_tile_size
.with_label_values(&["full"])
.observe(data_tile.len().as_f64());
data_tile.clear();
}
}

// Stage leftover partial data tile, if any.
if n != old_size && n % u64::from(TlogTile::FULL_WIDTH) != 0 {
stage_data_tile(n, &mut edge_tiles, &mut tile_uploads, &data_tile);
metrics.seq_data_tile_size.observe(data_tile.len().as_f64());
metrics
.seq_data_tile_size
.with_label_values(&["partial"])
.observe(data_tile.len().as_f64());
}

// Produce and stage new tree tiles.
Expand Down
5 changes: 3 additions & 2 deletions crates/ct_worker/src/metrics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ pub(crate) struct Metrics {
pub(crate) seq_delay: Histogram,
pub(crate) seq_leaf_size: Histogram,
pub(crate) seq_tiles: Counter,
pub(crate) seq_data_tile_size: Histogram,
pub(crate) seq_data_tile_size: HistogramVec,

pub(crate) tree_time: Gauge,
pub(crate) tree_size: Gauge,
Expand Down Expand Up @@ -114,9 +114,10 @@ impl Metrics {
r
)
.unwrap();
let seq_data_tile_size = register_histogram_with_registry!(
let seq_data_tile_size = register_histogram_vec_with_registry!(
"sequencing_data_tiles_bytes",
"Size of uploaded data tiles, including partials.",
&["type"],
vec![10_000.0, 100_000.0, 1_000_000.0],
r
)
Expand Down