Skip to content

feat(cli): time CDC transform stage separately from send - #188

Open
aatran14 wants to merge 1 commit into
a24films:mainfrom
aatran14:at/measure-embed-time
Open

feat(cli): time CDC transform stage separately from send#188
aatran14 wants to merge 1 commit into
a24films:mainfrom
aatran14:at/measure-embed-time

Conversation

@aatran14

Copy link
Copy Markdown

why

The transform/embed stage is the dominant cost of interest in the CDC pipeline i.e. the embedding round trip. But currently it has no means of being recorded.

So far the system does an excellent job of keeping the measurements simple and concise like:

  • whole-batch duration (which is just total time)
  • turbopuffer latency
  • replication lag

This PR entertains the integration of transform cost. Thus, we could measure transform / total time to measure how much the transform is soaking the CDC pipeline.

add

  • observability.rs: cdc_transform_duration: Histogram<f64> on Metrics, built as
    puffgres.cdc.transform_duration_ms in build_metrics
  • streaming.rs: wrap transform_batch in process_config_events with
    Instant::now() and record elapsed ms into that histogram

Live CDC only. Backfill is not covered yet (core::run_backfill takes no metrics param). Observability only, so no behavior change.

testing stuff

  • cargo check -p puffgres-cli and cargo test -p puffgres-cli
  • With OTLP enabled, confirm transform_duration_ms appears alongside the other CDC metrics
  • the dumbest sanity test you can do on your machine is verify the the signal measures = ~0 under --provider none. The reason why this check make sense is when a real embedding provider is wired up, the transform makes the network call to the embedding model, which takes O(us)-O(ms) depending on regional overhead. Decoding the row is microseconds (us). The turbopuffer write is also a network call and is already measured via turbopuffer_latency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant