enhancement(checks ipc): add new check data types and harden the protocol - #1855
enhancement(checks ipc): add new check data types and harden the protocol#1855nathan-b wants to merge 5 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
dd-octo-sts
Bot
added
area/io
| // Distinguishes the two sender upcalls. False (the default) | ||
| // routes to `Sender.OpenmetricsBucket`; true routes to | ||
| // `Sender.HistogramBucket`. Python `submit_histogram_bucket` | ||
| // always lands as false. |
There was a problem hiding this comment.
Nit: opaque comment that requires implicit cross-repository knowledge.
It's also... confusing at face value. The naming implies multiple buckets, but this type is, definitionally, a value representing a single bucket.
There was a problem hiding this comment.
This comment (and TBH this field) is a hot mess. It took me several passes and I'm still not sure I got it right, but my next commit should make this better.
Protocol-level messages: - Hello - HelloResp - ConfigData - CheckDataMsg - CheckDataAck - CheckResultMsg Add Histogram and EventPlatformEvent data types to the CheckData oneof. Extend MetricType with MonotonicCount and Historate. Remove gRPC service definition as it is no longer needed.
Restore the AcrIpc gRPC service with four RPCs: - Handshake (unary) - SendCheckData (unary with ACK) - SendCheckResult (unary) - StreamConfig (server streaming for config push). Re-enable server and client codegen for the checks proto compilation.
…init_config extend Log: - Add fields to the Log message to support all the metadata the agent and ADP support - The fields map 1:1 onto ADP's internal Log struct Histogram / sketch: - Renamed histogram.proto to sketch.proto and replaced its contents with the DDSketch wire format used by the agent's SketchPayload - CheckData oneof slot 5 is repurposed to carry pre-aggregated sketches Note: lib/saluki-components/src/sources/checks_ipc/mod.rs is left broken — it still imports the old Checks service. init_config: - The check API has both `config` and `init_config`, and many integrations rely on the init_config block for shared defaults - The previous CheckInstance message only carried the per-instance config, so an ACR-scheduled check would receive an empty init_config openmetrics: - Add support for the new openmetrics histogram bucket type
- Improve some AI-authored code comments - Change protobuf message naming to fit request / response pattern - Change name of AcrIpc to read a bit more naturally - Don't hard-code server ID
40ce8a8 to
d342d92
Compare
Binary Size Analysis (Agent Data Plane)Baseline: e448977 · Comparison: c319fb0 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (5)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
nathan-b
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c319fb098b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut out = Log::new(log.message) | ||
| .with_status(status) | ||
| .with_source(string_to_meta_opt(log.source)) | ||
| .with_hostname(string_to_meta_opt(log.hostname)) |
There was a problem hiding this comment.
When a check runner leaves Log.hostname empty, the new proto contract says the receiver should use its default hostname, but this conversion maps the empty string to None and Log::with_hostname(None) stores an empty host. In that common host-omitted case, logs sent through checks IPC lose the agent hostname instead of matching metric behavior, which can break host-scoped routing and correlation downstream.
Useful? React with 👍 / 👎.
| accepted: true, | ||
| reject_reason: String::new(), |
There was a problem hiding this comment.
If a runner sends PROTOCOL_VERSION_UNSPECIFIED or an unknown/future protocol integer, the handshake still returns accepted: true with no reject reason. Because SendCheckData is not gated on a negotiated version later, an incompatible runner can continue after a nominally successful handshake; the new accepted/reject_reason fields should be used to reject anything other than the supported V1 version.
Useful? React with 👍 / 👎.
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
log_to_event maps every new Log proto field (source, hostname, service, tags, additional_properties) except timestamp — since the saluki Log struct has no timestamp field, any non-zero producer timestamp from ACR is silently discarded and the log is always stamped at receive time. Separately, send_check_data returns success: true for batches containing only unsupported types (Sketch, MonotonicCount, etc.), giving callers no signal that their data was dropped.
🤖 Datadog Autotest · Commit c319fb0 · What is Autotest? · Any feedback? Reach out in #autotest
| .with_source(string_to_meta_opt(log.source)) | ||
| .with_hostname(string_to_meta_opt(log.hostname)) | ||
| .with_service(string_to_meta_opt(log.service)) | ||
| .with_tags(Some(proto_tags_to_tagset(log.tags))); |
There was a problem hiding this comment.
Logs from ACR checks appear in Datadog at the time ADP received them, not when the check event occurred. For batched or replayed logs this causes incorrect temporal ordering in the log explorer.
Assertion details
- Input: ACR sends a Log payload with
timestamp = 1_700_000_000_000_000_000(nanoseconds) and any non-empty message - Expected: The saluki
Logevent carries the producer-set timestamp so Datadog shows the event at the time the check detected it - Actual:
log_to_eventreadslog.level,log.message,log.source,log.hostname,log.service,log.tags, andlog.additional_properties— it never accesseslog.timestamp. The salukiLogstruct has no timestamp field. All non-zero producer timestamps are discarded; every log emitted from this path uses pipeline receive time.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · Any feedback? Reach out in #autotest
| } | ||
|
|
||
| Ok(Response::new(SendCheckPayloadResponse {})) | ||
| Ok(Response::new(SendCheckDataResponse { |
There was a problem hiding this comment.
ACR clients that start emitting DDSketch or histogram-bucket payloads (as the new proto now allows) will receive success: true indefinitely with no indication that their data is being dropped server-side, making the data gap invisible until a separate cardinality check.
Assertion details
- Input: A
SendCheckDataRequestbatch whose everyCheckDataitem is one of:Sketch,EventPlatformEvent,HistogramBucket,MonotonicCountmetric, orHistoratemetric - Expected:
success: falseor a non-emptyerrorfield to signal that no data was forwarded, so clients can log/alert or retry with a different code path - Actual:
data_to_eventreturnsNonefor all five types (with a debug-level log). The iteration loopcontinues past each one, and the function falls through toOk(SendCheckDataResponse { success: true, error: "" }). Clients receive a success ACK with zero items actually processed.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · Any feedback? Reach out in #autotest
2 participants
Summary
Implement improvements to the IPC protocol between ADP and ACR.
Adds new types (sketches for histograms, the new openmetrics histogram bucket, extend log type). Also tries to make the protocol more robust and build failure detection into it.
How did you test this PR?
Ran ACR against both ADP and the core agent to confirm data types are correctly sent