Skip to content

feat: envoyproxy health check event logging - #9381

Open
guydc wants to merge 21 commits into
envoyproxy:mainfrom
guydc:feat-hc-logs-fresh
Open

feat: envoyproxy health check event logging#9381
guydc wants to merge 21 commits into
envoyproxy:mainfrom
guydc:feat-hc-logs-fresh

Conversation

@guydc

@guydc guydc commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Adds support for configuring Envoy health check event logging via the EnvoyProxy API. Health check events (probe outcomes) can be written as JSON to a file sink (defaults to /dev/stdout).

The new healthCheckLog field under spec.telemetry of EnvoyProxy controls where logs are printed to and which probe outcomes are logged:

  • Failure - any HC failure
  • FailureSeriesStart - the first failure after previous success
  • Success - any HC success
  • HealthyTransition - the first success after previous failures AND the transition to healthy state (crossing health threshold)

Internally, the event log config is carried on ir.ActiveHealthCheck.EventLog so it flows naturally through TrafficFeatures into every cluster that has active health checks configured, including filter-created clusters (ext-proc, extauth, etc.) without any extra threading.

Closes #8753

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #


PR Checklist

  • Authorship & ownership: Coding agents / AI assistants are welcome, but I have reviewed every change, understand how and why it works, can explain and maintain it, and take full responsibility for this PR. I have not submitted generated output I do not understand.
  • DCO: All commits are signed off (git commit -s). See DCO: Sign your work.
  • API agreed first: If this PR contains API changes (changes under /api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes.
  • Required checks pass: make generate gen-check, make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, but gen-check, lint, and coverage MUST pass.)
  • Tests added/updated: New/changed code is covered by appropriate tests. N/A if this PR does not contain code changes.
  • Docs: User-facing changes update the docs, either in this PR or a follow-up PR. N/A if this PR does not contain user-facing changes.
  • Release notes: For any non-trivial change, added a release-note fragment under release-notes/current/<section>/<pr-number>-<slug>.md (see release-notes/current/README.md for sections and naming). N/A if this PR does not contain non-trivial changes.
  • Generated files committed: Ran make gen-check and committed the result if API/helm charts/modules changed.
  • Scope & compatibility: The PR is reasonably scoped (no unrelated changes) and preserves backward compatibility, or any breaking change is called out above and documented in release-notes/current/breaking_changes/.
  • Codex review: Requested a Codex review and addressed all of its comments.
  • Copilot review: Requested a Copilot review and addressed all of its comments.

@netlify

netlify Bot commented Jul 1, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 failed.

Name Link
🔨 Latest commit 9f09627
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a6aa66a3701550008e4889c

@guydc guydc mentioned this pull request Jul 1, 2026
11 tasks
@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.52174% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.71%. Comparing base (b499c27) to head (9f09627).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
internal/gatewayapi/backendtrafficpolicy.go 69.69% 7 Missing and 3 partials ⚠️
internal/xds/translator/cluster.go 78.94% 2 Missing and 2 partials ⚠️
internal/gatewayapi/securitypolicy.go 0.00% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9381      +/-   ##
==========================================
+ Coverage   75.68%   75.71%   +0.03%     
==========================================
  Files         254      254              
  Lines       42110    42174      +64     
==========================================
+ Hits        31870    31933      +63     
+ Misses       8093     8092       -1     
- Partials     2147     2149       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Adds support for configuring Envoy health check event logging via the
EnvoyProxy API. Health check events (probe outcomes) can be written as
JSON to a file sink (defaults to /dev/stdout).

The new `healthCheckLog` field under `spec.telemetry` of EnvoyProxy
controls which probe outcomes are logged (Failure, FailureTransition,
Success, SuccessTransition) and where the events are written.

Internally, the event log config is carried on ir.ActiveHealthCheck.EventLog
so it flows naturally through TrafficFeatures into every cluster that has
active health checks configured, including filter-created clusters
(ext-proc, extauth, etc.) without any extra threading.

Closes envoyproxy#8753

Signed-off-by: Guy Daich <guy.daich@sap.com>
@guydc
guydc force-pushed the feat-hc-logs-fresh branch from 54564b6 to ebd31ed Compare July 10, 2026 16:38
@guydc
guydc requested a review from Copilot July 10, 2026 16:50
@guydc

guydc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@codex

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Envoy health check event logging support to Envoy Gateway, exposing a new spec.telemetry.healthCheckLog field on the EnvoyProxy API and threading that configuration through IR into xDS HealthCheck.event_logger so that clusters with active health checks can emit JSON probe outcome events to a file sink (default /dev/stdout).

Changes:

  • Introduces ProxyTelemetry.healthCheckLog API types, CRD schema/CEL validations, and CEL validation tests.
  • Translates healthCheckLog into IR (ir.ActiveHealthCheck.EventLog) and renders corresponding Envoy xDS health check eventLogger configuration.
  • Adds docs, release notes, golden testdata updates, and e2e coverage (including ext-proc and backend HC logging assertions).

No concrete defects were identified in the reviewed diffs; remaining risk is primarily around correctness of the Envoy “matches → alwaysLog* booleans” semantic mapping and operational/log-volume effects in real deployments.

Reviewed changes

Copilot reviewed 38 out of 50 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/helm/gateway-crds-helm/envoy-gateway-crds.out.yaml Updates expected CRD schema output to include telemetry.healthCheckLog.
test/helm/gateway-crds-helm/e2e.out.yaml Updates expected e2e CRD schema output to include telemetry.healthCheckLog.
test/helm/gateway-crds-helm/all.out.yaml Updates expected “all” CRD schema output to include telemetry.healthCheckLog.
test/e2e/tests/ext_proc.go Adds e2e assertion that ext-proc cluster health check events appear in logs (via Loki).
test/e2e/tests/backend_health_check.go Adds e2e test validating health check event logging behavior via Loki.
test/e2e/testdata/ext-proc-envoyextensionpolicy.yaml Enables active gRPC health checks for ext-proc backend to generate events.
test/e2e/testdata/backend-health-check-event-log.yaml Adds route manifest used as a baseline phase for event-log test.
test/e2e/testdata/backend-health-check-event-log-hc.yaml Adds HC-enabled route + failing BTP to reliably generate HC log events.
test/config/gatewayclass.yaml Enables telemetry.healthCheckLog: {} in test GatewayClass config.
test/cel-validation/envoyproxy_test.go Adds CEL validation coverage for ProxyHealthCheckLog constraints.
site/content/en/latest/tasks/observability/proxy-health-check-log.md Adds user documentation for configuring and verifying HC event logs.
site/content/en/latest/api/extension_types.md Adds generated API docs for new health check logging types/fields.
release-notes/current/new_features/9381-envoyproxy-health-check-event-logging.md Adds release note fragment for the new feature.
internal/xds/translator/testdata/out/xds-ir/health-check-event-log.routes.yaml Adds golden output for routes in HC event log translator scenario.
internal/xds/translator/testdata/out/xds-ir/health-check-event-log.listeners.yaml Adds golden output for listeners in HC event log translator scenario.
internal/xds/translator/testdata/out/xds-ir/health-check-event-log.endpoints.yaml Adds golden output for endpoints in HC event log translator scenario.
internal/xds/translator/testdata/out/xds-ir/health-check-event-log.clusters.yaml Adds golden output demonstrating xDS health check eventLogger rendering.
internal/xds/translator/testdata/out/xds-ir/ext-proc-with-health-check-log.routes.yaml Adds golden output for ext-proc route scenario with HC logging.
internal/xds/translator/testdata/out/xds-ir/ext-proc-with-health-check-log.listeners.yaml Adds golden output for ext-proc listener scenario with HC logging.
internal/xds/translator/testdata/out/xds-ir/ext-proc-with-health-check-log.endpoints.yaml Adds golden output for ext-proc endpoints scenario with HC logging.
internal/xds/translator/testdata/out/xds-ir/ext-proc-with-health-check-log.clusters.yaml Adds golden output for ext-proc cluster health check logging config.
internal/xds/translator/testdata/in/xds-ir/health-check-event-log.yaml Adds IR input fixture for HC event log translation.
internal/xds/translator/testdata/in/xds-ir/ext-proc-with-health-check-log.yaml Adds IR input fixture for ext-proc HC log translation.
internal/xds/translator/cluster.go Implements xDS health check eventLogger rendering from IR event log config.
internal/ir/zz_generated.deepcopy.go Adds deepcopy support for new IR event log types.
internal/ir/xds.go Adds IR structs for health check event logging and wires into ActiveHealthCheck.
internal/gatewayapi/testdata/securitypolicy-with-extauth-with-hc-log.out.yaml Adds/updates gatewayapi golden output showing HC log config propagated into IR.
internal/gatewayapi/testdata/securitypolicy-with-extauth-with-hc-log.in.yaml Adds/updates gatewayapi input fixture enabling HC logging via EnvoyProxy telemetry.
internal/gatewayapi/testdata/health-check-log-gw-override.out.yaml Adds golden coverage for gateway-level override interactions with HC logging.
internal/gatewayapi/testdata/health-check-log-gw-override.in.yaml Adds input fixture for gateway override scenario.
internal/gatewayapi/testdata/health-check-log-disabled.out.yaml Adds golden coverage for disabled/default HC logging scenarios.
internal/gatewayapi/testdata/health-check-log-disabled.in.yaml Adds input fixture for disabled scenario.
internal/gatewayapi/testdata/health-check-log-defaults.out.yaml Adds golden coverage for default HC logging behavior.
internal/gatewayapi/testdata/health-check-log-defaults.in.yaml Adds input fixture for defaults scenario.
internal/gatewayapi/testdata/envoyextensionpolicy-with-extproc-with-hc-log.out.yaml Adds golden coverage for ext-proc cluster HC logging propagation.
internal/gatewayapi/testdata/envoyextensionpolicy-with-extproc-with-hc-log.in.yaml Adds input fixture for ext-proc HC logging propagation.
internal/gatewayapi/securitypolicy.go Threads gateway telemetry into backend traffic feature translation for policy-created clusters.
internal/gatewayapi/listener.go Adds translateHealthCheckLog helper and updates traffic translation signature usage.
internal/gatewayapi/listener_test.go Adds unit tests for translateHealthCheckLog mapping behavior.
internal/gatewayapi/envoyextensionpolicy.go Threads gateway telemetry into ext-proc backend settings translation.
internal/gatewayapi/contexts.go Adds GatewayContext.Telemetry() helper for safe telemetry access.
internal/gatewayapi/clustersettings.go Propagates telemetry HC log config into IR active health check event log.
internal/gatewayapi/backendtrafficpolicy.go Threads gateway telemetry into BackendTrafficPolicy translation (including route-targeted path).
internal/gatewayapi/backendtrafficpolicy_test.go Updates tests for new buildTrafficFeatures signature.
examples/grpc-ext-proc/main.go Adds gRPC health service implementation to support GRPC health checking.
charts/gateway-helm/charts/crds/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml Updates generated CRD with new healthCheckLog schema.
charts/gateway-crds-helm/templates/generated/gateway.envoyproxy.io_envoyproxies.yaml Updates Helm template CRD with new healthCheckLog schema.
api/v1alpha1/zz_generated.deepcopy.go Adds deepcopy support for new API types.
api/v1alpha1/envoyproxy_types.go Adds ProxyTelemetry.HealthCheckLog field to EnvoyProxy API.
api/v1alpha1/envoyproxy_healthchecklogging_types.go Introduces API types/enums/validations for health check event logging.
Files not reviewed (2)
  • api/v1alpha1/zz_generated.deepcopy.go: Generated file
  • internal/ir/zz_generated.deepcopy.go: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cdc7b9198e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/backendtrafficpolicy.go Outdated
Signed-off-by: Guy Daich <guy.daich@sap.com>
@guydc

guydc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@codex

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 50 changed files in this pull request and generated 2 comments.

Files not reviewed (2)
  • api/v1alpha1/zz_generated.deepcopy.go: Generated file
  • internal/ir/zz_generated.deepcopy.go: Generated file

Comment thread internal/gatewayapi/listener.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8b5ecba6a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/listener.go Outdated
@arkodg arkodg added this to the v1.9.0-rc.1 Release milestone Jul 12, 2026
guydc added 5 commits July 13, 2026 20:02
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
@guydc

guydc commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8862378600

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/listener.go Outdated
@guydc

guydc commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@guydc

guydc commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e329d9259

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread site/content/en/latest/tasks/observability/proxy-health-check-log.md Outdated
guydc added 2 commits July 28, 2026 14:55
Signed-off-by: Guy Daich <guy.daich@sap.com>
@guydc
guydc marked this pull request as ready for review July 28, 2026 22:30
@guydc
guydc requested a review from a team as a code owner July 28, 2026 22:30

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfecabeb30

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/backendtrafficpolicy.go Outdated
guydc added 5 commits July 28, 2026 19:40
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
zirain
zirain previously approved these changes Jul 29, 2026
guydc added 3 commits July 29, 2026 17:28
…icsfor empty list

Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants