[One Workflow] Restore compact validation telemetry for generate_workflow#271980
Draft
Kiryous wants to merge 1 commit into
Draft
[One Workflow] Restore compact validation telemetry for generate_workflow#271980Kiryous wants to merge 1 commit into
Kiryous wants to merge 1 commit into
Conversation
…flow Closes elastic/security-team#17641. The old low-level edit tools (deleted in elastic#269351) ran a compact validation pass against the generated YAML and surfaced the result via WorkflowsAiTelemetryClient.reportEditResult, which emits validation_passed, validation_error_count, and is_self_correction. The single-entry-point generate_workflow tool stopped doing that, leaving those telemetry fields permanently undefined and removing our ability to track generation-quality regressions. This commit reintroduces the helper, calls api.validateWorkflow against the stringified YAML after generation, and passes the result through to telemetry on the success path. The failure path (generation threw) still has no YAML to validate, so it keeps the existing behavior. Tests cover the valid case, the invalid case (with error-only diagnostic flattening), and the validator-throwing case. Spotted by @yiannisnikolopoulos on elastic#269351 review.
Kiryous
added
Team:One Workflow
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
Contributor
Author
|
/ci |
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes https://github.com/elastic/security-team/issues/17641
Follow-up to #269351 — addresses @yiannisnikolopoulos's review comment.
Summary
The old low-level edit tools (deleted in #269351) ran a compact validation pass against the resulting YAML and surfaced the outcome via
WorkflowsAiTelemetryClient.reportEditResult, emitting:validation_passed: booleanvalidation_error_count: number(when invalid)is_self_correction: boolean(when a previously-failed conversation produced a valid YAML)When
generate_workflowbecame the single entry point,reportEditResultstarted being called without avalidationfield, so those telemetry fields are permanently undefined and we can't track generation-quality regressions over time.This PR reintroduces the helper, calls
api.validateWorkflowagainst the stringified generated YAML, and passes the result through to telemetry on the success path. The failure path (generation threw) still has no YAML to validate, so it keeps the existing behavior.Test plan
generate_workflow.test.tsunit suite (10 cases, including new coverage):validateWorkflowthrowsworkflows_ai_edit_resultevent now carriesvalidation_passedin dev tools (not required for merge, but worth confirming once on a deployed env).