test: add nightly e2e acceptance test for declarative config builder#724
test: add nightly e2e acceptance test for declarative config builder#724Pittu-Sharma wants to merge 9 commits into
Conversation
✅ Deploy Preview for otel-ecosystem-explorer ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Pittu-Sharma
force-pushed
the
feat/issue-662
branch
from
242c692 to
891a354
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds a scheduled end-to-end acceptance test pipeline intended to validate YAML output from the Java Agent Declarative Configuration Builder by extracting a generated config via Playwright and running an OATS-based container smoke test. The PR also introduces a sizable, separate “Java Agent release comparison” feature (diff logic + UI) and includes regenerated Java Agent configuration data.
Changes:
- Add a nightly GitHub Actions workflow that builds the explorer, generates a declarative config via Playwright, downloads the latest javaagent, and runs OATS assertions in Docker.
- Add acceptance-test assets (Playwright automation script + docker-compose + oats.yaml) to support the declarative-config builder validation.
- Add Java Agent release comparison utilities, hook, UI components, and tests (plus related javaagent data updates).
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
ecosystem-explorer/src/features/java-agent/utils/release-diff.ts |
New release diff computation utilities for Java Agent instrumentations/telemetry/configs. |
ecosystem-explorer/src/features/java-agent/utils/release-diff.test.ts |
Vitest coverage for the new release diff logic. |
ecosystem-explorer/src/features/java-agent/java-release-comparison-page.tsx |
Replaces the prior skeleton with a full release comparison UI (tabs, changelog links, metrics table). |
ecosystem-explorer/src/features/java-agent/hooks/use-release-comparison.ts |
New hook to fetch two versions and compute release diffs. |
ecosystem-explorer/src/features/java-agent/components/release-comparison/release-version-selector.tsx |
Version selection UI for the comparison page. |
ecosystem-explorer/src/features/java-agent/components/release-comparison/instrumentation-diff-card.tsx |
Card UI to show per-instrumentation changes and expand to telemetry/config diffs. |
ecosystem-explorer/scripts/generate-test-config.mjs |
New Playwright script that serves dist/, drives the builder UI, and writes test-config.yaml. |
ecosystem-explorer/public/data/javaagent/global-configurations.json |
Updated/generated javaagent global configuration inventory consumed by the frontend. |
ecosystem-automation/v1-registry-sync/tests/test_reporter.py |
Formatting cleanup. |
ecosystem-automation/v1-registry-sync/tests/test_reader.py |
Formatting cleanup (line wrapping). |
ecosystem-automation/java-instrumentation-watcher/tests/test_java_instrumentation_client.py |
Adds a security lint suppression comment for a test token. |
ecosystem-automation/explorer-db-builder/tests/test_declarative_name_corrections.py |
Formatting cleanup. |
ecosystem-automation/explorer-db-builder/src/explorer_db_builder/declarative_name_corrections.py |
Formatting cleanup in schema injection block. |
acceptance-tests/declarative-config-acceptance/oats.yaml |
OATS test definition asserting a /greeting span and route attribute. |
acceptance-tests/declarative-config-acceptance/docker-compose.yml |
Compose file wiring the smoke-test app with the generated config and javaagent jar. |
.markdownlint-cli2.yaml |
Ignores ecosystem-explorer/dist/** from markdownlint. |
.gitignore |
Ignores generated acceptance-test artifacts (opentelemetry-javaagent.jar, test-config.yaml). |
.github/workflows/nightly-declarative-config-test.yml |
New scheduled + manual workflow that runs the acceptance pipeline and opens an issue on failure. |
.github/scripts/build-screenshots-comment.py |
Minor string/formatting refactor in screenshot comment builder. |
Pittu-Sharma
force-pushed
the
feat/issue-662
branch
from
891a354 to
8c938d3
Compare
jaydeluca
left a comment
jaydeluca
left a comment
There was a problem hiding this comment.
I tried testing this out locally and it's not working. I worked through it to investigate why and found a few issues that need to be fixed before this can be merged. Details inline.
Reference for a working OATS setup that mirrors this use case: opentelemetry-java-examples / javaagent-declarative-configuration/oats and its runner .mise/tasks/oats-tests.sh.
| image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-spring-boot:latest | ||
| environment: | ||
| JAVA_TOOL_OPTIONS: "-javaagent:/app/opentelemetry-javaagent.jar" | ||
| OTEL_JAVAAGENT_CONFIGURATION_FILE: "/app/test-config.yaml" |
There was a problem hiding this comment.
OTEL_JAVAAGENT_CONFIGURATION_FILE points the agent at a .properties file, not the declarative YAML — so the agent ignores test-config.yaml and falls back to env-var autoconfig (OTEL_EXPORTER_OTLP_ENDPOINT below makes spans flow regardless). I confirmed this by putting garbage in the config and the test still passed. Use OTEL_CONFIG_FILE (otel.config.file), which is what the generated YAML's own header tells you to use:
| OTEL_JAVAAGENT_CONFIGURATION_FILE: "/app/test-config.yaml" | |
| OTEL_CONFIG_FILE: "/app/test-config.yaml" |
| version: "3" | ||
| services: | ||
| app: | ||
| image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-spring-boot:latest |
There was a problem hiding this comment.
:latest doesn't exist for this image (manifest unknown at runtime) — ghcr only publishes jdkNN-<timestamp> tags. Pin a real one:
| image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-spring-boot:latest | |
| image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-spring-boot:jdk17-20260623.27999598977 |
| @@ -0,0 +1,15 @@ | |||
| version: "3" | |||
There was a problem hiding this comment.
version: "3" is obsolete under Compose v2 and emits a warning — safe to drop.
| - trace: | ||
| spans: | ||
| - name: "GET /greeting" | ||
| attributes: | ||
| http.route: "/greeting" |
There was a problem hiding this comment.
This isn't valid OATS v2 schema — the binary panics with field trace not found in type ExpectedTraces. Use a traceql selector with equals/attributes (OATS):
| - trace: | |
| spans: | |
| - name: "GET /greeting" | |
| attributes: | |
| http.route: "/greeting" | |
| - traceql: '{ span.http.route = "/greeting" }' | |
| equals: "GET /greeting" | |
| attributes: | |
| http.route: "/greeting" |
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| AGENT_URL=$(gh release view --repo open-telemetry/opentelemetry-java-instrumentation latest --json assets -q '.assets[] | select(.name == "opentelemetry-javaagent.jar") | .url') |
There was a problem hiding this comment.
gh release view … latest reads latest as a tag, which this repo doesn't have — the step fails with release not found, so $AGENT_URL is empty. Omit the arg (no tag = latest release):
| AGENT_URL=$(gh release view --repo open-telemetry/opentelemetry-java-instrumentation latest --json assets -q '.assets[] | select(.name == "opentelemetry-javaagent.jar") | .url') | |
| AGENT_URL=$(gh release view --repo open-telemetry/opentelemetry-java-instrumentation --json assets -q '.assets[] | select(.name == "opentelemetry-javaagent.jar") | .url') |
(Or simpler, replace the curl entirely with gh release download --repo … --pattern 'opentelemetry-javaagent.jar' --dir acceptance-tests/declarative-config-acceptance.)
| - name: Run OATS acceptance tests | ||
| run: | | ||
| cd acceptance-tests/declarative-config-acceptance | ||
| docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$(pwd):/workspace" -w /workspace grafana/oats:latest test |
There was a problem hiding this comment.
grafana/oats:latest isn't a runnable image — docker run fails with pull access denied … repository does not exist. OATS is a Go binary that orchestrates docker-compose + the lgtm stack itself. Install and run it instead, pinned to a version:
go install github.com/grafana/oats@v0.7.0
oats -timeout 5m acceptance-tests/declarative-config-acceptance/See .mise/tasks/oats-tests.sh for a working example.
| try { | ||
| await page | ||
| .getByRole("button", { name: /Expand Exporter/i }) | ||
| .first() | ||
| .click({ timeout: 5000 }); | ||
| await page.getByText("otlp_http").first().click({ timeout: 5000 }); | ||
| } catch (e) { | ||
| console.log( | ||
| "Could not toggle OTLP exporter, proceeding with default", | ||
| e?.message || String(e) | ||
| ); | ||
| } |
There was a problem hiding this comment.
Two things here:
(a) Select schema version 1.0.0 before extracting the YAML. The builder defaults to 1.1.0, but released agents reject it — with the config actually loaded (see the docker-compose comment), agent 2.29.0 fails to start: Unsupported file format '1.1'. Supported formats include 0.4, 1.0. Drive the version selector (#schema-version-select) to 1.0.0 so the output is file_format: "1.0". Worth a comment noting this means we don't catch the default (1.1) being ahead of released agents.
(b) Don't swallow the exporter-toggle failure. "Proceed with default" means a broken selector passes silently — and the default already emits otlp_http, so a regression here goes unnoticed. Assert the extracted YAML contains the expected exporter block before writing, and re-throw on selector failure. (Selectors also hardcode English UI text in an i18n'd app.)
| - name: Generate config file | ||
| run: | | ||
| cd ecosystem-explorer | ||
| bun run scripts/generate-test-config.mjs ../acceptance-tests/declarative-config-acceptance/test-config.yaml |
There was a problem hiding this comment.
The generated config lands in the same dir as oats.yaml. OATS scans the test dir and parses every *.yaml as a test case (it panics on a malformed config). Write test-config.yaml outside the OATS dir, or point OATS at the specific oats.yaml.
| jobs: | ||
| run-acceptance-test: | ||
| name: Run Acceptance Test | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Add timeout-minutes: to the job — with no bound a hung Playwright/OATS step runs to the 6h default. Also pin OATS (0.7.0) and the app image; the agent staying "latest" is correct (that's the point).
| - name: Install Playwright browsers | ||
| run: | | ||
| cd ecosystem-explorer | ||
| bunx playwright install chromium |
There was a problem hiding this comment.
Use bunx playwright install --with-deps chromium for reliability across runner image changes.
What changed
Added an end-to-end acceptance testing pipeline for the declarative configuration builder. This pipeline uses a Playwright script (
generate-test-config.mjs) to automate the browser UI and extract the generated YAML configuration. The configuration is then passed to thesmoke-test-spring-bootcontainer and validated natively using thegrafana/oatstesting framework via a new GitHub Action workflow (nightly-declarative-config-test.yml).Why
Fixes #662.
The configuration builder lacked automated validation. This ensures that the YAML configurations generated by the UI are valid, successfully parseable by the OpenTelemetry Java Agent, and correctly emit telemetry to an OTLP endpoint, catching any builder regressions automatically.
How it was tested
generate-test-config.mjsPlaywright script locally to verify successful extraction of thetest-config.yaml.docker-compose.ymlmounts the configuration into thesmoke-test-spring-bootcontainer and exports to thelgtmstack on port 4318.oats.yamlasserts the expected/greetingspan when hitting the container.pre-commithooks locally (formatting,ruff,eslint, etc.).Is this a breaking change?
No.
Special notes for your reviewer
I have added appropriate JSDoc documentation to the automation script and
.gitignoreexceptions for the runtime artifacts. The workflow triggers automatically on a schedule but can also be triggered manually viaworkflow_dispatchif you want to test it.Checklist