feat(e2e): generate → run → curl-compare driver scripts (hub & oca) by esraagamal6 · Pull Request #399 · camunda/api-test-generator

esraagamal6 · 2026-06-17T14:22:23Z

What

Two per-config end-to-end drivers + a shared curl oracle, encapsulating the manual flow we've been running by hand.

File	Role
`scripts/e2e/run-hub.sh`	Hub: generate → run → curl-compare. Mints Keycloak Bearer tokens (`c8-client` admin, `c8-client-deny` for rbac). Base `…:8088/api`, default profiles `secured rbac`.
`scripts/e2e/run-oca.sh`	OCA: same flow. HTTP Basic auth (`demo/demo`, overridable/empty). Base `…:8080`, default profile `unsecured`.
`scripts/e2e/curl_compare.py`	Shared cross-check oracle over the request-validation suite.

Each driver runs three steps (selectable via STEPS):

generate — positive (extract-graph → scenarios → codegen) + request-validation
run — Playwright: positive suite + each request-validation profile (JSON report saved)
curl-compare — re-issue each negative test independently with curl

Curl oracle

curl_compare.py reconstructs each negative test's request from the emitted .spec.ts (method, …/v2<path> with params substituted, headers by scenario kind, body normalised via node) and re-issues it with curl. It prints a table comparing expected vs Playwright vs curl status and diffs the response body on mismatch (--show-body). It does not import the suite's own code, so a bug in the emitter/support module can't mask a server discrepancy. Exits non-zero on any mismatch (CI-friendly).

Usage

./scripts/e2e/run-hub.sh                                   # all 3 steps
STEPS=curl RV_PROFILES=secured ./scripts/e2e/run-hub.sh    # re-curl one profile
SKIP_POSITIVE=1 ./scripts/e2e/run-hub.sh                   # negatives only
./scripts/e2e/run-oca.sh                                   # needs a C8 cluster at :8080

Reports + diffs land in test-results/e2e-<config>/.

Validation

The Hub curl oracle was run live (parser, URL/param/header reconstruction, status+body diff verified). run-oca.sh mirrors the structure with OCA-appropriate auth/base defaults (not exercised end-to-end — no local C8 cluster available at the time).

Runtime notes (not blockers for this PR)

Hub rbac relies on the c8-client-deny Keycloak client (from the unmerged request-validation: 403 auth-deny for uniformly-secured APIs (Hub) #394) and the positive suite's Bearer auth (feat(hub): authenticate + target a per-config base URL in the positive suite #398). These affect running, not the scripts themselves.

🤖 Generated with Claude Code

Adds two per-config end-to-end drivers plus a shared curl oracle: - scripts/e2e/run-hub.sh / run-oca.sh — one per config; each (1) generates the positive + request-validation suites, (2) runs Playwright (positive suite + each request-validation profile), and (3) runs an independent curl oracle over the request-validation profiles. Hub mints Keycloak Bearer tokens (c8-client admin, c8-client-deny for rbac); OCA uses HTTP Basic. All ports/URLs/profiles/ steps are env-overridable; reports + diffs land in test-results/e2e-<config>/. - scripts/e2e/curl_compare.py — reconstructs each negative test's request from the emitted .spec.ts (method, /v2<path> with params, headers by scenario kind, body via node) and re-issues it with curl, comparing expected vs Playwright vs curl status and diffing the response body on mismatch. It does NOT import the suite's own code, so it is a true cross-check oracle. Exits non-zero on any mismatch. Validated live against camunda-hub. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Adds per-config end-to-end driver scripts plus a shared “curl oracle” to automate the manual flow of generating suites, running Playwright, and cross-checking request-validation negatives via independent curl replays.

Changes:

Introduces two shell drivers (run-hub.sh, run-oca.sh) to orchestrate generate → run → curl-compare for Hub and OCA configs.
Adds curl_compare.py to reconstruct negative-test requests from emitted .spec.ts files and compare expected vs Playwright vs curl outcomes.
Writes Playwright JSON outputs and curl mismatch details under test-results/e2e-<config>/.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 12 comments.

File	Description
scripts/e2e/run-oca.sh	OCA driver for generate/run/curl-compare with Basic auth defaults and per-profile loop.
scripts/e2e/run-hub.sh	Hub driver for generate/run/curl-compare with Keycloak-minted Bearer tokens and per-profile loop.
scripts/e2e/curl_compare.py	Shared curl-based oracle that replays request-validation negatives reconstructed from emitted spec files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…n + propagate oracle exit curl_compare.py: - Reconstruct the request URL by running the suite's EXACT buildUrl() in node, so the 3-arg buildUrl(path, params, query) form, path-param substitution, and encodeURIComponent query encoding all match (was: single-quote-only regex that ignored query params and the 3-arg form). - Parse and re-issue multipart scenarios (multipartFields + multipart: formData) with `curl -F`, instead of only JSON `--data-binary`. - Quote-agnostic parsing of operationId/scenarioKind (handles both the JSON.stringify double-quote source and prettier's single-quote output). - Extract literals via balanced-delimiter scanning + node JSON normalisation (handles multi-line / nested objects). run-hub.sh / run-oca.sh: - Let the oracle's non-zero exit propagate: record per-profile, fail at the end unless E2E_SOFT=1 (was: `|| true`, which masked all mismatches). - run-oca.sh: strip newlines from base64 (macOS/BSD wrap at 76 cols would corrupt the Authorization header); correct the misleading "authenticates only via BEARER_TOKEN" comment. Re-validated live against camunda-hub (parity with the prior run; query URLs now encoded correctly). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

… failure curl_compare.py: - Skip auth-deny (403) scenarios when --deny-header is empty, instead of re-issuing them unauthenticated and reporting bogus 401-vs-403 mismatches. - Exit non-zero (2) when 0 comparable tests were produced (broken parser, wrong --spec-dir, or everything skipped) — was a silent 0/0 pass. run-hub.sh: - Build the deny header conditionally (empty when c8-client-deny mint fails) so curl never receives `Authorization: Bearer ` (empty token) → the oracle skips deny scenarios cleanly. run-oca.sh: - Compute a deny-probe Basic header from RBAC_DENY_PROBE_USER/PASSWORD (defaults match the suite's env.ts) and pass it to the oracle, so rbac deny tests are re-issued as the probe user rather than unauthenticated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

+        bm = re.search(r"const requestBody[^=]*=\s*", block)
+        if bm:
+            j = bm.end()
+            # literal starts at next { or [
+            br = min((p for p in (block.find("{", j), block.find("[", j)) if p != -1), default=-1)
+            if br != -1:
+                lit = extract_balanced(block, br, block[br], "}" if block[br] == "{" else "]")
+                body_json = node(f"process.stdout.write(JSON.stringify(({block[br]}{lit}{'}' if block[br]=='{' else ']'})))")


+  CORE_APPLICATION_URL="$CORE_URL" RV_PROFILE="$p" CONFIG="$CONFIG" \
+    ${OCA_USER:+CAMUNDA_BASIC_AUTH_USER="$OCA_USER"} ${OCA_PASS:+CAMUNDA_BASIC_AUTH_PASSWORD="$OCA_PASS"} \
+    npx playwright test -c "$cfg" --reporter=json > "$OUT/pw-$p.json" 2>/dev/null || true


Copilot AI review requested due to automatic review settings June 17, 2026 14:22

Copilot started reviewing on behalf of esraagamal6 June 17, 2026 14:22 View session

esraagamal6 self-assigned this Jun 17, 2026

Copilot AI reviewed Jun 17, 2026

View reviewed changes

esraagamal6 requested a review from Copilot June 17, 2026 14:43

Copilot started reviewing on behalf of esraagamal6 June 17, 2026 14:44 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread scripts/e2e/run-hub.sh

Comment thread scripts/e2e/run-oca.sh

Comment thread scripts/e2e/run-oca.sh

Comment thread scripts/e2e/curl_compare.py

Comment thread scripts/e2e/curl_compare.py

esraagamal6 requested a review from Copilot June 17, 2026 14:52

Copilot started reviewing on behalf of esraagamal6 June 17, 2026 14:53 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(e2e): generate → run → curl-compare driver scripts (hub & oca)#399

feat(e2e): generate → run → curl-compare driver scripts (hub & oca)#399
esraagamal6 wants to merge 3 commits into
mainfrom
feat/e2e-test-driver-scripts

esraagamal6 commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

esraagamal6 commented Jun 17, 2026

What

Curl oracle

Usage

Validation

Runtime notes (not blockers for this PR)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants