test(SM-8.9): tune Optimize import for faster e2e nightlies#6434
Merged
Conversation
The SM-8.9 e2e long pole is the Optimize-import-bound flow (optimize-user-flows > New Instances Updated Flow). Optimize's default import backoff (max 30s) and currentTimeBackoff (5m) leave newly-run process data invisible to reports for tens of seconds, inflating the test wall-clock and forcing long fixed waits. Tune the integration deploy (base.yaml, Layer 1 so it applies to all non-qa nightly scenarios, not just qa-prefixed ones): - import.handler.backoff.max 30s -> 1s - import.currentTimeBackoffMilliseconds 300000 -> 1000 - zeebe.maxImportPageSize 200 -> 1000 Safe in the low-write QA env. Validated live on matrix-89-eske-inst-gke: New Instances passed 4.5m (+35s setup); optimize 3 heavy tests + setup green in 5.0m. helm template renders the env into the optimize deployment from base.yaml alone; go test ./optimize/... green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the dedicated shadow-e2e scenario from 8.7/8.10 in the 8.9 CI registry. It is skip-e2e (the per-scenario e2e run is skipped) so the shadow-e2e-full-suite job in test-integration-runner.yaml runs the complete SM-8.9 Playwright suite against a single keycloak + elasticsearch + gke deployment, giving a representative full-suite signal and wall-clock alongside the Optimize import tuning in this PR. Registry snapshot regenerated via make go.update-registry-golden; matrix + scenarios Go tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eamonnmoloney
requested review from
Ian-wang-liyang and
Copilot
and removed request for
a team and
Copilot
Ian-wang-liyang
approved these changes
Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which problem does the PR fix?
The SM-8.9 cross-component e2e suite's long pole is the Optimize-import-bound
flow (
optimize-user-flows.spec.ts > New Instances Updated Flow). Under CIsharding the full-suite wall-clock collapses to the slowest single test, so
this flow gates the whole run.
Optimize's default import scheduling makes newly-run process data invisible to
reports for tens of seconds —
import.handler.backoff.maxis 30s andimport.currentTimeBackoffMillisecondsis 5 minutes. That inflates the testwall-clock and forces long fixed waits in the e2e specs.
What's in this PR?
1. Optimize import tuning (
base.yaml, Layer 1 — applied to every non-qa-nightly scenario;
base-qa.yamlonly loads forqa--prefixed scenarios, so itwould silently no-op for eske/oske/kemt/kerba):
CAMUNDA_OPTIMIZE_IMPORT_HANDLER_BACKOFF_MAX: 30s → 1sCAMUNDA_OPTIMIZE_IMPORT_CURRENTTIMEBACKOFFMILLISECONDS: 300000 → 1000CAMUNDA_OPTIMIZE_ZEEBE_MAXIMPORTPAGESIZE: 200 → 1000Safe in the low-write QA env. Scope is
test/integrationscenario values only —no user-facing chart change.
2.
shadow-e2escenario — mirrors the dedicated full-suite scenario alreadypresent in 8.7 and 8.10. It is
skip-e2e(no per-scenario e2e), so theshadow-e2e-full-suitejob runs the complete SM-8.9 Playwright suite against asingle keycloak + elasticsearch + gke deployment — a representative full-suite
signal and wall-clock for this tuning. Registry snapshot regenerated via
make go.update-registry-golden; matrix + scenarios Go tests green.Validation (live,
matrix-89-eske-inst-gke):New Instances Updated Flowpassed in 4.5m (+35s setup)helm templaterenders the env into the optimize deployment frombase.yamlalonego test ./optimize/...greenCoordinated with the e2e-side change in
camunda/c8-cross-component-e2e-tests#2634 (retry-helper bounding). Merge order:
this Helm PR first (gates the nightly matrix), then #2634.
🤖 Generated with Claude Code