chore(e2e): kind flake-fix grab-bag — image load, RPC waits, log routing#210
Closed
puneet2019 wants to merge 4 commits into
Closed
chore(e2e): kind flake-fix grab-bag — image load, RPC waits, log routing#210puneet2019 wants to merge 4 commits into
puneet2019 wants to merge 4 commits into
Conversation
Targets the most common e2e/kind local flakes diagnosed across the last few runs. Each fix is small but targets a specific failure mode: 1. Image-load: replace `kind load docker-image` with docker save | ctr import via the new `kind_load_image` helper. The default kind load silently fails with the desktop-linux buildx driver — the manifest gets exported but the image doesn't actually land in the cluster's snapshotter under the expected SHA, leading to ErrImageNeverPull. 2. Force kind delete when building fresh images: `setup-kind.sh` now deletes the existing cluster when FW_SKIP_BUILD=false. Stale clusters retain old image SHAs under the same name in containerd, so a new build that loads under the same tag silently uses the old SHA. 3. Per-pod RPC readiness: `wait_for_all_validator_rpcs <num>` polls each validator pod's internal RPC after start / restart. The existing `wait_for_chain_ready` only checks the host's NodePort RPC, which one healthy pod satisfies — but a tx routed via kubectl exec to a different pod whose RPC isn't yet serving fails with empty stderr. 4. EVM RPC readiness: `wait_for_evm_rpc_ready` polls eth_blockNumber until non-zero. Cosmos /status can be live before the EVM JSON-RPC is fully attached after start / rolling restart, leading to "server returned a null response" on first cast send. 5. log_error / log_warn -> stderr: the helpers wrote to stdout with no `>&2`, so any caller using `var=$(some_func)` captured the error message INTO `var` (hiding it from the log AND polluting downstream values). Confirmed root cause of several "Test exited early" with no visible error reason. Wire-up: - `lib.sh`: new helpers `wait_for_all_validator_rpcs`, `wait_for_evm_rpc_ready`, `kind_load_image`; log_error/log_warn now to stderr - `setup-kind.sh`: force-delete stale cluster when building - `build-images.sh`, `framework.sh`, `upgrade-chain.sh`: use kind_load_image - `framework.sh:fw_start_chain*`, `upgrade-chain.sh`: call new RPC waits Verified locally: pre-upgrade phase passes cleanly (was previously silent-failing); post-upgrade evm_deploy still has a residual flake where cast send hits a null-response window AFTER EVM RPC reports block > 0 but before some other path is fully warmed (eth_estimateGas or similar). That's a separate fix; this PR closes the most common flake classes.
puneet2019
added a commit
that referenced
this pull request
Jul 8, 2026
… quorum
Root cause of the hardfork-upgrade flake ('Chain did not reach height 41
within 240s'): upgrade-chain.sh loaded the new-binary image with
'kind load docker-image ... 2>/dev/null || true' — a silent failure leaves
validators restarting into a missing/old image — and wait_for_chain_ready is
satisfied by a single healthy pod, so the harness declared 'chain resumed'
while 2/3 quorum was still absent. The post-upgrade height wait then burned
its whole budget measuring the wrong precondition.
- kind_load_image: docker save | ctr import into the kind node with explicit
verification (crictl/ctr image listing), no silent-failure path; used by
build-images.sh, framework.sh, and upgrade-chain.sh.
- wait_for_all_validator_rpcs: gate on EVERY validator serving its own RPC
(real quorum signal) after rolling restarts, plus wait_for_evm_rpc_ready
for the JSON-RPC attach lag.
- setup-kind.sh: delete stale clusters on fresh builds so containerd cannot
serve an old image SHA under the new tag.
Carved from the closed grab-bag #210 (chore/e2e-kind-flake-fixes); the
log-to-stderr slice of that branch is #319 and is deliberately not included
here (no overlapping hunks; the two compose cleanly).
Validated end-to-end: full hardfork upgrade e2e (v1.3.0 -> current) 5/5 with
the new gates visible in the log, and the RPC suite 11/11 for framework
regression.
Signed-off-by: puneetmahajan <59960662+puneet2019@users.noreply.github.com>
puneet2019
added a commit
that referenced
this pull request
Jul 8, 2026
… quorum
Root cause of the hardfork-upgrade flake ('Chain did not reach height 41
within 240s'): upgrade-chain.sh loaded the new-binary image with
'kind load docker-image ... 2>/dev/null || true' — a silent failure leaves
validators restarting into a missing/old image — and wait_for_chain_ready is
satisfied by a single healthy pod, so the harness declared 'chain resumed'
while 2/3 quorum was still absent. The post-upgrade height wait then burned
its whole budget measuring the wrong precondition.
- kind_load_image: docker save | ctr import into the kind node with explicit
verification (crictl/ctr image listing), no silent-failure path; used by
build-images.sh, framework.sh, and upgrade-chain.sh.
- wait_for_all_validator_rpcs: gate on EVERY validator serving its own RPC
(real quorum signal) after rolling restarts, plus wait_for_evm_rpc_ready
for the JSON-RPC attach lag.
- setup-kind.sh: delete stale clusters on fresh builds so containerd cannot
serve an old image SHA under the new tag.
Carved from the closed grab-bag #210 (chore/e2e-kind-flake-fixes); the
log-to-stderr slice of that branch is #319 and is deliberately not included
here (no overlapping hunks; the two compose cleanly).
Validated end-to-end: full hardfork upgrade e2e (v1.3.0 -> current) 5/5 with
the new gates visible in the log, and the RPC suite 11/11 for framework
regression.
Signed-off-by: puneetmahajan <59960662+puneet2019@users.noreply.github.com>
puneet2019
added a commit
that referenced
this pull request
Jul 9, 2026
… quorum
Root cause of the hardfork-upgrade flake ('Chain did not reach height 41
within 240s'): upgrade-chain.sh loaded the new-binary image with
'kind load docker-image ... 2>/dev/null || true' — a silent failure leaves
validators restarting into a missing/old image — and wait_for_chain_ready is
satisfied by a single healthy pod, so the harness declared 'chain resumed'
while 2/3 quorum was still absent. The post-upgrade height wait then burned
its whole budget measuring the wrong precondition.
- kind_load_image: docker save | ctr import into the kind node with explicit
verification (crictl/ctr image listing), no silent-failure path; used by
build-images.sh, framework.sh, and upgrade-chain.sh.
- wait_for_all_validator_rpcs: gate on EVERY validator serving its own RPC
(real quorum signal) after rolling restarts, plus wait_for_evm_rpc_ready
for the JSON-RPC attach lag.
- setup-kind.sh: delete stale clusters on fresh builds so containerd cannot
serve an old image SHA under the new tag.
Carved from the closed grab-bag #210 (chore/e2e-kind-flake-fixes); the
log-to-stderr slice of that branch is #319 and is deliberately not included
here (no overlapping hunks; the two compose cleanly).
Validated end-to-end: full hardfork upgrade e2e (v1.3.0 -> current) 5/5 with
the new gates visible in the log, and the RPC suite 11/11 for framework
regression.
Signed-off-by: puneetmahajan <59960662+puneet2019@users.noreply.github.com>
puneet2019
added a commit
that referenced
this pull request
Jul 13, 2026
… quorum
Root cause of the hardfork-upgrade flake ('Chain did not reach height 41
within 240s'): upgrade-chain.sh loaded the new-binary image with
'kind load docker-image ... 2>/dev/null || true' — a silent failure leaves
validators restarting into a missing/old image — and wait_for_chain_ready is
satisfied by a single healthy pod, so the harness declared 'chain resumed'
while 2/3 quorum was still absent. The post-upgrade height wait then burned
its whole budget measuring the wrong precondition.
- kind_load_image: docker save | ctr import into the kind node with explicit
verification (crictl/ctr image listing), no silent-failure path; used by
build-images.sh, framework.sh, and upgrade-chain.sh.
- wait_for_all_validator_rpcs: gate on EVERY validator serving its own RPC
(real quorum signal) after rolling restarts, plus wait_for_evm_rpc_ready
for the JSON-RPC attach lag.
- setup-kind.sh: delete stale clusters on fresh builds so containerd cannot
serve an old image SHA under the new tag.
Carved from the closed grab-bag #210 (chore/e2e-kind-flake-fixes); the
log-to-stderr slice of that branch is #319 and is deliberately not included
here (no overlapping hunks; the two compose cleanly).
Validated end-to-end: full hardfork upgrade e2e (v1.3.0 -> current) 5/5 with
the new gates visible in the log, and the RPC suite 11/11 for framework
regression.
Signed-off-by: puneetmahajan <59960662+puneet2019@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Targets the most common e2e/kind local flakes diagnosed across recent runs. Each fix is small but targets a specific failure mode. Bundled into one PR since they share the same files (`lib.sh`, `framework.sh`).
Five fixes
1. Image-load: `kind load docker-image` -> `docker save | ctr import`
The default `kind load` silently fails with the desktop-linux buildx driver — the manifest gets exported but the image doesn't actually land in the cluster's containerd snapshotter under the expected SHA, leading to `ErrImageNeverPull` at pod start.
Replaced with new `kind_load_image` helper that uses `docker save | docker exec ctr import` directly.
2. Force kind delete when building fresh images
`setup-kind.sh` now deletes the existing cluster when `FW_SKIP_BUILD=false`. Stale clusters retain old image SHAs under the same tag in containerd → new build silently uses the old SHA. CI runners always start clean so this is a no-op there; locally it's the difference between "passes first try" and "fails mysteriously".
3. Per-pod RPC readiness
New `wait_for_all_validator_rpcs ` polls each validator pod's internal RPC after start / restart. The existing `wait_for_chain_ready` only checks the host's NodePort RPC, which one healthy pod satisfies — but a tx routed via kubectl exec to a different pod whose RPC isn't yet serving fails with empty stderr.
4. EVM RPC readiness
New `wait_for_evm_rpc_ready` polls `eth_blockNumber` until non-zero. Cosmos `/status` can be live before the EVM JSON-RPC is fully attached, leading to "server returned a null response" on first `cast send`.
5. log_error / log_warn -> stderr
`log_error` and `log_warn` wrote to stdout with no `>&2`. Any caller using `var=$(some_func)` captured the error message INTO `var` (hiding it from the log AND polluting downstream values). This was the root cause of several "Test exited early (code 1)" with no visible error reason — the error was getting eaten by command substitution.
Wire-up
5 files, +122/-14.
Test plan
Known residual flake (out of scope)
Post-upgrade `evm_deploy` still has a window where `cast send` hits "null response" AFTER `eth_blockNumber` reports non-zero — likely `eth_estimateGas` / `eth_chainId` warming at a different rate. The new log routing makes this visible in logs (it was completely silent before). Separate follow-up.
Why draft
Marked draft because:
🤖 Generated with Claude Code