fix(e2e): restore over-stripped fallback guards in upgrade-chain.sh (review)

phenix flagged that the gov-authority query lost its `|| true`, so under
`set -euo pipefail` a transient kubectl-exec/query failure terminates the
script and the fallback + diagnostic become unreachable. Auditing that
revealed the carve from the grab-bag had actually dropped ALL 10 of main's
`|| true` in this file, not just the intended image-load one — including two
more fallback-critical query paths (proposal-id) and four best-effort
kubectl scale/wait guards.

Reset upgrade-chain.sh to main and re-applied only the intended changes: image
load -> kind_load_image (fail loudly, the one deliberate removal), the
post-restart per-validator + EVM RPC gates, and the two gov-authority queries
converted to the explicit `if ! gov_authority=$(...); then gov_authority=""; fi`
form per the review (preserves the fallback explicitly instead of relying on
`|| true`). The other seven defensive `|| true` are preserved.

Validated on the governance upgrade path (which the earlier hardfork run did
not exercise): full gov e2e 5/5 — gov-authority query resolves, proposal
submitted/voted 4/4, chain upgrades and resumes, all post-upgrade tests pass.

Signed-off-by: puneetmahajan <59960662+puneet2019@users.noreply.github.com>

Author: puneet2019

CHANGELOG.md

@@ -64,7 +64,6 @@ Ref: https://keepachangelog.com/en/1.0.0/
 - (types,utils) [#320](https://github.com/mocachain/moca/pull/320) Delete dead ethermint-era files orphaned by the `rpc/`/`x/evm`/`x/erc20` removals: `types/{block,gasmeter,int,protocol,validation,constant}.go`, `types/openapiutil/`, and `utils/eth/eth.go` — every exported symbol verified unreferenced across the repo and all known external consumers (moca-storage-provider, moca-go-sdk, moca-challenger). Externally-used files (`chain_id.go` — `ParseChainID` is production code in the storage provider's signer — `grn.go`, `s3util`, `resource`, `common`) are untouched.
 - (client) [#321](https://github.com/mocachain/moca/pull/321) Delete the dead `client/testnet.go` (579 LOC): an unreferenced near-duplicate of `cmd/mocad/cmd/testnet.go`, which is what the `mocad testnet` command actually wires.
 - (types,server) [#322](https://github.com/mocachain/moca/pull/322) Use cosmos/evm's `HasDynamicFeeExtensionOption` and `NewIndexTxCmd` instead of the in-tree copies (semantically identical; both already operated on cosmos/evm's own types) — the only two fork-neutral swaps the five-folder sweep found.
-- (e2e) [#324](https://github.com/mocachain/moca/pull/324) Fix the hardfork-upgrade e2e flake ("Chain did not reach height N within 240s"): the new-binary image load into kind could fail silently (`kind load ... || true`) and readiness accepted a single healthy pod as "chain resumed" while 2/3 quorum was still absent. Load images via `docker save | ctr import` with verification (`kind_load_image`), and gate post-restart on every validator's own RPC plus the EVM JSON-RPC (`wait_for_all_validator_rpcs`, `wait_for_evm_rpc_ready`); also delete stale kind clusters on fresh builds so containerd can't serve an old image SHA under the new tag.
 
 - (contracts) [#314](https://github.com/mocachain/moca/pull/314) Slim `contracts/` to what moca actually uses (mirroring cosmos/evm's top-level `contracts/` convention): delete the four ERC20 test doubles (`MinterBurnerDecimals`, `Burnable`, `DirectBalanceManipulation`, `MaliciousDelayed`) — dead since the `x/erc20` removal (#221), available upstream where still needed — plus their `.sol` sources, artifacts, and npm manifest. The moca-specific `ERC721NonTransferable` (bucket/object/group NFT facade the storage keeper mints/burns through) stays put, now pinned by a guard test covering its ABI surface, fixed token/hub addresses, and its intentionally bytecode-less (ABI-only) nature. Slither retargets to `solidity/`; the now-subjectless solhint workflow and broken `lint-contracts` Make targets are removed.
 

e2e/kind/scripts/upgrade-chain.sh

@@ -32,21 +32,27 @@ log_info "  Image:  ${NEW_DOCKER_IMAGE}"
 _upgrade_governance() {
     log_info "Submitting software-upgrade proposal..."
 
-    # Get the gov module authority address
+    # Get the gov module authority address. Use `if !` (not a bare assignment) so
+    # a failing query under `set -euo pipefail` falls through to the alternative /
+    # diagnostic below instead of terminating the script.
     local gov_authority=""
-    gov_authority=$(kubectl exec -n "${K8S_NAMESPACE}" validator-0-0 -c mocad -- \
+    if ! gov_authority=$(kubectl exec -n "${K8S_NAMESPACE}" validator-0-0 -c mocad -- \
         mocad query auth module-account gov \
         --node tcp://localhost:26657 \
         --home /root/.mocad \
-        --output json 2>/dev/null | jq -r '.account.base_account.address // .account.value.address // empty' 2>/dev/null)
+        --output json 2>/dev/null | jq -r '.account.base_account.address // .account.value.address // empty' 2>/dev/null); then
+        gov_authority=""
+    fi
 
     if [ -z "$gov_authority" ]; then
         log_warn "Could not query gov module address, trying alternative..."
-        gov_authority=$(kubectl exec -n "${K8S_NAMESPACE}" validator-0-0 -c mocad -- \
+        if ! gov_authority=$(kubectl exec -n "${K8S_NAMESPACE}" validator-0-0 -c mocad -- \
             mocad query auth module-accounts \
             --node tcp://localhost:26657 \
             --home /root/.mocad \
-            --output json 2>/dev/null | jq -r '.accounts[] | select(.name=="gov") | .base_account.address // .value.address // empty' 2>/dev/null)
+            --output json 2>/dev/null | jq -r '.accounts[] | select(.name=="gov") | .base_account.address // .value.address // empty' 2>/dev/null); then
+            gov_authority=""
+        fi
     fi
 
     if [ -z "$gov_authority" ]; then
@@ -80,7 +86,7 @@ PROPOSAL_EOF
 
     # Write proposal file into the pod
     echo "$proposal_json" | kubectl exec -i -n "${K8S_NAMESPACE}" validator-0-0 -c mocad -- \
-        bash -c "cat > /tmp/upgrade-proposal.json"
+        bash -c "cat > /tmp/upgrade-proposal.json" || true
 
     # Submit proposal via sync broadcast + wait for inclusion
     local submit_out="" submit_hash=""
@@ -110,9 +116,9 @@ PROPOSAL_EOF
         --node tcp://localhost:26657 \
         --chain-id "${CHAIN_ID}" \
         --home /root/.mocad \
-        --output json 2>&1)
+        --output json 2>&1) || true
 
-    proposal_id=$(echo "$query_out" | jq -r '.proposals[-1].id // .proposals[-1].proposal_id // empty' 2>/dev/null)
+    proposal_id=$(echo "$query_out" | jq -r '.proposals[-1].id // .proposals[-1].proposal_id // empty' 2>/dev/null) || true
 
     if [ -z "$proposal_id" ]; then
         log_error "Could not find upgrade proposal"
@@ -206,9 +212,9 @@ _wait_for_upgrade_halt() {
     # Chain halts but doesn't exit — scale down, replace binary, scale back up
     log_info "Scaling down validators..."
     for ((i = 0; i < NUM_VALIDATORS; i++)); do
-        kubectl scale statefulset "validator-${i}" --replicas=0 -n "${K8S_NAMESPACE}" 2>/dev/null
+        kubectl scale statefulset "validator-${i}" --replicas=0 -n "${K8S_NAMESPACE}" 2>/dev/null || true
     done
-    kubectl wait --for=delete pod -l app=validator -n "${K8S_NAMESPACE}" --timeout=60s 2>/dev/null
+    kubectl wait --for=delete pod -l app=validator -n "${K8S_NAMESPACE}" --timeout=60s 2>/dev/null || true
     log_info "All validators stopped"
 
     # Update all validator StatefulSets to use the new image, then scale back up
@@ -218,8 +224,9 @@ _wait_for_upgrade_halt() {
 _update_validator_images() {
     log_info "Updating validator images to ${NEW_DOCKER_IMAGE}..."
 
-    # Load new image into Kind if not already loaded (docker save | ctr import
-    # — see kind_load_image in lib.sh; avoids buildx desktop-linux quirks)
+    # Load new image into Kind (docker save | ctr import with verification — see
+    # kind_load_image in lib.sh; the plain `kind load` silently no-op'd on a
+    # failed load, leaving validators to restart into a missing/old image).
     kind_load_image "${NEW_DOCKER_IMAGE}"
 
     # Patch each validator StatefulSet with the new image
@@ -236,7 +243,7 @@ _update_validator_images() {
     # Scale validators back up with new image
     log_info "Starting validators with new image..."
     for ((i = 0; i < NUM_VALIDATORS; i++)); do
-        kubectl scale statefulset "validator-${i}" --replicas=1 -n "${K8S_NAMESPACE}" 2>/dev/null
+        kubectl scale statefulset "validator-${i}" --replicas=1 -n "${K8S_NAMESPACE}" 2>/dev/null || true
     done
 
     # Wait for all validators to come back up
@@ -246,21 +253,20 @@ _update_validator_images() {
         kubectl wait --for=condition=ready "pod/validator-${i}-0" \
             -n "${K8S_NAMESPACE}" --timeout=180s 2>/dev/null || {
             log_error "Validator-${i} failed to restart"
-            kubectl logs -n "${K8S_NAMESPACE}" "validator-${i}-0" --tail=50 2>/dev/null
+            kubectl logs -n "${K8S_NAMESPACE}" "validator-${i}-0" --tail=50 2>/dev/null || true
             return 1
         }
     done
 
     log_success "All validators restarted with new image"
 
-    # Wait for chain to resume producing blocks (NodePort RPC, fast signal)
+    # Wait for chain to resume producing blocks (NodePort RPC — a single healthy
+    # pod satisfies this), then gate on EVERY validator's own RPC: a tx broadcast
+    # routed via kubectl exec to a pod whose RPC isn't yet serving fails with
+    # empty stderr, and the EVM JSON-RPC can lag the cosmos /status by several
+    # seconds after a rolling restart ("null response" on the first cast send).
     wait_for_chain_ready "http://localhost:26657" 180
-    # Then poll EACH validator's RPC — wait_for_chain_ready is satisfied by a
-    # single healthy pod, but a tx broadcast routed via kubectl exec to a pod
-    # whose RPC isn't yet serving will fail with empty stderr. Belt-and-braces.
     wait_for_all_validator_rpcs "$NUM_VALIDATORS" 60
-    # And the EVM JSON-RPC, which can lag the cosmos /status by several seconds
-    # after a rolling restart, leading to "null response" on first cast send.
     wait_for_evm_rpc_ready "http://localhost:8545" 60
     log_success "Chain resumed after upgrade"
 }