Protect CHK recreate with live Raft quorum safety

[alex-zaitsev]

Summary

Fixes #2069 (CHK Raft quorum safety during rolls) and #2035 (remove pointless same-size settle delay). Partial #2059 (status persistence).

#2069 — quorum-safe rolls

On a multi-node CHK, the operator must not disrupt a Ready Keeper replica if doing so would drop the ensemble below Raft majority.

Flow (CHI #1704-shaped late gate, CHK-specific early snapshot):

1. snapshotHostEnsemble — before any STS disruption, record rolling and live Ready counts. rolling is frozen through force-restart so ReadyReplicas→0 does not re-classify the pass as bootstrap.
2. PrepareHostStatefulSetWithStatus → set STS wait mode (Ready if rolling, Started only if bootstrap).
3. hostDisruptionWouldBreakQuorum — single late gate (same placement as CHI hostDisruptionWouldDegradeShard).
4. If blocked: waitForQuorumSafeToDisruptHost (poll 5s, max 2m) → ErrCRUDDeferred (not whole-CR abort). Status gets [RaftQuorumUnsafe]. Deferral is per replica — the operator continues with the remaining Keeper nodes in the ensemble, then requeues.
5. Recovery-first ordering — not-Ready replicas are reconciled before Ready peers (helps interrupted rolls and tight ensembles).
6. chkStatefulSetFallback → ErrCRUDAbort on STS create/update wait failure — stops advancing to the next replica while the previous one never rejoined (ClickHouseKeeperInstallation controller can recreate every replica of an ensemble without keeping a Raft quorum running #2069 root cause).

Rolling vs bootstrap (from live Ready, not CR ancestor):

Mode	Condition	STS wait
Rolling	n≤1 or ready ≥ quorum	Ready (unless probe opts out)
Bootstrap	below live quorum	Started only

Not in this PR: full #2041 committed-config / mntr barrier (verifyHostEnsembleMembership is a no-op hook). Staged rescale (3→2→1) and smarter per-step downscale settle are follow-ups.

#2035 / partial #2059

• Same-size reconcile: membershipSettleDelay → 0 (was a fixed 10s).
• Downscale still uses 120s pre-settle + 60s post-purge (unchanged).
• Reconcile-start / completion status persist errors propagated to controller-runtime.
• Completion metrics emitted only after Completed is persisted.

Comparison with CHI shard safety (#1704)

	CHI	CHK (this PR)
Gate predicate	≥1 healthy peer in shard	Raft majority of Ready members
On refuse	ErrCRUDDeferred	Wait (up to 2m), then ErrCRUDDeferred
Early snapshot	No	Yes (force-restart / Ready=0)
Recovery-first ordering	Yes	Yes

Test plan

• Unit: go test ./pkg/controller/chk/ (quorum, snapshot, gate, wait/defer, recovery-first, settle delay, status persist)
• Regression: test_020005 xfail removed
• E2e: test_020003 / test_020005 (keeper upgrade / scale)
• E2e: test_020003_3 — 3-node CHK, broken-image roll stops on replica 0, peers stay Ready through operator restart, recovery to good image (CHI test_010083 analogue)
• Close fix(chk): let healthy keeper reconciles complete #2059 after merge (remaining items intentionally not included)

Follow-ups (separate PRs)

• Staged Raft rescale (one peer per step, e.g. 3→2→1)
• Per-step quorum settle replacing blind 120s+60s on downscale
• Raft-safe ClickHouseKeeperInstallation rescale #2041 membership barrier in verifyHostEnsembleMembership