Replace WaitForAllNodesReady inline polling with readiness.Wait

The previous implementation polled cat-nodes for 60s with a single
boolean predicate, producing "Condition never satisfied" on timeout.
Switch to readiness.Wait(t, ctx, TargetClusterReady, WithCluster(c))
which observes per-node progression through LayerHTTP, LayerClusterJoin,
and LayerStatsReady, and dumps a structured per-node diagnostic with
the full last cat-nodes response on failure.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

Author: sean-

CHANGELOG.md

@@ -177,6 +177,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Fixed
 
+- Replace `WaitForAllNodesReady` inline `require.Eventually` loop with a layered readiness FSM (`internal/test/readiness`) that observes per-node progression through `LayerTCP -> LayerHTTP -> LayerClusterJoin -> LayerStatsReady`, records transitions including regressions, and emits a structured per-node diagnostic with the full last cat-nodes response on timeout. Per-layer budgets are tuned for CI pessimism (cold JVM startup is the long pole); total budget for `TargetClusterReady` is 6.5 minutes. ([#650](https://github.com/opensearch-project/opensearch-go/issues/650))
 - Fix bulk indexer HTML-escaping `_id` and `routing` values containing `<`, `>`, or `&` characters, causing OpenSearch to store escaped values (e.g., `\u003croot_account\u003e` stored instead of `<root_account>`), leading to duplicate documents, unreachable data on read-by-ID paths, and potential shard routing mismatches. Present since the `json.Marshal` migration in 2021 (commit `3da59092`). Replace `json.Marshal` with `json.NewEncoder` + `SetEscapeHTML(false)` in `opensearchutil.worker.writeMeta` and `opensearchutil.JSONReader`; replace per-worker `aux []byte` with `sync.Pool`-backed `*bytes.Buffer`; add table-driven test coverage for `writeMeta` edge cases and refactor remaining `TestBulkIndexer` subtests to table-driven `require`-based style ([#824](https://github.com/opensearch-project/opensearch-go/pull/824))
 - Fix pool replacement orphaning resurrection goroutines during node discovery, causing connections to become permanently dead with no active health checker ([#786](https://github.com/opensearch-project/opensearch-go/pull/786))
 - Fix multi-to-single pool demotion leaking resurrection goroutines by giving each `multiServerPool` its own derived context and cancelling it on demotion ([#830](https://github.com/opensearch-project/opensearch-go/pull/830))

osapi/testutil/helpers.go

@@ -26,6 +26,7 @@ import (
 	"golang.org/x/sync/semaphore"
 
 	"github.com/opensearch-project/opensearch-go/v4"
+	"github.com/opensearch-project/opensearch-go/v4/internal/test/readiness"
 	tptestutil "github.com/opensearch-project/opensearch-go/v4/opensearchtransport/testutil"
 	"github.com/opensearch-project/opensearch-go/v4/osapi"
 )
@@ -320,25 +321,20 @@ func WaitForClusterReady(t *testing.T, client *osapi.Client) error {
 	return fmt.Errorf("cluster not ready after %d attempts (version %d.%d.%d)", maxAttempts, major, minor, patch)
 }
 
-// WaitForAllNodesReady polls /_cat/nodes until every node reports non-nil cpu
-// and heap.percent metrics. This prevents flakes from nodes that haven't fully
-// initialized in CI (e.g. stats not yet collected after a fresh cluster start).
+// WaitForAllNodesReady blocks until every node in the test cluster has
+// reached LayerStatsReady — i.e. cluster-health reports the expected
+// node count AND _cat/nodes returns non-nil cpu+heap.percent for each
+// node. It uses the layered readiness FSM in internal/test/readiness so
+// that timeouts produce a structured per-node diagnostic instead of a
+// "Condition never satisfied" stub.
+//
+// Expected node count comes from OPENSEARCH_NODE_COUNT (defaults to 1).
+// Per-layer budgets are tuned for CI pessimism (cold JVM startup is the
+// long pole); see readiness.DefaultBudgets for the exact values.
 func WaitForAllNodesReady(t *testing.T, client *osapi.Client) {
 	t.Helper()
-	require.Eventually(t, func() bool {
-		resp, err := client.Cat.Nodes(t.Context(), &osapi.CatNodesReq{
-			Params: &osapi.CatNodesParams{DebugParams: osapi.DebugParams{Format: "json"}},
-		})
-		if err != nil || resp == nil || len(resp.Records) == 0 {
-			return false
-		}
-		for _, node := range resp.Records {
-			if node.Cpu == nil || node.HeapPercent == nil {
-				return false
-			}
-		}
-		return true
-	}, 60*time.Second, 1*time.Second, "not all nodes reporting stats (cpu/heap.percent nil)")
+	readiness.Wait(t, t.Context(), readiness.TargetClusterReady,
+		readiness.WithCluster(client))
 }
 
 // CompareRawJSONwithParsedJSON is a helper function to determine the difference between the parsed JSON and the raw JSON.