KUBE-181: Validate MongoDBSearch GA operator upgrades

[anandsyncs]

[skip-ci]

Summary

MongoDBSearch upgrades must continuously validate migration from the first GA MCK release to the current operator. This pins the released operator fixture to MCK 1.9.0, keeps every Search and mongot-version assertion strict, and allows the post-upgrade AppDB up to 600 seconds to reach its actual Running state. The non-static Search upgrade task is permanently enabled while the static duplicate remains excluded.

Jira: KUBE-181

Proof of Work

• Regression patch 6a5a466cd981fd0007360290: 57 requested tasks (all 56 executable Search targets plus unit_tests_golang) and 12 dependencies; 56/57 requested and 12/12 dependencies succeeded. The sole failure was deterministic, not infrastructure: AppDB convergence exceeded the former 300-second wait after every Search creation, query, post-upgrade Running, and mongot-version assertion passed.
• Authoritative patch 6a5a542dd939420008b7043d: the same 57 requested tasks and 12 dependencies all succeeded (69/69), including e2e_search_connectivity_tool_mc_sharded, with zero retries, flakes, or static variants.
• Focused Python compilation, formatting, import ordering, type checking, and collection of all 21 upgrade tests passed. Evergreen configuration validation also passed.

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • skip-changelog applies because this changes test and CI coverage only.

skip-ci

[anandsyncs]

The 1.9.0 pin is an input, not an asserted invariant: nothing checks the running operator is actually 1.9.0 after install. If GA_OPERATOR_VERSION silently floated to another valid version (or the param were dropped), every later check still passes because all version assertions are relative. Suggest asserting the operator Deployment's image tag == GA_OPERATOR_VERSION right after wait_for_operator_ready().

[anandsyncs]

Strict ==, but both sides come from the same operator (its MDB_SEARCH_VERSION env vs its own mongot image) — this validates internal consistency, so a no-op 'upgrade' (same image both sides) passes. Capture MDB_SEARCH_VERSION pre-upgrade and assert post-upgrade it differs (or pre == the 1.9.0-expected tag). Related, pre-existing: get_mongot_image_tag (~line 48) swallows all exceptions with except Exception: continue, which can mask a transient API error as 'not found' — worth tightening since it's the oracle these assertions rest on.