[U-05][J-05] Document upgrades and multi-AZ behavior

[aaronmgn]

Summary

The chart exposes updateStrategy.rollingUpdate.partition and type: OnDelete, but the README only lists them in the values tables. There is no procedural upgrade documentation, and nothing covers how availability zones interact with WaitForFirstConsumer storage, which pins each existing ordinal to a zone at first volume bind. Add two README sections.

Details

Current state: partition exists for both tiers (values.yaml:246-250 and 395-399) and renders in both StatefulSets. The README describes it in one line per values table, there is no Upgrading section (Maintenance covers only MongoDB backup and restore), and no AZ content exists in the README or docs/. topologySpreadConstraints is not exposed. The scheduling knobs are nodeSelector, tolerations, and affinity, and setting affinity replaces the default anti-affinity for that tier. One gotcha: values.schema.json (lines 352 and 540) types partition as string or null, so a plain --set with an integer fails validation. Use --set-string or a quoted values-file entry.

Section 1, Upgrading:

• Partition canary: set partition to N-1, upgrade, verify the canary via GET /api/system/cluster/nodes plus actual throughput, then walk partition down to 0. Same for the Datanode, adding OpenSearch health checks.
• type: OnDelete for full manual cadence control.
• Note that StatefulSets update from the highest ordinal down, so pod-0 goes last automatically.
• Mixed-version UI churn during rolls: recommend ingress session affinity via the existing ingress.web.annotations, with one controller-specific example.
• Start a per-release upgrade-notes.md with the next release and wire it into docs/RELEASING.md.

Section 2, Multi-AZ:

• Zone pinning: PVCs bind where the pod first schedules and zonal volumes cannot cross zones, so existing ordinals need replacement capacity in their bound zone or sit Pending. The chart's own AWS gp3 class uses WaitForFirstConsumer (templates/config/sc/aws-gp3.yaml:19).
• Non-graceful node loss: roughly 6 minutes before the volume is force-detached, so replacement pods stall at least that long. State the order of magnitude so operators do not read it as a chart bug.
• A zone-spread example. Decide first: write it with the affinity override (documenting that it replaces the default anti-affinity), or add a topologySpreadConstraints passthrough to both StatefulSets and document that instead.
• Scope note: applies to dynamically provisioned per-ordinal PVCs, not existingClaim or disabled persistence.

Reference: U-05, U-06, J-05 (Production Readiness Review)

Impact

Operators upgrade without a canary because nothing tells them how, hit the --set schema error with no documented workaround, and first learn about zone pinning when a pod sticks in Pending after a node failure.

Notes for maintainers

Both sections go in charts/graylog/README.md plus its table of contents. The partition schema typing deserves a small follow-up fix to also accept integers. Related: #14 tracks HA behavior changes, while this issue only documents current behavior.

[aaronmgn]

This issue will benefit from creating an admin token, so we're going to hold off for now based on my discussion with Alix.