docs: add batch stickiness, SKYHOOK_NODE_ORDER, and reset changes for PR #183

Author: lockwobr

agent/README.md

@@ -31,4 +31,5 @@ There are a number of environment variables that can be used to control how the
 
 The following are enviroment variables expected to be set by either the build system or skyhook-operator. It is not recommended they be changed manually.
 1. `OVERLAY_FRAMEWORK_VERSION` this the version of the current overlay. It is expected that this gets set by the docker build system. It is required to be able to manage the history file. It must be in the format of `{package name}-{version}`
-1. `SKYHOOK_RESOURCE_ID` this is used to determine if an interrupt should be rerun. Interrupts are only run once per `SKYHOOK_RESOURCE_ID`. Skyhook operator should make this unique per conifguration of the package.
+1. `SKYHOOK_RESOURCE_ID` this is used to determine if an interrupt should be rerun. Interrupts are only run once per `SKYHOOK_RESOURCE_ID`. Skyhook operator should make this unique per conifguration of the package.
+2. `SKYHOOK_NODE_ORDER` zero-indexed monotonic position of this node in the rollout. The first batch's nodes get `0, 1, 2, ...` and subsequent batches continue from where the previous batch left off. Useful for kubeadm upgrade workflows where the first node (`SKYHOOK_NODE_ORDER=0`) runs a different command than subsequent nodes. See [Node Order Within a Rollout](../docs/ordering_of_skyhooks.md#node-order-within-a-rollout) for details.

docs/cli.md

@@ -147,7 +147,7 @@ kubectl skyhook reset gpu-init --skip-batch-reset --confirm
 | `--confirm, -y` | Skip confirmation prompt |
 | `--skip-batch-reset` | Skip resetting deployment policy batch state |
 
-> **Note:** By default, `reset` also resets the deployment policy batch state so the next rollout starts from batch 1. Use `--skip-batch-reset` to preserve the existing batch state.
+> **Note:** By default, `reset` also resets the deployment policy batch state so the next rollout starts from batch 1, and clears node ordering state (`NodeOrderOffset` and `NodePriority`) so `SKYHOOK_NODE_ORDER` restarts from `0`. Use `--skip-batch-reset` to preserve the existing batch and ordering state.
 
 ### Deployment Policy Commands
 
@@ -171,6 +171,7 @@ The `deployment-policy reset` command resets the batch processing state for all
 - Consecutive failure count
 - Completed and failed node counts
 - Stop flag
+- Node ordering state (`NodeOrderOffset` and `NodePriority`) — `SKYHOOK_NODE_ORDER` restarts from `0`
 
 | Flag | Description |
 |------|-------------|

docs/deployment_policy.md

@@ -171,6 +171,16 @@ When strategy parameters are not specified, the operator applies these defaults:
 
 ---
 
+## Batch Stickiness
+
+Nodes selected for a batch remain in that batch until every node has reached a definitive outcome — all packages complete, erroring, or blocked. The controller will not select new nodes for the next batch while the current batch has nodes still running between packages.
+
+Batch membership is tracked via `NodePriority` in the Skyhook status. A node stays in `NodePriority` from the time it is picked for a batch until it completes all packages. This state is persisted in the CRD, so it survives controller restarts.
+
+Each package pod also receives a `SKYHOOK_NODE_ORDER` environment variable reflecting the node's monotonic position in the rollout. See [Node Order Within a Rollout](ordering_of_skyhooks.md#node-order-within-a-rollout) for details.
+
+---
+
 ## Selectors and Node Matching
 
 Compartments use standard Kubernetes label selectors:
@@ -326,6 +336,8 @@ kubectl skyhook reset my-skyhook --confirm
 kubectl skyhook reset my-skyhook --skip-batch-reset --confirm
 ```
 
+Both `reset` and `deployment-policy reset` also clear `NodeOrderOffset` and `NodePriority`, so the next rollout starts with fresh node ordering (`SKYHOOK_NODE_ORDER` begins at `0`).
+
 See [CLI documentation](cli.md) for full command details.
 
 ---

docs/operator-status-definitions.md

@@ -81,4 +81,15 @@ uninstall (if downgrading) → cordon → wait → drain → apply → config 
 cordon → wait → drain → upgrade (if upgrading) → config → interrupt → post-interrupt
 ```
 
-**Note**: The cordon, wait, and drain phases ensure that workloads are safely removed from the node before any package operations that require interrupts (such as reboots or kernel module changes) are executed.
+**Note**: The cordon, wait, and drain phases ensure that workloads are safely removed from the node before any package operations that require interrupts (such as reboots or kernel module changes) are executed.
+
+## Skyhook Status Fields
+
+The Skyhook resource's `.status` object includes fields that track batch rollout state. Two fields are particularly relevant for [batch stickiness](deployment_policy.md#batch-stickiness) and [node ordering](ordering_of_skyhooks.md#node-order-within-a-rollout):
+
+| Field | Definition |
+|-------|------------|
+| `NodePriority` | Tracks which nodes are in the current active batch. A node stays in `NodePriority` from the time it is selected for a batch until it completes all packages. Prevents the controller from selecting new nodes while current batch nodes are between packages. |
+| `NodeOrderOffset` | Cumulative count of nodes removed from `NodePriority`. Combined with a node's position in the sorted `NodePriority` map, this produces the monotonic `SKYHOOK_NODE_ORDER` value injected into package pods. |
+
+Both fields are persisted in the CRD and survive controller restarts. They are cleared by `kubectl skyhook reset` and `kubectl skyhook deployment-policy reset`.

docs/ordering_of_skyhooks.md

@@ -70,6 +70,37 @@ In this example, fast nodes can install drivers independently, but all nodes mus
 
 ---
 
+## Node Order Within a Rollout
+
+The sections above cover ordering of Skyhooks relative to each other. This section covers ordering of **nodes** within a single Skyhook's rollout.
+
+When a [DeploymentPolicy](deployment_policy.md) controls the batch rollout, each package pod receives a `SKYHOOK_NODE_ORDER` environment variable — a zero-indexed integer reflecting the node's position in the overall rollout order.
+
+- The first batch's nodes are assigned `0, 1, 2, ...`
+- The second batch continues from where the first left off (e.g., `3, 4, 5, ...`)
+- Values are monotonically increasing across batches and never reused within a rollout
+- Within a batch, nodes are sorted by name for deterministic tiebreaking
+
+### Use case: kubeadm upgrades
+
+The primary motivation is kubeadm-style Kubernetes upgrades where the first control-plane node must run `kubeadm upgrade apply` and all subsequent nodes run `kubeadm upgrade node`:
+
+```bash
+if [ "$SKYHOOK_NODE_ORDER" -eq 0 ]; then
+    kubeadm upgrade apply v1.35.0
+else
+    kubeadm upgrade node
+fi
+```
+
+### Scope
+
+`SKYHOOK_NODE_ORDER` reflects rollout order within a single Skyhook only. Cross-Skyhook ordering is controlled by `priority` and `sequencing` (documented above). If a Skyhook is reset via `kubectl skyhook reset`, the node order restarts from `0`.
+
+See [Batch Stickiness](deployment_policy.md#batch-stickiness) for details on how batches are kept intact during rollout.
+
+---
+
 ## Flow Control Annotations
 
 Two flow control features can be set in the annotations of each skyhook: