docs: add Burrito scheduling recommendations (#919)

* docs: add scheduling recommendations for shared components and runners

* docs: mention service.trafficDistribution for zone-local traffic

Address review comment to mention trafficDistribution for shared components.
This complements pod-level scheduling by routing service traffic to endpoints
in the same zone, reducing cross-zone latency and cost.

* docs: mention .spec.trafficDistribution Kubernetes Service spec field

Addresses review comment from corrieriluca on PR #919

* docs: update trafficDistribution to PreferSameZone per reviewer feedback

PreferClose is deprecated in favor of PreferSameZone for Kubernetes Service
trafficDistribution field.

Addresses review comment from corrieriluca on PR #919.

Author: narutamaaurum

docs/operator-manual/runner-scheduling.md

@@ -1,6 +1,6 @@
-# Fine-tuning the scheduling of runner pods
+# Fine-tuning the scheduling of Burrito pods
 
-Burrito creates runner pods to execute plans and apply changes on your infrastructure. The scheduling of these pods can be fine-tuned to better fit your needs. (e.g. to avoid running too many pods at the same time, or to reduce the cost of your underlying infrastructure).
+Burrito creates runner pods to execute plans and apply changes on your infrastructure, and it also runs shared components such as the controller, the datastore, and optionally Hermitcrab. Scheduling those workloads carefully can reduce inter-zone traffic, isolate noisy jobs, and keep costs under control.
 
 ## Limit the number of runner pods in parallel
 
@@ -13,3 +13,67 @@ You can also set this value in the TerraformRepository CRD by setting the `spec.
 If the value of this parameter is set to `0`, there is no limit to the number of runner pods that can run in parallel.
 
 When Burrito creates a pod, if the setting is both set in the controller and in the TerraformRepository, the TerraformRepository value will take precedence.
+
+## Spread shared components across failure domains
+
+When you run Burrito across multiple availability zones, spread the shared components that serve every reconciliation path:
+
+- `datastore`
+- `hermitcrab` when provider caching is enabled
+- the Burrito server or controllers when you run multiple replicas
+
+The Helm chart already exposes `deployment.nodeSelector`, `deployment.tolerations`, `deployment.affinity`, and `deployment.topologySpreadConstraints` for these components. A practical baseline is:
+
+- use `topologySpreadConstraints` to keep replicas distributed across zones or nodes
+- use anti-affinity to avoid placing every replica on the same node
+- keep at least one `datastore` and `hermitcrab` pod per zone when cross-zone traffic is expensive
+
+For example, `hermitcrab.deployment.topologySpreadConstraints` and `hermitcrab.deployment.affinity` can be used together to keep the provider cache available close to runner pods while avoiding a single-node concentration.
+
+### Prefer zone-local traffic with `trafficDistribution`
+
+The Helm chart exposes `service.trafficDistribution` for the `datastore`, `hermitcrab`, and `server` components. This value maps to the Kubernetes Service spec field [`.spec.trafficDistribution`](https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution). Setting this to `PreferSameZone` instructs the Service to route traffic to endpoints in the same zone as the client, reducing cross-zone traffic latency and cost. This complements the pod-level scheduling controls above by shaping where established connections land.
+
+Example:
+
+```yaml
+hermitcrab:
+  service:
+    trafficDistribution: PreferSameZone
+datastore:
+  service:
+    trafficDistribution: PreferSameZone
+server:
+  service:
+    trafficDistribution: PreferSameZone
+```
+
+## Isolate runner pods onto dedicated nodes
+
+Runner pods are usually the burstiest Burrito workload because they download code, contact providers, and execute Terraform plans or applies. If you want to keep that activity away from shared services, target a dedicated node pool for runners.
+
+The recommended approach is to configure `spec.overrideRunnerSpec` with scheduling fields such as:
+
+- `nodeSelector`
+- `tolerations`
+- `affinity`
+
+That lets you direct runners to nodes reserved for infrastructure jobs while keeping the control plane components on a different pool. The full override mechanism is documented in [Override the runner pod spec](../user-guide/override-runner.md).
+
+A common pattern is:
+
+- label a dedicated node pool for Burrito runners
+- taint those nodes so only runner pods land there
+- add matching `nodeSelector` and `tolerations` in `overrideRunnerSpec`
+- optionally add pod anti-affinity if you want to avoid stacking too many heavy runs on the same node
+
+## Choose the knob that matches the goal
+
+Use different scheduling controls depending on what you are optimizing for:
+
+- **Cost control:** lower `maxConcurrentRunnerPods`, then place runners on cheaper or autoscaled nodes.
+- **Lower latency:** spread `datastore` and `hermitcrab` close to the zones where runners execute; set `trafficDistribution: PreferSameZone` on their Services.
+- **Isolation:** separate runners from controllers, server, and datastore with dedicated node pools.
+- **Resilience:** add topology spread constraints and anti-affinity for shared multi-replica components.
+
+Start with a simple policy, observe runner throughput and inter-zone traffic, then tighten the scheduling rules only where they solve a concrete bottleneck.