feat: lease-aware scheduler gating, leader visibility, failover e2e

README.md

@@ -530,13 +530,14 @@ spec:
 
 ### Heartbeat Scheduler
 
-Paperclip runs a heartbeat scheduler for periodic agent tasks. In multi-replica deployments, only pod-0 (ordinal 0) runs the scheduler to prevent duplicate execution:
+Paperclip runs a heartbeat scheduler for periodic agent tasks. In multi-replica deployments only one replica may run it; by default the operator pins it to pod-0 (ordinal 0), and `schedulerGating` selects lease-based failover instead -- see [Scheduler gating and failover](#scheduler-gating-and-failover):
 
 ```yaml
 spec:
   heartbeat:
-    enabled: true        # default: true
-    intervalMS: 60000    # default: 60000 (1 minute)
+    enabled: true             # default: true
+    intervalMS: 60000         # default: 60000 (1 minute)
+    schedulerGating: ordinal  # default; "lease" enables automatic failover
 ```
 
 ### Persistent Storage
@@ -637,6 +638,46 @@ scaleTargetRef:
   name: my-paperclip
 ```
 
+#### Scheduler gating and failover
+
+The heartbeat scheduler must run on exactly one replica. `spec.heartbeat.schedulerGating` selects how that is enforced at `replicas > 1`:
+
+```yaml
+spec:
+  heartbeat:
+    schedulerGating: lease   # "ordinal" (default), "lease", or "auto"
+```
+
+| Mode | How it works | Failover |
+|------|--------------|----------|
+| `ordinal` (default) | The operator wraps the container entrypoint so only pod-0 of the StatefulSet sets `HEARTBEAT_SCHEDULER_ENABLED=true`. StatefulSet only -- Deployment pods have no stable ordinals, so the wrapper is skipped and the operator reports `SchedulerGatingValid: False` | None: while pod-0 is down, no scheduler runs |
+| `lease` | The operator sets no scheduler env at all and delegates to the app's lease-based leader election (requires an app version with scheduler leases, paperclipai/paperclip#7995) | Automatic: a surviving replica takes over the lease |
+| `auto` | Currently resolves to `ordinal`; will flip to `lease` once the minimum supported app version ships lease leadership | Follows the resolved mode |
+
+**Version skew.** What actually runs for each combination of operator gating mode and app image:
+
+| Operator gating | App without leases | App with leases (>= the #7995 release) |
+|-----------------|--------------------|----------------------------------------|
+| `ordinal` (default) | pod-0 pinned, no failover | pod-0 pinned (the wrapper wins: only pod-0 is a lease candidate) |
+| `lease` | ALL replicas run the scheduler -- unsafe, do not use | automatic failover |
+
+**Migrating from ordinal to lease.** Order matters: setting `lease` against an app image without lease support removes the only gate and every replica runs the scheduler.
+
+1. Upgrade the app image to a version that includes lease-based scheduler leadership (paperclipai/paperclip#7995).
+2. Set `spec.heartbeat.schedulerGating: lease`.
+3. Optionally remove any manual `HEARTBEAT_SCHEDULER_ENABLED` pinning you carry in `spec.env`.
+
+**Leader observability.** While lease gating is active at `replicas > 1`, the operator polls each server pod's unauthenticated `/api/health` and surfaces the lease holder:
+
+- `status.schedulerLeader` records the leader pod name:
+
+  ```bash
+  kubectl get instance my-paperclip -o jsonpath='{.status.schedulerLeader}'
+  ```
+
+- server pods are labeled `paperclip.inc/role=scheduler` (lease holder) or `paperclip.inc/role=web`
+- on Deployment workloads the leader pod also carries the `controller.kubernetes.io/pod-deletion-cost` annotation, so ReplicaSet scale-in prefers removing web replicas and avoids needless failovers
+
 #### Horizontal Pod Autoscaler
 
 ```yaml

api/v1alpha1/paperclipinstance_types.go

@@ -18,6 +18,7 @@ package v1alpha1
 
 import (
 	corev1 "k8s.io/api/core/v1"
+	networkingv1 "k8s.io/api/networking/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
@@ -502,6 +503,14 @@ type ObjectStorageSpec struct {
 	// CredentialsSecretRef references a Secret containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
 	// +optional
 	CredentialsSecretRef *corev1.LocalObjectReference `json:"credentialsSecretRef,omitempty"`
+
+	// ForcePathStyle forces path-style S3 addressing (bucket in the URL path
+	// rather than the hostname). Defaults to true when provider is "minio"
+	// (virtual-hosted addressing needs wildcard DNS that in-cluster MinIO
+	// deployments don't have), false otherwise. Maps to
+	// PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE.
+	// +optional
+	ForcePathStyle *bool `json:"forcePathStyle,omitempty"`
 }
 
 // HeartbeatSpec configures the agent heartbeat scheduler.
@@ -515,6 +524,21 @@ type HeartbeatSpec struct {
 	// +kubebuilder:default=60000
 	// +optional
 	IntervalMS int32 `json:"intervalMS,omitempty"`
+
+	// SchedulerGating selects how the heartbeat scheduler is pinned to a single
+	// replica when replicas > 1.
+	//  - "ordinal" (default): a shell wrapper enables the scheduler only on the
+	//    StatefulSet's ordinal-0 pod. Works with every app version, but has no
+	//    failover and requires the StatefulSet workload.
+	//  - "lease": no env manipulation - every replica participates in the app's
+	//    lease-based leader election with automatic failover. Requires an app
+	//    version with scheduler leases.
+	//  - "auto": currently behaves like "ordinal"; will default to "lease" once
+	//    the minimum supported app version includes lease leadership.
+	// +kubebuilder:validation:Enum=ordinal;lease;auto
+	// +kubebuilder:default=ordinal
+	// +optional
+	SchedulerGating string `json:"schedulerGating,omitempty"`
 }
 
 // AdaptersSpec configures agent runtime adapters.
@@ -861,9 +885,12 @@ type SecuritySpec struct {
 // NetworkPolicySpec configures network isolation.
 type NetworkPolicySpec struct {
 	// Enabled controls whether a NetworkPolicy is created. Defaults to true.
+	// Pointer so an explicit `false` survives marshaling (a plain bool with
+	// omitempty is dropped and re-defaulted to true by the API server on
+	// every controller update — same bug class as PersistenceSpec.Enabled).
 	// +kubebuilder:default=true
 	// +optional
-	Enabled bool `json:"enabled,omitempty"`
+	Enabled *bool `json:"enabled,omitempty"`
 
 	// AllowIngressCIDRs specifies additional CIDR blocks allowed to reach the Paperclip service.
 	// +optional
@@ -876,6 +903,14 @@ type NetworkPolicySpec struct {
 	// +listType=set
 	// +kubebuilder:validation:items:Pattern=`^([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}$`
 	AllowEgressCIDRs []string `json:"allowEgressCIDRs,omitempty"`
+
+	// ExtraEgress appends additional egress rules verbatim to the
+	// operator-managed NetworkPolicy — e.g. to reach an in-cluster
+	// object-storage endpoint (spec.objectStorage) or other services the
+	// default rules don't cover. Required when networkPolicy is enabled and
+	// objectStorage points at an in-cluster endpoint.
+	// +optional
+	ExtraEgress []networkingv1.NetworkPolicyEgressRule `json:"extraEgress,omitempty"`
 }
 
 // RBACSpec configures ServiceAccount and RBAC.
@@ -1264,6 +1299,14 @@ type BackupS3Spec struct {
 	// CredentialsSecretRef references a Secret containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
 	// +optional
 	CredentialsSecretRef *corev1.LocalObjectReference `json:"credentialsSecretRef,omitempty"`
+
+	// ForcePathStyle forces path-style S3 addressing (bucket in the URL path
+	// rather than the hostname). Defaults to true when provider is "minio"
+	// (virtual-hosted addressing needs wildcard DNS that in-cluster MinIO
+	// deployments don't have), false otherwise. Maps to
+	// PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE.
+	// +optional
+	ForcePathStyle *bool `json:"forcePathStyle,omitempty"`
 }
 
 // --- Status types ---
@@ -1326,6 +1369,10 @@ type InstanceStatus struct {
 	// Selector is the label selector for the scale subresource (string form).
 	// +optional
 	Selector string `json:"selector,omitempty"`
+
+	// SchedulerLeader is the pod currently holding the scheduler lease (lease gating only).
+	// +optional
+	SchedulerLeader string `json:"schedulerLeader,omitempty"`
 }
 
 // ManagedResources tracks the names of managed Kubernetes resources.