feat(availability): configurable terminationGracePeriodSeconds + preStop for server drain (#106)

feat: configurable server termination grace + preStop drain hook

The Paperclip server pod hardcoded terminationGracePeriodSeconds=30 with no
preStop hook, so a deploy/rollout SIGKILLed in-flight agent runs 30s after
SIGTERM -- far shorter than a typical multi-minute run.

- api: add AvailabilitySpec.TerminationGracePeriodSeconds and a ServerDrain
  block (Enabled default true, TimeoutSeconds) to the Instance CRD.
- podtemplate: source the server pod grace from the spec, defaulting HIGH
  (1800s, matching the 30-min PAPERCLIP_HEARTBEAT_REAP_STALE_MS run window)
  so a rollout never kills an active run. The grace is a ceiling, not a fixed
  wait -- an idle pod whose server exits promptly still terminates fast.
- statefulset: add a gated preStop hook (sleep, default 15s, clamped to
  grace-1) on the server container that deregisters the terminating pod from
  the Service before SIGTERM, closing the endpoint-removal race. Shared by the
  Deployment workload path via BuildServerPodTemplate.
- tests: default/configured grace, default/disabled/custom/zero/clamped
  preStop, and the Deployment path.

Scope note: the preStop delivers the traffic-cutover half. Making in-flight
runs FINISH also requires the server image to soft-drain on SIGTERM; the
current fork handler hard-interrupts and requeues runs on SIGTERM, so the
higher grace only takes full effect once that soft-drain lands. Regenerated
CRDs/deepcopy/docs and synced the Helm chart CRD templates.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: stubbi

api/v1alpha1/paperclipinstance_types.go

@@ -1247,6 +1247,52 @@ type AvailabilitySpec struct {
 	// TopologySpreadConstraints specifies topology spread constraints.
 	// +optional
 	TopologySpreadConstraints []corev1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"`
+
+	// TerminationGracePeriodSeconds is the pod termination grace period for the
+	// Paperclip server pod. On shutdown (deploy/rollout/node drain) the kubelet
+	// sends SIGTERM and waits up to this many seconds before SIGKILL. The server
+	// uses this window to let in-flight agent runs finish (soft-drain) instead of
+	// being interrupted mid-run. When unset the operator defaults it high (see
+	// DefaultServerTerminationGracePeriodSeconds) so a rollout never kills an active
+	// run; the previous hardcoded 30s was far shorter than a typical multi-minute
+	// agent run. The grace period is a ceiling, not a fixed wait: the pod terminates
+	// as soon as the server exits, so a high value only delays SIGKILL for pods that
+	// still have work to drain.
+	// +kubebuilder:validation:Minimum=0
+	// +optional
+	TerminationGracePeriodSeconds *int64 `json:"terminationGracePeriodSeconds,omitempty"`
+
+	// ServerDrain configures graceful draining of in-flight agent runs when the
+	// Paperclip server pod is shutting down.
+	// +optional
+	ServerDrain *ServerDrainSpec `json:"serverDrain,omitempty"`
+}
+
+// ServerDrainSpec configures how the Paperclip server pod drains in-flight agent
+// runs on shutdown. The server itself soft-drains on SIGTERM (it stops accepting
+// new work and waits for active runs to finish within the pod's termination grace
+// period). This spec adds an optional container preStop hook that holds the
+// container in the "Terminating" state for a short, bounded window BEFORE SIGTERM
+// is delivered, so the pod's endpoints are deregistered from the Service first.
+// That prevents new requests from being routed to a pod that is about to drain,
+// which would otherwise race the soft-drain.
+type ServerDrainSpec struct {
+	// Enabled controls whether the server container gets a preStop drain hook.
+	// The soft-drain on SIGTERM is a property of the server image and is unaffected
+	// by this toggle; disabling only removes the pre-SIGTERM endpoint-deregistration
+	// delay. Defaults to true.
+	// +kubebuilder:default=true
+	// +optional
+	Enabled *bool `json:"enabled,omitempty"`
+
+	// TimeoutSeconds bounds how long the preStop hook sleeps before the kubelet
+	// delivers SIGTERM, giving Endpoints/EndpointSlice controllers time to remove
+	// this pod from the Service so in-flight-only traffic remains. Must be shorter
+	// than terminationGracePeriodSeconds (which also has to cover the SIGTERM
+	// soft-drain that follows). Defaults to DefaultServerDrainTimeoutSeconds.
+	// +kubebuilder:validation:Minimum=0
+	// +optional
+	TimeoutSeconds *int64 `json:"timeoutSeconds,omitempty"`
 }
 
 // PDBSpec configures a PodDisruptionBudget.

api/v1alpha1/zz_generated.deepcopy.go

@@ -1661,6 +1661,45 @@ spec:
                     format: int32
                     minimum: 1
                     type: integer
+                  serverDrain:
+                    description: |-
+                      ServerDrain configures graceful draining of in-flight agent runs when the
+                      Paperclip server pod is shutting down.
+                    properties:
+                      enabled:
+                        default: true
+                        description: |-
+                          Enabled controls whether the server container gets a preStop drain hook.
+                          The soft-drain on SIGTERM is a property of the server image and is unaffected
+                          by this toggle; disabling only removes the pre-SIGTERM endpoint-deregistration
+                          delay. Defaults to true.
+                        type: boolean
+                      timeoutSeconds:
+                        description: |-
+                          TimeoutSeconds bounds how long the preStop hook sleeps before the kubelet
+                          delivers SIGTERM, giving Endpoints/EndpointSlice controllers time to remove
+                          this pod from the Service so in-flight-only traffic remains. Must be shorter
+                          than terminationGracePeriodSeconds (which also has to cover the SIGTERM
+                          soft-drain that follows). Defaults to DefaultServerDrainTimeoutSeconds.
+                        format: int64
+                        minimum: 0
+                        type: integer
+                    type: object
+                  terminationGracePeriodSeconds:
+                    description: |-
+                      TerminationGracePeriodSeconds is the pod termination grace period for the
+                      Paperclip server pod. On shutdown (deploy/rollout/node drain) the kubelet
+                      sends SIGTERM and waits up to this many seconds before SIGKILL. The server
+                      uses this window to let in-flight agent runs finish (soft-drain) instead of
+                      being interrupted mid-run. When unset the operator defaults it high (see
+                      DefaultServerTerminationGracePeriodSeconds) so a rollout never kills an active
+                      run; the previous hardcoded 30s was far shorter than a typical multi-minute
+                      agent run. The grace period is a ceiling, not a fixed wait: the pod terminates
+                      as soon as the server exits, so a high value only delays SIGKILL for pods that
+                      still have work to drain.
+                    format: int64
+                    minimum: 0
+                    type: integer
                   tolerations:
                     description: Tolerations specifies pod tolerations.
                     items:

charts/paperclip-operator/templates/crds/paperclip.inc_instances.yaml

@@ -1655,6 +1655,45 @@ spec:
                     format: int32
                     minimum: 1
                     type: integer
+                  serverDrain:
+                    description: |-
+                      ServerDrain configures graceful draining of in-flight agent runs when the
+                      Paperclip server pod is shutting down.
+                    properties:
+                      enabled:
+                        default: true
+                        description: |-
+                          Enabled controls whether the server container gets a preStop drain hook.
+                          The soft-drain on SIGTERM is a property of the server image and is unaffected
+                          by this toggle; disabling only removes the pre-SIGTERM endpoint-deregistration
+                          delay. Defaults to true.
+                        type: boolean
+                      timeoutSeconds:
+                        description: |-
+                          TimeoutSeconds bounds how long the preStop hook sleeps before the kubelet
+                          delivers SIGTERM, giving Endpoints/EndpointSlice controllers time to remove
+                          this pod from the Service so in-flight-only traffic remains. Must be shorter
+                          than terminationGracePeriodSeconds (which also has to cover the SIGTERM
+                          soft-drain that follows). Defaults to DefaultServerDrainTimeoutSeconds.
+                        format: int64
+                        minimum: 0
+                        type: integer
+                    type: object
+                  terminationGracePeriodSeconds:
+                    description: |-
+                      TerminationGracePeriodSeconds is the pod termination grace period for the
+                      Paperclip server pod. On shutdown (deploy/rollout/node drain) the kubelet
+                      sends SIGTERM and waits up to this many seconds before SIGKILL. The server
+                      uses this window to let in-flight agent runs finish (soft-drain) instead of
+                      being interrupted mid-run. When unset the operator defaults it high (see
+                      DefaultServerTerminationGracePeriodSeconds) so a rollout never kills an active
+                      run; the previous hardcoded 30s was far shorter than a typical multi-minute
+                      agent run. The grace period is a ceiling, not a fixed wait: the pod terminates
+                      as soon as the server exits, so a high value only delays SIGKILL for pods that
+                      still have work to drain.
+                    format: int64
+                    minimum: 0
+                    type: integer
                   tolerations:
                     description: Tolerations specifies pod tolerations.
                     items:

config/crd/bases/paperclip.inc_instances.yaml

@@ -216,6 +216,8 @@ _Appears in:_
 | `affinity` _[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#affinity-v1-core)_ | Affinity specifies pod affinity rules. |  | Optional: \{\} <br /> |
 | `priorityClassName` _string_ | PriorityClassName sets the scheduling PriorityClass on the product pod so<br />it can preempt lower-priority workloads instead of sitting Pending when the<br />node pool is full. Leave empty for the cluster default priority. |  | Optional: \{\} <br /> |
 | `topologySpreadConstraints` _[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#topologyspreadconstraint-v1-core) array_ | TopologySpreadConstraints specifies topology spread constraints. |  | Optional: \{\} <br /> |
+| `terminationGracePeriodSeconds` _integer_ | TerminationGracePeriodSeconds is the pod termination grace period for the<br />Paperclip server pod. On shutdown (deploy/rollout/node drain) the kubelet<br />sends SIGTERM and waits up to this many seconds before SIGKILL. The server<br />uses this window to let in-flight agent runs finish (soft-drain) instead of<br />being interrupted mid-run. When unset the operator defaults it high (see<br />DefaultServerTerminationGracePeriodSeconds) so a rollout never kills an active<br />run; the previous hardcoded 30s was far shorter than a typical multi-minute<br />agent run. The grace period is a ceiling, not a fixed wait: the pod terminates<br />as soon as the server exits, so a high value only delays SIGKILL for pods that<br />still have work to drain. |  | Minimum: 0 <br />Optional: \{\} <br /> |
+| `serverDrain` _[ServerDrainSpec](#serverdrainspec)_ | ServerDrain configures graceful draining of in-flight agent runs when the<br />Paperclip server pod is shutting down. |  | Optional: \{\} <br /> |
 
 
 #### BackupS3Spec
@@ -655,6 +657,7 @@ _Appears in:_
 | `backend` _string_ | Backend selects the per-run workload primitive. "job" runs each agent as a<br />fire-and-forget batch/v1 Job (log-scraped output); "sandbox-cr" creates a<br />long-lived agent-sandbox CR (agents.x-k8s.io) that the server execs into.<br />Maps to PAPERCLIP_K8S_BACKEND. | job | Enum: [job sandbox-cr] <br />Optional: \{\} <br /> |
 | `runtimeClassName` _string_ | RuntimeClassName is the RuntimeClass applied to agent pods (e.g. "gvisor")<br />for an extra kernel-isolation boundary. Maps to<br />PAPERCLIP_K8S_RUNTIME_CLASS_NAME. When set, the execution ClusterRole also<br />grants get/use on the named RuntimeClass. |  | Optional: \{\} <br /> |
 | `egressMode` _string_ | EgressMode selects how per-tenant egress is enforced. "standard" uses plain<br />NetworkPolicy (CIDR-only, cannot match FQDNs); "cilium" uses a<br />CiliumNetworkPolicy for exact FQDN allow-listing (requires the Cilium CNI).<br />Maps to PAPERCLIP_K8S_EGRESS_MODE. |  | Enum: [standard cilium] <br />Optional: \{\} <br /> |
+| `egressPolicy` _string_ | EgressPolicy selects the overall egress posture for tenant sandboxes.<br />"allowlist" (default) restricts egress to EgressAllowFQDNs/EgressAllowCIDRs.<br />"open-internet" allows public internet on ports 80/443 while blocking<br />private ranges, link-local metadata, and CGNAT. | allowlist | Enum: [allowlist open-internet] <br />Optional: \{\} <br /> |
 | `egressAllowFQDNs` _string array_ | EgressAllowFQDNs is the list of fully-qualified domain names tenant agent<br />pods may reach (e.g. the LLM gateway and required APIs). Enforced exactly<br />only under EgressMode "cilium". Maps to PAPERCLIP_K8S_EGRESS_ALLOW_FQDNS<br />(comma-separated). |  | Optional: \{\} <br /> |
 | `egressAllowCIDRs` _string array_ | EgressAllowCIDRs is the list of CIDR blocks tenant agent pods may reach, in<br />addition to (or as the standard-mode substitute for) the FQDN allow-list.<br />Maps to PAPERCLIP_K8S_EGRESS_ALLOW_CIDRS (comma-separated). |  | Optional: \{\} <br /> |
 | `namespacePrefix` _string_ | NamespacePrefix is prepended to each derived per-tenant namespace name,<br />letting multiple instances share a cluster without namespace collisions.<br />Maps to PAPERCLIP_K8S_NAMESPACE_PREFIX. |  | Optional: \{\} <br /> |
@@ -1152,6 +1155,30 @@ _Appears in:_
 | `allowedActions` _[SelfConfigAction](#selfconfigaction) array_ | AllowedActions restricts which action categories the agent can perform.<br />If empty and enabled is true, no actions are allowed (fail-safe). |  | Enum: [plugins config envVars] <br />MaxItems: 3 <br />Optional: \{\} <br /> |
 
 
+#### ServerDrainSpec
+
+
+
+ServerDrainSpec configures how the Paperclip server pod drains in-flight agent
+runs on shutdown. The server itself soft-drains on SIGTERM (it stops accepting
+new work and waits for active runs to finish within the pod's termination grace
+period). This spec adds an optional container preStop hook that holds the
+container in the "Terminating" state for a short, bounded window BEFORE SIGTERM
+is delivered, so the pod's endpoints are deregistered from the Service first.
+That prevents new requests from being routed to a pod that is about to drain,
+which would otherwise race the soft-drain.
+
+
+
+_Appears in:_
+- [AvailabilitySpec](#availabilityspec)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `enabled` _boolean_ | Enabled controls whether the server container gets a preStop drain hook.<br />The soft-drain on SIGTERM is a property of the server image and is unaffected<br />by this toggle; disabling only removes the pre-SIGTERM endpoint-deregistration<br />delay. Defaults to true. | true | Optional: \{\} <br /> |
+| `timeoutSeconds` _integer_ | TimeoutSeconds bounds how long the preStop hook sleeps before the kubelet<br />delivers SIGTERM, giving Endpoints/EndpointSlice controllers time to remove<br />this pod from the Service so in-flight-only traffic remains. Must be shorter<br />than terminationGracePeriodSeconds (which also has to cover the SIGTERM<br />soft-drain that follows). Defaults to DefaultServerDrainTimeoutSeconds. |  | Minimum: 0 <br />Optional: \{\} <br /> |
+
+
 #### ServiceMonitorSpec