feat: Deployment workload profile, multi-replica preconditions, scale subresource (#81)

* feat(api): spec.workload field selecting the server workload kind

Adds spec.workload (StatefulSet | Deployment | auto, default StatefulSet)
to InstanceSpec. Deployment is intended for stateless multi-replica
instances (external/managed database + object storage, persistence
disabled); auto picks Deployment when persistence is disabled and the
database mode is not embedded. Includes regenerated CRD and synced Helm
chart CRD template.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(resources): extract shared server pod template

Mechanically moves the pod template construction out of BuildStatefulSet
into BuildServerPodTemplate (internal/resources/podtemplate.go) so a
Deployment workload builder can share it. Renames StatefulSetReplicas to
WorkloadReplicas since it will serve both workload kinds (no other
callers, so no deprecated alias).

The built StatefulSet is byte-identical: golden JSON dumps of eight
instance variants (persistence on/off, multi-replica heartbeat,
tailscale, platform-admin, k8s execution, object storage, suspended)
match exactly before and after the extraction, and all existing builder
tests pass unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(resources): Deployment workload builder

Adds BuildDeployment, sharing BuildServerPodTemplate, the workload name
(DeploymentName == StatefulSetName), labels, selector, and replica
handling (WorkloadReplicas, incl. suspend scale-to-zero) with the
StatefulSet builder so the Service keeps selecting the server pods by
label across workload kinds. Rollouts surge (maxSurge=1,
maxUnavailable=0) so capacity never drops during updates.

Also adds UseDeploymentWorkload resolving spec.workload, including the
auto rule (no persistence AND non-embedded database). Covered by builder
tests plus a truth-table test for the workload resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(controller): workload-kind reconcile with safe kind migration

Reconcile the server workload as a Deployment when
spec.workload selects it (explicitly or via auto) and delete the stale
counterpart kind afterwards (Normal event WorkloadKindMigrated), so a
spec.workload flip migrates cleanly. reconcileDeployment mirrors
reconcileStatefulSet including HPA replica preservation and suspend
scale-to-zero, and maintains a DeploymentReady condition (the
counterpart kind's readiness condition is removed to avoid stale gating).

PVC-safety override: spec.workload=Deployment with persistence enabled
keeps the StatefulSet and reports the advisory WorkloadProfileValid=False
(reason PersistenceRequiresStatefulSet, Warning event) instead of
attaching the RWO data PVC to surging Deployment pods. The HPA
scaleTargetRef follows the effective workload kind via the new
resources.EffectiveWorkloadIsDeployment helper.

Fix a round-trip bug that blocked stateless profiles entirely:
PersistenceSpec.Enabled was a plain bool with omitempty, so
enabled=false was dropped on marshal and re-defaulted to true by the
API server on every controller update of the CR (e.g. adding the
finalizer). It is now *bool (generated CRD schema is unchanged) read
through resources.PersistenceEnabled.

Adds apps/deployments RBAC (marker, role.yaml, Helm chart) and the
Deployment to the controller's Owns watches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(controller): multi-replica precondition and PDB warnings

Before the workload step, maintain the advisory
MultiReplicaPreconditions condition whenever effective replicas > 1:
False (Warning event, once per transition) when the embedded database
is selected (it cannot be shared between replicas) or objectStorage is
unset (required for shared file state), True when both prerequisites
are met. At replicas <= 1 the condition is removed entirely so
single-replica instances carry no stale noise. Advisory conditions are
excluded from the Ready aggregation.

Also emit a Warning event PDBMayBlockDrains (event only, no condition)
when podDisruptionBudget.minAvailable >= autoScaling.minReplicas, since
disruptionsAllowed is then 0 at minimum scale and node drains can
stall.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(api): scale subresource and managedResources.deployment tracking

Adds the scale subresource to the Instance CRD
(specpath=.spec.availability.replicas, statuspath=.status.replicas,
selectorpath=.status.selector) so kubectl scale and KEDA/HPA-style
tooling can drive replicas through /scale. updateStatus populates
status.replicas and status.selector from the active server workload
(Deployment or StatefulSet; names and selectors are identical by
construction, so the selector string is stable across kind migrations).

kubectl scale writes spec.availability.replicas, which the reconcilers
already ignore while autoScaling is enabled (HPA replica preservation),
so scaling via /scale is a documented no-op under HPA and there is no
controller/autoscaler fight.

Also adds status.managedResources.deployment, set in reconcileDeployment
and cleared in the StatefulSet path (and vice versa), closing the Task 2
leftover where a kind migration left no record of the active workload.

Covered by an envtest case that GETs /scale, scales 1 -> 3 through the
subresource, and asserts the Deployment spec, status.replicas, and
status.selector follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: workload profiles, scale subresource, and PDB guidance in scaling section

Rewrites the README Scaling section: workload profile table
(StatefulSet default vs Deployment for stateless multi-replica, auto),
multi-replica preconditions and the MultiReplicaPreconditions condition,
the PVC-safety rule behind WorkloadProfileValid, kubectl scale / KEDA
scaleTargetRef usage of the new scale subresource (and that it is a
no-op under HPA), and PDB drain guidance (minAvailable strictly below
autoScaling.minReplicas, unhealthyPodEvictionPolicy: AlwaysAllow
example).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: regenerate API reference for workload, scale, and condition fields

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: stubbi

README.md

@@ -595,6 +595,23 @@ spec:
 
 ### Scaling
 
+#### Workload profiles
+
+`spec.workload` selects how the server runs:
+
+```yaml
+spec:
+  workload: auto   # "StatefulSet" (default), "Deployment", or "auto"
+```
+
+| Profile | Use for | Behavior |
+|---------|---------|----------|
+| `StatefulSet` (default) | Single replica, persistence, or embedded database | Stable pod identity with a per-instance PVC; rolling updates replace pods in place |
+| `Deployment` | Stateless multi-replica (external/managed database + `objectStorage`, persistence off) | Surge rollouts (`maxSurge: 1`, `maxUnavailable: 0`) so capacity never drops, no AZ-pinned per-ordinal PVCs, HPA-friendly scale-in |
+| `auto` | Let the operator decide | Deployment when persistence is disabled and the database is not embedded; StatefulSet otherwise |
+
+> **PVC safety:** `workload: Deployment` requires `storage.persistence.enabled: false` -- the ReadWriteOnce data PVC cannot be shared by surging Deployment pods. If persistence is still enabled, the operator keeps the StatefulSet and reports the `WorkloadProfileValid: False` condition.
+
 #### Manual replicas
 
 ```yaml
@@ -603,7 +620,22 @@ spec:
     replicas: 3
 ```
 
-When running multiple replicas, use `database.mode: external` with a production-grade PostgreSQL service and configure `objectStorage` for shared file access. The operator ensures only pod-0 runs the heartbeat scheduler.
+When running multiple replicas, use `database.mode: external` (or `managed`) with a production-grade PostgreSQL service and configure `objectStorage` for shared file access -- the operator surfaces a `MultiReplicaPreconditions: False` condition (plus a Warning event) at `replicas > 1` until both are in place. The operator ensures only one pod runs the heartbeat scheduler.
+
+The Instance CRD exposes the scale subresource (`status.replicas` / `status.selector` track the active workload), so standard tooling works:
+
+```bash
+kubectl scale instance/my-paperclip --replicas=3
+```
+
+External autoscalers like KEDA can target the instance directly:
+
+```yaml
+scaleTargetRef:
+  apiVersion: paperclip.inc/v1alpha1
+  kind: Instance
+  name: my-paperclip
+```
 
 #### Horizontal Pod Autoscaler
 
@@ -618,7 +650,7 @@ spec:
       targetMemoryUtilizationPercentage: 70       # optional
 ```
 
-When auto-scaling is enabled, the HPA manages the replica count and the StatefulSet's `replicas` field is set to nil.
+When auto-scaling is enabled, the HPA owns the replica count: the operator preserves the workload's current `replicas` on every reconcile, and `spec.availability.replicas` (including writes via `kubectl scale`) is ignored.
 
 #### Pod Disruption Budget
 
@@ -631,6 +663,23 @@ spec:
       # or: maxUnavailable: 1
 ```
 
+Keep `minAvailable` strictly below `autoScaling.minReplicas`: when they are equal, the PDB allows zero disruptions at minimum scale and node drains stall (the operator emits a `PDBMayBlockDrains` warning event). If unhealthy pods blocking drains is a concern, manage your own PDB instead of the operator's and set the eviction policy:
+
+```yaml
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: my-paperclip-pdb
+spec:
+  minAvailable: 1
+  unhealthyPodEvictionPolicy: AlwaysAllow   # evict crash-looping pods during drains
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+      app.kubernetes.io/instance: my-paperclip
+      app.kubernetes.io/component: server
+```
+
 #### Topology Spread Constraints
 
 Spread pods across zones or nodes for improved availability:

api/v1alpha1/paperclipinstance_types.go

@@ -124,6 +124,20 @@ type InstanceSpec struct {
 	// +optional
 	Availability AvailabilitySpec `json:"availability,omitempty"`
 
+	// Workload selects the server workload kind.
+	//  - "StatefulSet" (default): per-instance PVC, stable identity. Required for
+	//    embedded database mode and for persistence-backed instances.
+	//  - "Deployment": stateless pods (ephemeral scratch); intended for
+	//    database.mode external/managed with objectStorage when replicas > 1.
+	//    Enables surge rollouts and avoids AZ-pinned per-ordinal PVCs under
+	//    autoscaling. Requires storage.persistence.enabled=false.
+	//  - "auto": Deployment when persistence is disabled AND database mode is
+	//    not embedded; StatefulSet otherwise.
+	// +kubebuilder:validation:Enum=StatefulSet;Deployment;auto
+	// +kubebuilder:default=StatefulSet
+	// +optional
+	Workload string `json:"workload,omitempty"`
+
 	// Probes configures liveness, readiness, and startup probes.
 	// +optional
 	Probes ProbesSpec `json:"probes,omitempty"`
@@ -445,10 +459,14 @@ type StorageSpec struct {
 
 // PersistenceSpec configures PVC settings.
 type PersistenceSpec struct {
+	// A pointer is required for round-tripping: with a plain bool plus
+	// omitempty, enabled=false is dropped on marshal and the API server
+	// re-defaults it to true on every controller update of the CR.
+
 	// Enabled controls whether a PVC is created. Defaults to true.
 	// +kubebuilder:default=true
 	// +optional
-	Enabled bool `json:"enabled,omitempty"`
+	Enabled *bool `json:"enabled,omitempty"`
 
 	// Size is the PVC storage size.
 	// +kubebuilder:default="5Gi"
@@ -1065,8 +1083,10 @@ type LoggingSpec struct {
 
 // AvailabilitySpec configures scaling and pod scheduling.
 type AvailabilitySpec struct {
-	// Replicas is the desired number of Paperclip server pods.
-	// Ignored when autoScaling is enabled (the HPA manages replicas).
+	// Replicas is the desired number of Paperclip server pods. This field is
+	// also the target of the scale subresource, so `kubectl scale` writes it.
+	// Ignored when autoScaling is enabled (the HPA manages replicas), which
+	// makes `kubectl scale` a no-op while the HPA is active.
 	// +kubebuilder:default=1
 	// +kubebuilder:validation:Minimum=1
 	// +optional
@@ -1298,13 +1318,23 @@ type InstanceStatus struct {
 	// AutoUpdate tracks the state of automatic image update checks.
 	// +optional
 	AutoUpdate *AutoUpdateStatus `json:"autoUpdate,omitempty"`
+
+	// Replicas is the observed replica count of the active server workload (scale subresource).
+	// +optional
+	Replicas int32 `json:"replicas,omitempty"`
+
+	// Selector is the label selector for the scale subresource (string form).
+	// +optional
+	Selector string `json:"selector,omitempty"`
 }
 
 // ManagedResources tracks the names of managed Kubernetes resources.
 type ManagedResources struct {
 	// +optional
 	StatefulSet string `json:"statefulSet,omitempty"`
 	// +optional
+	Deployment string `json:"deployment,omitempty"`
+	// +optional
 	Service string `json:"service,omitempty"`
 	// +optional
 	ConfigMap string `json:"configMap,omitempty"`
@@ -1375,6 +1405,7 @@ type AutoUpdateStatus struct {
 
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
+// +kubebuilder:subresource:scale:specpath=.spec.availability.replicas,statuspath=.status.replicas,selectorpath=.status.selector
 // +kubebuilder:resource:shortName=pci
 // +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
 // +kubebuilder:printcolumn:name="Endpoint",type=string,JSONPath=`.status.endpoint`

api/v1alpha1/zz_generated.deepcopy.go

@@ -1589,8 +1589,10 @@ spec:
                   replicas:
                     default: 1
                     description: |-
-                      Replicas is the desired number of Paperclip server pods.
-                      Ignored when autoScaling is enabled (the HPA manages replicas).
+                      Replicas is the desired number of Paperclip server pods. This field is
+                      also the target of the scale subresource, so `kubectl scale` writes it.
+                      Ignored when autoScaling is enabled (the HPA manages replicas), which
+                      makes `kubectl scale` a no-op while the HPA is active.
                     format: int32
                     minimum: 1
                     type: integer
@@ -8269,6 +8271,23 @@ spec:
                         type: object
                     type: object
                 type: object
+              workload:
+                default: StatefulSet
+                description: |-
+                  Workload selects the server workload kind.
+                   - "StatefulSet" (default): per-instance PVC, stable identity. Required for
+                     embedded database mode and for persistence-backed instances.
+                   - "Deployment": stateless pods (ephemeral scratch); intended for
+                     database.mode external/managed with objectStorage when replicas > 1.
+                     Enables surge rollouts and avoids AZ-pinned per-ordinal PVCs under
+                     autoscaling. Requires storage.persistence.enabled=false.
+                   - "auto": Deployment when persistence is disabled AND database mode is
+                     not embedded; StatefulSet otherwise.
+                enum:
+                - StatefulSet
+                - Deployment
+                - auto
+                type: string
             type: object
           status:
             description: InstanceStatus defines the observed state of Instance.
@@ -8381,6 +8400,8 @@ spec:
                     type: string
                   databaseStatefulSet:
                     type: string
+                  deployment:
+                    type: string
                   grafanaDashboardInstance:
                     type: string
                   grafanaDashboardOperator:
@@ -8421,6 +8442,11 @@ spec:
                 - Updating
                 - Suspended
                 type: string
+              replicas:
+                description: Replicas is the observed replica count of the active
+                  server workload (scale subresource).
+                format: int32
+                type: integer
               restore:
                 description: Restore tracks the state of the latest restore operation.
                 properties:
@@ -8432,10 +8458,18 @@ spec:
                     description: Result is the result of the restore.
                     type: string
                 type: object
+              selector:
+                description: Selector is the label selector for the scale subresource
+                  (string form).
+                type: string
             type: object
         type: object
     served: true
     storage: true
     subresources:
+      scale:
+        labelSelectorPath: .status.selector
+        specReplicasPath: .spec.availability.replicas
+        statusReplicasPath: .status.replicas
       status: {}
 {{- end }}

charts/paperclip-operator/templates/crds/paperclip.inc_instances.yaml

@@ -70,7 +70,7 @@ rules:
     verbs: ["get", "create", "delete"]
   # Apps
   - apiGroups: ["apps"]
-    resources: ["statefulsets"]
+    resources: ["deployments", "statefulsets"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   # Networking
   - apiGroups: ["networking.k8s.io"]

charts/paperclip-operator/templates/rbac.yaml

@@ -1583,8 +1583,10 @@ spec:
                   replicas:
                     default: 1
                     description: |-
-                      Replicas is the desired number of Paperclip server pods.
-                      Ignored when autoScaling is enabled (the HPA manages replicas).
+                      Replicas is the desired number of Paperclip server pods. This field is
+                      also the target of the scale subresource, so `kubectl scale` writes it.
+                      Ignored when autoScaling is enabled (the HPA manages replicas), which
+                      makes `kubectl scale` a no-op while the HPA is active.
                     format: int32
                     minimum: 1
                     type: integer
@@ -8263,6 +8265,23 @@ spec:
                         type: object
                     type: object
                 type: object
+              workload:
+                default: StatefulSet
+                description: |-
+                  Workload selects the server workload kind.
+                   - "StatefulSet" (default): per-instance PVC, stable identity. Required for
+                     embedded database mode and for persistence-backed instances.
+                   - "Deployment": stateless pods (ephemeral scratch); intended for
+                     database.mode external/managed with objectStorage when replicas > 1.
+                     Enables surge rollouts and avoids AZ-pinned per-ordinal PVCs under
+                     autoscaling. Requires storage.persistence.enabled=false.
+                   - "auto": Deployment when persistence is disabled AND database mode is
+                     not embedded; StatefulSet otherwise.
+                enum:
+                - StatefulSet
+                - Deployment
+                - auto
+                type: string
             type: object
           status:
             description: InstanceStatus defines the observed state of Instance.
@@ -8375,6 +8394,8 @@ spec:
                     type: string
                   databaseStatefulSet:
                     type: string
+                  deployment:
+                    type: string
                   grafanaDashboardInstance:
                     type: string
                   grafanaDashboardOperator:
@@ -8415,6 +8436,11 @@ spec:
                 - Updating
                 - Suspended
                 type: string
+              replicas:
+                description: Replicas is the observed replica count of the active
+                  server workload (scale subresource).
+                format: int32
+                type: integer
               restore:
                 description: Restore tracks the state of the latest restore operation.
                 properties:
@@ -8426,9 +8452,17 @@ spec:
                     description: Result is the result of the restore.
                     type: string
                 type: object
+              selector:
+                description: Selector is the label selector for the scale subresource
+                  (string form).
+                type: string
             type: object
         type: object
     served: true
     storage: true
     subresources:
+      scale:
+        labelSelectorPath: .status.selector
+        specReplicasPath: .spec.availability.replicas
+        statusReplicasPath: .status.replicas
       status: {}

config/crd/bases/paperclip.inc_instances.yaml

@@ -83,6 +83,7 @@ rules:
 - apiGroups:
   - apps
   resources:
+  - deployments
   - statefulsets
   verbs:
   - create

config/rbac/role.yaml

@@ -208,7 +208,7 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `replicas` _integer_ | Replicas is the desired number of Paperclip server pods.<br />Ignored when autoScaling is enabled (the HPA manages replicas). | 1 | Minimum: 1 <br />Optional: \{\} <br /> |
+| `replicas` _integer_ | Replicas is the desired number of Paperclip server pods. This field is<br />also the target of the scale subresource, so `kubectl scale` writes it.<br />Ignored when autoScaling is enabled (the HPA manages replicas), which<br />makes `kubectl scale` a no-op while the HPA is active. | 1 | Minimum: 1 <br />Optional: \{\} <br /> |
 | `podDisruptionBudget` _[PDBSpec](#pdbspec)_ | PodDisruptionBudget configures the PDB. |  | Optional: \{\} <br /> |
 | `autoScaling` _[AutoScalingSpec](#autoscalingspec)_ | AutoScaling configures the HorizontalPodAutoscaler. |  | Optional: \{\} <br /> |
 | `nodeSelector` _object (keys:string, values:string)_ | NodeSelector specifies node selection constraints. |  | Optional: \{\} <br /> |
@@ -601,6 +601,7 @@ _Appears in:_
 | `networking` _[NetworkingSpec](#networkingspec)_ | Networking configures service, ingress, and WebSocket settings. |  | Optional: \{\} <br /> |
 | `observability` _[ObservabilitySpec](#observabilityspec)_ | Observability configures metrics, logging, and monitoring. |  | Optional: \{\} <br /> |
 | `availability` _[AvailabilitySpec](#availabilityspec)_ | Availability configures scaling, PDB, and pod scheduling. |  | Optional: \{\} <br /> |
+| `workload` _string_ | Workload selects the server workload kind.<br /> - "StatefulSet" (default): per-instance PVC, stable identity. Required for<br />   embedded database mode and for persistence-backed instances.<br /> - "Deployment": stateless pods (ephemeral scratch); intended for<br />   database.mode external/managed with objectStorage when replicas > 1.<br />   Enables surge rollouts and avoids AZ-pinned per-ordinal PVCs under<br />   autoscaling. Requires storage.persistence.enabled=false.<br /> - "auto": Deployment when persistence is disabled AND database mode is<br />   not embedded; StatefulSet otherwise. | StatefulSet | Enum: [StatefulSet Deployment auto] <br />Optional: \{\} <br /> |
 | `probes` _[ProbesSpec](#probesspec)_ | Probes configures liveness, readiness, and startup probes. |  | Optional: \{\} <br /> |
 | `backup` _[BackupSpec](#backupspec)_ | Backup configures periodic backup to S3-compatible storage. |  | Optional: \{\} <br /> |
 | `restoreFrom` _string_ | RestoreFrom specifies a remote backup path to restore from on first boot. |  | Optional: \{\} <br /> |