Skip to content

SKILL.md

Latest commit

 

History

History
113 lines (100 loc) · 32.2 KB

SKILL.md

File metadata and controls

113 lines (100 loc) · 32.2 KB
Raw
name openshift-deployment
description BC Gov Private Cloud OpenShift workload-manifest guidance: picking the right controller (Deployment, StatefulSet, DaemonSet, Job, CronJob) and shaping it for the platform — container resources and QoS, namespace LimitRange defaults, pod and container securityContext hardening (runAsNonRoot, drop ALL capabilities, no privilege escalation, read-only root filesystem) for the restricted-v2 SCC, liveness/readiness/startup probes, terminationGracePeriod and SIGTERM handling, PID-1 reaping with tini or dumb-init, PodDisruptionBudgets, horizontal scaling (HPA, VPA), and CronJob admission rules. Use when authoring or reviewing a workload manifest bound for a license-plate runtime namespace. Out of scope: NetworkPolicies, Vault/ESO, Argo CD/GitOps, Artifactory image pulls, Porter TCP exposure, storage class choice, Sysdig, ACS runtime policies, security assessment.
owner bcgov
tags
openshift
kubernetes
bcgov
deployment
statefulset
cronjob
hpa
pdb

OpenShift Deployment — BC Gov Private Cloud Workloads

Use When

  • Authoring or reviewing a new workload manifest (Deployment, StatefulSet, DaemonSet, Job, or CronJob — raw, Helm chart, or Kustomize base) bound for a license-plate -dev / -test / -prod namespace on Silver, Gold, GoldDR, or Emerald.
  • Picking which controller type fits a given workload (stateless API → Deployment; ordered identity + per-pod PVC → StatefulSet; one pod per node → DaemonSet; one-shot / parallel work → Job; scheduled → CronJob).
  • Hardening an existing workload to meet the BC Gov platform's mandatory pod hygiene — resource requests / QoS class, the pod and container securityContext (runAsNonRoot, allowPrivilegeEscalation: false, drop ALL capabilities, readOnlyRootFilesystem, seccompProfile), probes, SIGTERM and terminationGracePeriodSeconds, PID-1 reaping, PDB, HPA (with explicit rampup / rampdown), and pod anti-affinity / nodeAffinity / topologySpreadConstraints.
  • Setting the securityContext so a pod is admitted by the namespace's default restricted-v2 Security Context Constraint, and adding the writable emptyDir volumes (e.g. for /tmp) that a readOnlyRootFilesystem: true container needs.
  • Adding VerticalPodAutoscaler recommendations (updateMode: "Off") on a workload with a drifting resource footprint.
  • Getting past a Kyverno workload-admission rejection — CronJob cadence (≥ 5 min, ≥ 1 h with a PVC), CRON_TZ= in schedule, or the Emerald-only DataClass label rule.
  • Diagnosing pod-level failures on a workload you own — CrashLoopBackOff, OOMKilled, Evicted, TerminationGracePeriodExceeded, or a defunct-PIDs platform alert.

Don't Use When

The following concerns are each their own scope and belong in a sibling BC Gov OpenShift skill. Those siblings are planned — until they land in this repo, use the linked upstream Kubernetes / Red Hat docs or the BC Gov Private Cloud TechDocs:

  • Writing or debugging NetworkPolicy objects (default-deny, allow-from-router, allow-from-same-namespace, SDN-vs-NSX-T egress) — openshift-networking (planned).
  • Sourcing secrets from Vault via External Secrets Operator, or rotating Vault data — openshift-secrets (planned).
  • Choosing or configuring an image registry pull path, creating an ArtifactoryServiceAccount, or rotating Artifactory credentials — openshift-images (planned).
  • Onboarding to Argo CD, authoring a GitOpsTeam CR, or designing a bcgov-c/tenant-gitops-<licenseplate> repo layout — openshift-gitops (planned).
  • Picking a storage class, requesting a quota tier, or designing PVC backups — openshift-storage (planned).
  • Onboarding to Sysdig Monitor, building dashboards, or producing the evidence for a quota-increase request — openshift-sysdig (planned).
  • Configuring Red Hat Advanced Cluster Security (ACS) runtime policies, image scanning, or producing a security-assessment artifact — openshift-acs (planned). The pod and container securityContext manifest field itself is in scope here; ACS is the cluster-side enforcement / scanning layer on top of it.
  • Exposing a non-HTTPS TCP port via the Porter Operator (TransportServerClaim) — openshift-networking (planned).
  • Designing CI workflows (GitHub Actions chassis, OIDC, fork-gate) — sibling github-actions skill.
  • Anything on AKS, EKS, or vanilla Kubernetes outside the BC Gov Private Cloud.

Workflow

  1. Pick the controller from the workload's identity model, not personal habit. Stateless, fungible replicas behind a Service → Deployment. Ordered identity, stable network ID, per-pod PVC, controlled rolling restarts (databases, brokers, leader-elected services) → StatefulSet. Exactly one pod per node (node-agent, log shipper, CSI side-car) → DaemonSet. Run-to-completion batch work → Job (completions / parallelism for fan-out). Scheduled batch → CronJob (subject to the cadence rules in step 8). Avoid DeploymentConfig for new work — it's an OpenShift-only legacy resource and the platform's automation treats it identically to Deployment anyway.
  2. Set requests on every container and pick a deliberate QoS class. Guaranteed (requests == limits, both set for CPU and memory) is what ideal production workloads should run — the scheduler evicts these last under node pressure. Burstable (requests < limits, or only request is set) is reasonable for most workloads in BCGov. If you omit a value, the namespace LimitRange injects cpu: 50m / memory: 256Mi requests and a memory limit equal to the namespace memory quota (capped at 16 GiB). Those defaults are conservative cluster-wide guardrails, so override them with values that fit your workload.
  3. Right-size requests using observed data. Build the manifest with a starting guess, deploy to -dev, let it run a representative load (or steady idle for a week), then read CPU/memory P95 and P99 from Sysdig and adjust. Add VerticalPodAutoscaler in updateMode: "Off" (recommend-only) for long-lived workloads that drift over time; never run VPA in Auto mode alongside HPA on the same metric.
  4. Configure liveness, readiness, and (if needed) startup probes; keep them cheap and downstream-independent. Readiness probe gates Service endpoints: it must go not ready when the pod can't serve traffic (warming caches, lost downstream dep, draining) and ready when it can. Liveness probe restarts the container: only fail it when the process is genuinely wedged (deadlocked, stuck event loop) — not when a downstream is down. Startup probe is for slow-booting apps (legacy JVMs, large model loads): set generous failureThreshold * periodSeconds so the liveness probe doesn't restart the pod mid-boot. HTTP probes should be a dedicated /livez / /readyz endpoint that's cheap and doesn't touch downstreams; exec probes work but cost a process exec each interval, so keep them simple.
  5. Make sure PID 1 reaps zombie children. If your container ENTRYPOINT is your app process directly (["/app/server"]), it's PID 1 — and most app runtimes don't reap. Defunct children accumulate, the platform sends a defunct-PIDs alert, and the quick fix is oc delete pod (the root cause returns next deploy). The proper fix is to set the entrypoint to an init that reaps and forwards signals: tini (ENTRYPOINT ["tini", "--", "/app/server"]), dumb-init, or s6-overlay. Distroless and most OCP base images do not include one — add it in your Dockerfile.
  6. Trap SIGTERM, set a sane terminationGracePeriodSeconds, and never exceed 600 s. When a pod is being evicted (rollout, drain, scale-down), Kubernetes sends SIGTERM, waits up to terminationGracePeriodSeconds, then sends SIGKILL. Apps must use that window to stop accepting new work (close listeners, deregister from the LB), finish in-flight requests, flush buffers, and exit. Use ≤ 60 s for stateless services, up to 600 s for stateful (the Platform Operations team caps it at 600 s on node drains and force-kills anything still running). Add a preStop lifecycle hook if you need a "drain" pause between the LB deregister and the actual shutdown (["/bin/sleep", "10"] is a common pattern to let the Service's endpoint propagate).
  7. Always ship a PodDisruptionBudget for any multi-replica workload. Use maxUnavailable for stateless (e.g. maxUnavailable: 1 on a 3-replica Deployment means up to 1 can be down during a drain). Use minAvailable for stateful where you need a quorum (e.g. minAvailable: 2 on a 3-pod Postgres). Never set maxUnavailable: 0 — the Platform Operations team overrides it during node drains, so it only masks the misconfiguration until the next maintenance window.
  8. Use CronJob only when it fits the platform's admission rules; otherwise run an in-pod scheduler. Kyverno admission policies reject: schedule more frequent than every 5 minutes; schedule more frequent than every 1 hour if the pod mounts a PVC; any CRON_TZ= prefix in schedule (use spec.timeZone instead). For high-frequency cron with a PVC, convert to a long-running Deployment with go-crond or any in-pod scheduler so you pay the pod-startup cost once. Always set successfulJobsHistoryLimit and failedJobsHistoryLimit (defaults of 3 and 1 are usually fine — set to 0 only if you have external observability for the runs), concurrencyPolicy (Forbid is the safe default for tasks that hold a lock), startingDeadlineSeconds, and a sensible activeDeadlineSeconds on the Job spec to bound stuck runs.
  9. Always ship a HorizontalPodAutoscaler with explicit rampup and rampdown windows. HPA on CPU or memory is the baseline; set minReplicas ≥ 2 for any production workload (one replica + a node drain = an outage); set maxReplicas from your quota (one runaway HPA can exhaust compute-long-running-quota and starve siblings). Always set spec.behavior.scaleUp (rampup) and spec.behavior.scaleDown (rampdown) — never leave them defaulted — so traffic spikes don't get throttled by slow scale-up and so scale-down doesn't thrash mid-rollout (typical baseline: scaleUp.stabilizationWindowSeconds: 30 with a Percent: 100 / periodSeconds: 60 policy; scaleDown.stabilizationWindowSeconds: 300 with a Percent: 50 / periodSeconds: 60 policy — tune from observed traffic, but never ship without them). Pair every HPA with the PDB from step 7.
  10. Always set pod anti-affinity for any multi-replica workload, and add node affinity when the workload must land on (or avoid) a specific node pool / zone. Use podAntiAffinity with topologyKey: kubernetes.io/hostname so replicas don't co-locate on the same node (preferredDuringSchedulingIgnoredDuringExecution for soft spread on small clusters; requiredDuringSchedulingIgnoredDuringExecution for hard spread when you have node headroom). On Gold/GoldDR add topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone so replicas survive a zone loss. Use nodeAffinity (not hard-coded node names — they rotate) when the workload needs GPU, storage-class-local, or other node-pool-specific scheduling.
  11. Set an explicit securityContext on every container so the pod is admitted by the namespace's default restricted-v2 SCC and stays hardened. At the container level set allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], runAsNonRoot: true, readOnlyRootFilesystem: true, and seccompProfile.type: RuntimeDefault. restricted-v2 already enforces most of these, but stating them in the manifest makes intent explicit, survives an SCC change, and is portable. Do not hard-code runAsUser, runAsGroup, fsGroup, or supplementalGroups — OpenShift assigns a UID/GID from the namespace's allocated range, and a hard-coded value outside that range is rejected by restricted-v2 (unable to validate against any security context constraint). When you set readOnlyRootFilesystem: true, mount a writable emptyDir for every path the process writes to (commonly /tmp, plus any cache/run dirs); use emptyDir.medium: Memory with a sizeLimit for small scratch space so it counts against the pod's memory, not node disk.

Rules

  • Always set requests on every container, and pick a deliberate QoS class. Prefer Guaranteed (requests == limits for CPU and memory) for workloads where eviction is costly — databases, leader-elected services, anything stateful or otherwise critical. Burstable (requests set, CPU limit omitted or memory limit ≥ request) is the practical norm for typical BC Gov application workloads. Avoid BestEffort (no requests, no limits) outside opportunistic batch. (Why: a pod without requests is BestEffort, gives the scheduler nothing to place against, and is the first evicted under node pressure. The namespace LimitRange defaults of cpu: 50m / memory: 256Mi are conservative cluster-wide guardrails, not workload-tuned values. A missing memory limit risks OOM-thrashing siblings; omitting a CPU limit is the BC Gov norm so apps can burst into spare capacity, but the pod becomes throttle-prone whenever a node neighbour is busy.)
  • Never run a database — or anything else with persistent state — as a Deployment with a single ReadWriteOnce PVC. (Why: nodes are not guaranteed up for 24 h, and a Deployment+RWO+rolling-update will deadlock the second pod waiting for the volume the first pod hasn't released. Use a StatefulSet with one PVC per replica, or an HA pattern like CrunchyDB / Patroni / Mongo ReplicaSet, three pods minimum.)
  • Always set terminationGracePeriodSeconds deliberately — ≤ 60 s for stateless services, ≤ 600 s for stateful — and ensure the app traps SIGTERM to stop accepting new work, drain in-flight, and exit. (Why: the Platform Operations team caps terminationGracePeriodSeconds at 600 s during node drains and force-kills anything still running; ignoring SIGTERM is data-loss waiting to happen on the next drain.)
  • Never set maxUnavailable: 0 on a PodDisruptionBudget, and never ship a multi-replica long-running workload without one. (Why: the Platform Operations team overrides maxUnavailable: 0 during node drains, so it only masks the misconfiguration until the next maintenance window; a missing PDB lets a node drain take all replicas down simultaneously. Set a real number — e.g. maxUnavailable: 1 on a 3-replica Deployment, or minAvailable: 2 on a 3-pod StatefulSet — so drains are gracefully orchestrated.)
  • Always make readiness and liveness probes cheap and downstream-independent. (Why: a readiness probe that checks a downstream DB will cascade an outage when the DB blips; a liveness probe that fails on transient downstream errors will restart the pod loop and accelerate the failure. Dedicate /livez and /readyz endpoints that only check in-process health.)
  • Always set PID 1 to an init that reaps children (tini, dumb-init, or s6-overlay) unless the application runtime is documented to reap. (Why: defunct <defunct> children accumulate when PID 1 doesn't reap, the platform sends an automated alert to the Product Owner and Technical Lead, and oc delete pod is only a band-aid — the bug returns until PID 1 is fixed.)
  • Never schedule a CronJob more often than every 5 minutes (every 1 hour if it mounts a PVC), and never put CRON_TZ= in the schedule string. (Why: a Kyverno cluster policy denies the CronJob create on all three counts. Use spec.timeZone for the timezone, reduce the cadence, or switch to a long-running Deployment driven by an in-pod scheduler like go-crond.)
  • Always set concurrencyPolicy: Forbid (or Replace) on a CronJob whose work cannot safely overlap. (Why: the default Allow lets a slow run pile up parallel pods on the next tick — for tasks that hold a row lock, mutate shared state, or push to an external system that doesn't dedupe, this is a data-corruption incident, not a performance issue.)
  • Always set minReplicas ≥ 2 for any production workload — Deployment or HPA-driven. (Why: a single-replica workload becomes unavailable the moment its node is drained for patching, which happens routinely. Two replicas + a PDB is the minimum for "stays available across voluntary disruptions".)
  • Always ship a HorizontalPodAutoscaler for any long-running workload, and always set both spec.behavior.scaleUp (rampup) and spec.behavior.scaleDown (rampdown) explicitly — never rely on defaults. (Why: HPA defaults are conservative — scale-up can lag a traffic spike by minutes and scale-down can thrash during a rollout, both of which cause user-visible incidents on this platform. Baseline: scaleUp.stabilizationWindowSeconds: 30 with a Percent: 100 / periodSeconds: 60 policy; scaleDown.stabilizationWindowSeconds: 300 with a Percent: 50 / periodSeconds: 60 policy. Tune from observed traffic, but the keys must be present.)
  • Always set pod anti-affinity on every multi-replica workload, and use nodeAffinity (never hard-coded node names) when scheduling must target a specific node pool or zone. (Why: without podAntiAffinity the scheduler is free to stack all replicas on one node — one node drain or hardware failure then takes the whole workload down, regardless of the PDB. Use topologyKey: kubernetes.io/hostname for per-node spread, and on Gold/GoldDR add topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone so a zone outage doesn't take all replicas. Hard-coded node names rotate and break the manifest the next time the cluster is reconciled.)
  • Always scale horizontally (more replicas) rather than vertically (bigger pods), and pair HPA with a PDB. (Why: large single pods exhaust schedulability — a 32 GiB pod requires a node with 32 GiB of unallocated memory, which becomes rare as the cluster fills; horizontal scaling spreads load and tolerates node loss. Without a PDB, HPA's scale-down can take the workload below the budget during a drain.)
  • Always set an explicit container securityContext with allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], runAsNonRoot: true, readOnlyRootFilesystem: true, and seccompProfile.type: RuntimeDefault. (Why: these are the hardening defaults the platform expects under the restricted-v2 SCC; declaring them in the manifest makes the posture explicit, portable, and resilient to an SCC change instead of relying on cluster-side defaulting. readOnlyRootFilesystem: true also blocks a whole class of tamper-the-binary attacks — pair it with writable emptyDir mounts for the paths the app actually writes to.)
  • Never hard-code runAsUser, runAsGroup, fsGroup, or supplementalGroups in a workload bound for a license-plate namespace. (Why: OpenShift's restricted-v2 SCC assigns a UID/GID from the namespace's pre-allocated range; a hard-coded value outside that range fails admission with unable to validate against any security context constraint, and even a value inside the range will break the next time the range is reassigned. Set runAsNonRoot: true and let the platform pick the UID. Build images so they run as an arbitrary non-root UID — group-writable, group 0 — rather than a fixed one.)
  • Never set privileged: true, add Linux capabilities, or use hostNetwork / hostPID / hostPath on a workload bound for a license-plate namespace. (Why: the restricted-v2 SCC rejects all of them. If you genuinely believe you need one, that is a platform-team conversation via a Public Cloud Service Request — not a manifest change.)
  • Never run VerticalPodAutoscaler in updateMode: "Auto" on a workload that already has an HorizontalPodAutoscaler on the same metric (CPU or memory). (Why: VPA changes requests, HPA reads utilization-vs-requests; the two fight and both can oscillate. Use VPA in updateMode: "Off" for recommendations only when an HPA is in play.)
  • Always set successfulJobsHistoryLimit, failedJobsHistoryLimit, startingDeadlineSeconds, and an activeDeadlineSeconds on the Job template of any CronJob. (Why: default unbounded history fills compute-time-bound-quota with completed pods and blocks new runs; missing activeDeadlineSeconds lets a stuck run consume a worker slot indefinitely.)

Examples

  • "Standard production HTTP API" → Deployment, replicas: 3, container with explicit requests == limits for CPU and memory (Guaranteed QoS), a hardened container securityContext (allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], runAsNonRoot: true, readOnlyRootFilesystem: true, seccompProfile.type: RuntimeDefault) with a writable emptyDir mounted at /tmp, livenessProbe and readinessProbe on dedicated /livez and /readyz endpoints, terminationGracePeriodSeconds: 30, preStop: { exec: { command: ["/bin/sleep", "10"] } }, podAntiAffinity on kubernetes.io/hostname, a mandatory PodDisruptionBudget with maxUnavailable: 1, and a mandatory HorizontalPodAutoscaler with minReplicas: 3 / maxReplicas: 10 on CPU utilization 70% and explicit behavior.scaleUp (stabilizationWindowSeconds: 30, Percent: 100 / periodSeconds: 60) + behavior.scaleDown (stabilizationWindowSeconds: 300, Percent: 50 / periodSeconds: 60).
  • "Pod rejected with unable to validate against any security context constraint" → the manifest is asking for something restricted-v2 won't grant. Most often a hard-coded runAsUser / fsGroup outside the namespace's allocated range, a privileged container, an added capability, or hostNetwork/hostPath. Remove the offending field; set runAsNonRoot: true and let OpenShift assign the UID. Confirm the range your namespace allows from its annotations — oc get project <ns> -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.uid-range}' (reading the cluster SecurityContextConstraints object itself with oc get scc is cluster-scoped and normally Forbidden for project users — you don't need it).
  • "HA Postgres for a production service" → StatefulSet of 3 pods (CrunchyDB Operator or Patroni image from bcgov-docker-local), volumeClaimTemplates for one PVC per pod on netapp-block-standard, PodDisruptionBudget with minAvailable: 2, paired with a backup-container CronJob writing to a netapp-file-backup PVC. Not a Deployment.
  • "Daily 02:00 cleanup job that mounts the PVC the app uses" → CronJob with schedule: "0 2 * * *", spec.timeZone: "America/Vancouver", concurrencyPolicy: Forbid, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1, startingDeadlineSeconds: 300, and activeDeadlineSeconds: 3600 on the Job template. ≥ 1 h cadence satisfies the Kyverno PVC rule.
  • "Process queue messages every 30 seconds" → CronJob is rejected at admission (< 5 min). Convert to a long-running Deployment with replicas: 2 that polls (or subscribes to) the queue in a loop and shuts down cleanly on SIGTERM; size replica count and per-replica concurrency for steady-state throughput. Still ship the mandatory PodDisruptionBudget (maxUnavailable: 1), podAntiAffinity on kubernetes.io/hostname, and a CPU-based HorizontalPodAutoscaler with minReplicas: 2 / maxReplicas: 6 and the standard behavior.scaleUp (stabilizationWindowSeconds: 30, Percent: 100 / periodSeconds: 60) + behavior.scaleDown (stabilizationWindowSeconds: 300, Percent: 50 / periodSeconds: 60) — queue-depth autoscaling is not in scope for this skill, but the CPU-based HPA keeps the mandatory-HPA rule satisfied and protects the workload when poll cost spikes.
  • "Slow-booting Java app keeps getting killed by liveness probe during startup" → add a startupProbe with failureThreshold: 30, periodSeconds: 10 (5-minute boot budget). Liveness probe only kicks in after startup probe passes once, so the boot is no longer a restart loop.
  • "Defunct PIDs alert email" → container ENTRYPOINT is the app process and the runtime doesn't reap. Switch the entrypoint to tini (ENTRYPOINT ["tini", "--", "/app/server"]) or dumb-init; quick band-aid is oc delete pod but the bug returns until PID 1 is fixed.
  • "Pod stuck OOMKilled on every restart" → memory request is below working set. Read Sysdig memory P99 and set requests.memory and limits.memory to that + headroom (~25%). If working set legitimately spikes, profile for leaks (pprof, JVM heap dump, etc.) before throwing more memory at it.
  • "Pod stuck Pending with FailedScheduling insufficient memory" → the requested memory is larger than any node's free allocatable. Either scale horizontally (more, smaller pods) or check if the namespace is sitting on a stuck terminating pod holding the request. Use oc get pods --field-selector=status.phase=Pending to list the pending pods (the API has no Terminating phase — Pending, Running, Succeeded, Failed, and Unknown are the only valid values), and oc get pods -o json | jq '.items[] | select(.metadata.deletionTimestamp) | .metadata.name' to list pods that are mid-terminate (the API signals that with metadata.deletionTimestamp, not a phase).

Edge Cases

  • Job completes but the pod sticks around. That's intentional — Job keeps the terminated pod for log access. Tune with ttlSecondsAfterFinished on the Job spec (e.g. 86400 to garbage-collect after a day). CronJob history limits do this for you.
  • StatefulSet rolling update hangs at one pod. Ordered rollout means the next pod won't start until the current one is Ready. Check the failing pod's events / logs; if it's a probe failure, your readiness probe may be too strict or the new image is genuinely broken — rolling back via oc rollout undo statefulset/<name> is safe.
  • DaemonSet won't schedule on a node. Likely a taint without a matching toleration, or a node selector / affinity rule. oc describe pod <ds-pod> will surface the scheduling failure; add the toleration or relax the selector.
  • HPA reports unknown for a metric. The metric server can't read the source — for Resource metrics (CPU/memory) the pods must have requests set on that resource (no requests means no utilization-vs-requests math).
  • Eviction event with reason TerminationGracePeriodExceeded. Your app didn't exit within terminationGracePeriodSeconds after SIGTERM. Either bump the grace period (to the cap of 600 s) or — more often the right fix — fix the app to handle SIGTERM and drain.
  • A CronJob accidentally set to schedule: "*/1 * * * *" is rejected at admission. Kyverno denies it. Reduce the cadence (≥ 5 min, ≥ 1 h with a PVC), or convert to a long-running scheduler if you genuinely need higher frequency.
  • Need to pin a workload to a specific node pool / zone. Use nodeSelector or affinity.nodeAffinity with the labels published on the cluster's nodes (oc get nodes --show-labels). Avoid hard-coding node names — they rotate. For DR / topology spread on Gold/GoldDR use topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone.
  • Pod stuck CrashLoopBackOff for more than a day. Read the prior run's stdout/stderr with oc logs <pod> --previous, push the fix, and roll out (oc rollout restart deployment/<name>); if you can't fix it immediately, scale the workload to 0 (oc scale deployment/<name> --replicas=0) so it stops consuming a worker slot while you investigate.
  • StatefulSet PVC needs to grow. All BC Gov NetApp storage classes are resize-up-only: edit .spec.volumeClaimTemplates[*].spec.resources.requests.storage, then restart the pods (rolling update doesn't reshape existing PVCs — you may need to delete-and-recreate or use the platform's documented expansion procedure). Resize-down is not supported.
  • readOnlyRootFilesystem: true makes the app crash with read-only file system / EROFS. The process is writing somewhere on the root filesystem. Find the path from the error (commonly /tmp, /var/run, a framework cache, or a PID/lock file) and mount a writable emptyDir there. For small scratch space prefer emptyDir.medium: Memory with a sizeLimit so it's bounded and counts against the pod's memory rather than node disk. Keep readOnlyRootFilesystem: true — adding the writable mount is the fix, turning the flag off is not.
  • App can't bind a privileged port (< 1024) under runAsNonRoot. A non-root UID can't bind :80 / :443. Configure the app to listen on a high port (e.g. :8080) and let the Service map port: 80 → targetPort: 8080. Do not reach for NET_BIND_SERVICE or root — restricted-v2 won't grant either.

References

Scope of this skill: container-level and controller-level workload manifests inside an existing license-plate -dev / -test / -prod namespace. Cluster onboarding, network policy, secrets, GitOps, image-registry, storage class choice, monitoring, and security-assessment concerns live in their own sibling skills (see below).

See references/REFERENCE.md for: the namespace LimitRange and quota objects that shape what your manifests are allowed to request; a worked Deployment / StatefulSet / DaemonSet / CronJob template set; the probe, lifecycle, and preStop recipe; the restricted-v2 SCC securityContext recipe (what to set, what never to hard-code, and the writable-emptyDir pattern for readOnlyRootFilesystem); PDB sizing tables; an HPA and VPA decision matrix; the Kyverno workload-admission policy list (no-fast-cronjob, no-fast-cronjob-with-pvc, no-unsupported-timezone, plus the Emerald-only DataClass label rule that affects pod templates); the defunct-PID + PID-1 init container recipe; and a manifest-level error cheat sheet (OOMKilled, Evicted, CreateContainerConfigError, ImagePullBackOff, TerminationGracePeriodExceeded).

Sibling BC Gov OpenShift skills (each tightly scoped to one concern — planned, not yet shipped in this repo; use upstream docs in the interim):

  • openshift-networking — NetworkPolicy starter kit, default-deny, allow-from-router, SDN-vs-NSX-T egress, Porter TransportServerClaim for direct TCP exposure.
  • openshift-secrets — Vault + External Secrets Operator, SecretStore and ExternalSecret patterns, hyphen-free key gotcha, rotation.
  • openshift-images — Artifactory remote/local repos, prebuilt platform images, ArtifactoryServiceAccount + paired artifacts-* Secrets, npm login --auth-type=legacy.
  • openshift-gitops — Argo CD shared-per-cluster, GitOpsTeam onboarding, bcgov-c/tenant-gitops-<licenseplate> repo layout, sync waves, Helm OCI through Artifactory.
  • openshift-storage — netapp-file-standard / netapp-file-backup / netapp-block-standard, quota tiers, backup-container pattern, OCIO Object Storage Service.
  • openshift-sysdig — Sysdig Monitor onboarding, dashboards, alerts, evidence for quota-increase requests.
  • openshift-acs — Red Hat Advanced Cluster Security policies, image scanning, runtime alerts.

External (use upstream docs, not duplicates here):