v2.8.0

What's Changed

• Gate CRD installation behind values.installCrds by @pcholakov in #161
• Update Rust toolchain to 1.95.0 by @pcholakov in #162
• Release 2.8.0: Gate CRD installation behind installCrds by @pcholakov in #163

Full Changelog: v2.7.0...v2.8.0

v2.7.0

What's Changed

• chore(deps-dev): bump esbuild from 0.25.12 to 0.28.1 in /examples/services/greeter in the npm_and_yarn group across 1 directory by @dependabot[bot] in #149
• Align generated Pkl bindings; pin and fix the Pkl generator by @pcholakov in #152
• Release 2.7.0: RestateCluster pod lifecycle, sidecars, extra volumes by @pcholakov in #156

Full Changelog: v2.6.1...v2.7.0

v2.6.1

Restate Operator v2.6.1 Release Notes

Highlights

• Fix: in-process Restate Cloud tunnels now accept BYOC regions - RestateCloudEnvironment.spec.region may now contain multiple DNS labels, such as inl4edhpbxasp9yuz1n0yvvkme.byoc, when used with RestateDeployment in tunnelMode: in-process.

Bug Fixes

In-process tunnel mode rejected multi-label BYOC regions

The tunnelMode: in-process validation for RestateCloudEnvironment regions
only allowed a single lowercase DNS label ([a-z0-9-]). This rejected BYOC
Cloud environments whose regions are represented as multiple labels, for
example:

spec:
  region: inl4edhpbxasp9yuz1n0yvvkme.byoc

Those regions are valid for the tunnel hostname
tunnel.<region>.restate.cloud, and the centralized tunnel path already
accepted them. The operator now allows dots in the region while still rejecting
invalid dotted forms with empty labels, such as .us, us., and us..eu.

Impact on Users:

• BYOC Restate Cloud environments using RestateDeployment with
  tunnelMode: in-process can now reconcile and register successfully.
• Standard single-label regions such as us and eu are unchanged.
• Malformed region values are still rejected during reconciliation with an
  InvalidRestateConfig error.

Migration Guidance:
No configuration changes are required. Upgrade the operator if you use
in-process tunnels against BYOC Restate Cloud environments.

Related: PR #147

---

Upgrading

No CRD changes in this release. Upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.6.1

v2.6.0

Restate Operator v2.6.0 Release Notes

Highlights

• Per-version autoscaling for draining RestateDeployment versions — an opt-in spec.autoscaling block lets the operator run a HorizontalPodAutoscaler per non-latest version, so old versions shed compute as their invocations drain instead of holding full replicas for the entire (potentially multi-hour) drain window.
• tunnelMode: in-process for Restate Cloud deployments — pods hold their own outbound tunnel connections (e.g. via @restatedev/restate-sdk-tunnel), serving a Restate Cloud environment with zero inbound networking, while keeping the operator's per-revision registration and draining.
• Fix: HPA metrics on a RestateDeployment are no longer polluted by draining versions — the scale subresource's label selector is now scoped to the latest version's pods, fixing under-provisioning of a busy latest version during drains.
• tolerations and nodeSelector in the Helm chart — schedule the operator pod through chart values instead of forking the chart.

New Features

Per-version autoscaling for draining RestateDeployment versions

A new optional spec.autoscaling field on RestateDeployment (ReplicaSet mode)
lets the operator manage a HorizontalPodAutoscaler per non-latest version.
It is a pass-through HPA .spec (minReplicas, maxReplicas, metrics,
behavior); the operator injects scaleTargetRef per version and owns the HPA,
so it is garbage-collected with the RestateDeployment. A non-latest version
gets an HPA while Restate still reports it active, and the HPA is removed when it
goes inactive — before the operator scales the ReplicaSet to zero.

Previously every old version was held at its full replicas for the entire
drain window — which can last hours for long-running or stuck workflows —
multiplying compute cost across concurrently-draining versions, with no lever to
reduce it (an HPA on the RestateDeployment scale subresource only ever scales
the latest version). Now old versions shed compute as their load falls, while
remaining available for in-flight invocations.

Impact on Users:

• Existing deployments: no impact — the field is optional, and omitting it
  preserves today's behaviour (draining versions stay at full replicas).
• Knative mode: unaffected (Knative's own autoscaler handles this); the field is
  rejected in Knative mode.

Usage:

spec:
  autoscaling:
    minReplicas: 1
    maxReplicas: 10
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 70

Notes:

• The latest version is not covered here — autoscale it with your own HPA
  targeting the RestateDeployment scale subresource (now correct thanks to the
  selector fix below).
• minReplicas is floored at 1 (there is no scale-to-zero in ReplicaSet mode).
  Only Resource/ContainerResource/Pods metrics are per-version (scoped via
  the target's pod selector); an Object/External metric in the shared
  template is read globally, so every draining version would scale on the same
  value.
• CPU/memory metrics require container resource requests; prefer CPU (memory
  does not scale back down).
• RBAC: the bundled Helm chart now grants the operator
  get,list,watch,create,patch,delete on horizontalpodautoscalers (the
  autoscaling API group). If you deploy the operator with your own RBAC,
  add this, or per-version HPAs will silently fail to be created.

Related: Issue #140, PR #143

tunnelMode: in-process for Restate Cloud deployments

RestateDeployment gained spec.restate.tunnelMode (takes effect only with
register.cloud). With tunnelMode: in-process, the deployment's pods hold
their own outbound tunnel connections to Restate Cloud — for example with the
@restatedev/restate-sdk-tunnel npm package — instead of being reached through
the tunnel-client pods and each version's Service. Workloads in private
networks can therefore serve a Restate Cloud environment with zero inbound
networking, while keeping the operator's transparent per-revision registration
and draining.

For such deployments the operator injects RESTATE_INPROC_TUNNEL_NAME (the
versioned name of the revision), RESTATE_INPROC_ENVIRONMENT_ID,
RESTATE_INPROC_CLOUD_REGION and RESTATE_INPROC_SIGNING_PUBLIC_KEY into every
container and init container of the pod template (declaring one of these
yourself is a reconcile error), and registers each version under its tunnel URL
(https://tunnel.<region>.restate.cloud:9080/<env>/<versioned-name>/http/in-process/9080/)
instead of the Service URL. Credentials are never injected —
RESTATE_INPROC_AUTH_TOKEN_FILE is reserved for your own Secret mount. A change
to the referenced RestateCloudEnvironment's environment id, region or signing
key mints a new version, exactly like a pod-template change (these values are
folded into the revision hash only when the mode is set, so existing deployments
keep their hashes). tunnelMode: in-process is rejected in Knative mode.

Impact on Users:

• Existing deployments: no impact. The field is optional, and hashes of
  deployments that don't set it are unchanged, so no ReplicaSets roll on upgrade.

Usage:

spec:
  restate:
    register:
      cloud: my-cloud-environment
    tunnelMode: in-process

Run an in-process tunnel client in your pods and mount an API key Secret (point
RESTATE_INPROC_AUTH_TOKEN_FILE at it). To adopt the mode on an existing
cloud-registered deployment, add tunnelMode: in-process — this creates a new
version (new hash), registered through the tunnel while the old Service-routed
version drains as usual.

Related: PR #144

Tolerations and nodeSelector in the Helm chart

tolerations and nodeSelector are now declared in
charts/restate-operator-helm/values.yaml, so the operator pod can be scheduled
onto tainted or dedicated nodes through chart values instead of forking the
chart or layering a post-render patch. The tolerations template was already
wired; this exposes it via the values file and adds a matching nodeSelector
block.

Impact on Users: existing deployments — no impact (both fields are optional).

Usage:

tolerations:
  - key: "service"
    operator: "Exists"
    effect: "NoSchedule"
nodeSelector:
  workload: controllers

Related: Issue #127

Bug Fixes

RestateDeployment

• Scale-subresource labelSelector scoped to the latest version
  (#139). The
  RestateDeployment scale subresource's .status.labelSelector now appends the
  latest pod-template-hash. Previously it was written verbatim from
  spec.selector with no version filter, so an HPA targeting a
  RestateDeployment averaged metrics across every ReplicaSet the deployment
  had ever owned — the latest plus any old ones still draining pinned
  invocations. Over the (potentially multi-hour) drain window this polluted the
  averaged metric, under-provisioning — or even scaling down — the genuinely-busy
  latest version. No action required; deployments without an HPA are unaffected.

Upgrading

Reapply the CRDs before using the new fields:

kubectl apply --server-side -f crd/restatedeployments.yaml

v2.5.1

Restate Operator v2.5.1 Release Notes

Highlights

• Fix: Knative deployments now work with Restate Cloud — The Knative reconciler was passing raw in-cluster Route URLs to the Restate admin API, which Restate Cloud cannot reach. Service URLs are now routed through the cloud tunnel, matching the existing ReplicaSet behavior.

Bug Fixes

Knative service URLs not tunneled for Restate Cloud

When using RestateDeployment in Knative mode with a Restate Cloud endpoint
(spec.restate.register.cloud), the operator passed the in-cluster Knative
Route URL directly to register_service_with_restate. Restate Cloud cannot
reach in-cluster URLs, so registration failed and the operator looped
indefinitely.

The fix extracts a maybe_tunnel_url() method from RestateAdminEndpoint and
calls it in the Knative reconciler after resolving the Route URL, matching how
the ReplicaSet reconciler already handled this case.

Impact on Users:

• Knative + Restate Cloud deployments: This was broken; it now works.
• Knative + non-Cloud deployments: No change; maybe_tunnel_url is a no-op
  when cloud is not set.
• ReplicaSet deployments: No change. The internal refactoring of
  service_url_for_deployment() is equivalent to the previous behavior.

Related: Issue #120, PR #122

---

Upgrading

No CRD changes in this release. Upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.5.1

v2.5.0

Restate Operator v2.5.0 Release Notes

Highlights

• Custom pod annotations and labels — RestateCluster now supports spec.compute.annotations and spec.compute.labels, enabling integrations that require pod-level metadata (GKE ComputeClass, Vault agent injection, Prometheus scraping, etc.).
• Default canary image changed to alpine:3.21 — Fixes trustedCaCerts (introduced in v2.4.0), which required a CA bundle that the previous busybox:uclibc default did not ship.
• Default tunnel client image bumped to 0.6.0 — Picks up the latest restate-cloud-tunnel-client release for new RestateCloudEnvironment deployments.

New Features

Custom pod annotations and labels

Added spec.compute.annotations and spec.compute.labels to the RestateCluster
CRD. Both fields are propagated to the Restate StatefulSet pod template.

User-specified annotations and labels are merged with any the operator sets
internally (e.g. for Workload Identity, trusted CA certs). In case of conflict,
operator-managed values take precedence.

This unblocks integrations that rely on pod-level metadata, including GKE
ComputeClass scheduling (cloud.google.com/compute-class), Vault agent
injection, Datadog, Prometheus scraping, and custom scheduling constraints.

Impact on Users:

• Existing deployments: No impact, both fields are optional.
• New deployments: Can now set annotations and labels for integrations that
  require them on the pod template.

Usage:

spec:
  compute:
    annotations:
      cloud.google.com/compute-class: "restate-workload"
    labels:
      team: "platform"

Related: Issue #45, PR #119

---

Bug Fixes

Default canary image now ships a CA bundle

The default canaryImage has changed from busybox:uclibc to alpine:3.21.

The trustedCaCerts feature added in v2.4.0 uses an init container (the canary
image) to concatenate system CA certificates with custom trusted CAs. The init
container reads the system CA bundle from /etc/ssl/certs/ca-certificates.crt,
but busybox:uclibc does not ship a CA bundle at that path, causing the init
container to fail with:

cat: can't open '/etc/ssl/certs/ca-certificates.crt': No such file or directory

This made trustedCaCerts non-functional with the default canary image.

Impact on Users:

• Existing deployments using trustedCaCerts: Will work after upgrading. If
  you previously worked around this by setting canaryImage to an image with a
  CA bundle, you can remove that override.
• Existing deployments not using trustedCaCerts: No impact. The canary
  image is also used for Pod Identity and Workload Identity canary jobs, which
  do not depend on the CA bundle and will continue to work with alpine:3.21.
• Custom canaryImage overrides: If you use a custom canary image, ensure
  it includes a CA bundle at /etc/ssl/certs/ca-certificates.crt if you plan to
  use trustedCaCerts.

Migration Guidance:
No action required. The default will change automatically on upgrade.

If you override canaryImage in your Helm values and want to use
trustedCaCerts, ensure your image includes a CA certificate bundle:

# Image must have /etc/ssl/certs/ca-certificates.crt and provide cat, grep, wget
canaryImage: my-registry.example.com/alpine:3.21

Related: PR #116

---

Improvements

Default tunnel client image bumped to 0.6.0

The default tunnelClientDefaultImage has been updated from
ghcr.io/restatedev/restate-cloud-tunnel-client:0.5.0 to 0.6.0. This applies
to new RestateCloudEnvironment deployments that don't explicitly override the
tunnel client image.

Impact on Users:

• Existing deployments pinning their own tunnel client image: no impact.
• Deployments relying on the operator default: will pick up 0.6.0 on the next
  reconcile after upgrade.

Override via the --tunnel-client-default-image CLI flag or the
OPERATOR_TUNNEL_CLIENT_DEFAULT_IMAGE environment variable (settable through
the chart's generic env value) if you need to pin a specific version.

Related: PR #118

---

Upgrading

CRD Update Required: Helm does not upgrade CRDs on helm upgrade. Before upgrading the operator, manually apply the updated CRDs:

kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.5.0/restateclusters.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.5.0/restatedeployments.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.5.0/restatecloudenvironments.yaml

Then upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.5.0

v2.4.0

Restate Operator v2.4.0 Release Notes

Highlights

• Trusted CA certificates - RestateCluster now supports custom trusted CA certificates via spec.security.trustedCaCerts, removing the need for custom Restate images when using internal CAs.
• Configurable canary image - The canary job image is now configurable via Helm, supporting air-gapped and restricted registry environments.
• IPv6 support - The operator now binds to a dual-stack address, fixing readiness probe failures on IPv6-only clusters.
• Faster drain cleanup - Old deployment versions are now polled every 10 seconds during drain, instead of waiting up to 5 minutes.

New Features

Trusted CA certificates

You can now configure custom trusted CA certificates for RestateCluster via
spec.security.trustedCaCerts. This is useful when Restate needs to trust internal CAs, for example when
calling services behind an internal load balancer with a private certificate.

The operator adds an init container that concatenates the system CA bundle with
your custom certificates into a single PEM file, and sets SSL_CERT_FILE on
the Restate container to point to the combined bundle.

Changing the Secret references (name or key) triggers a pod rollout.

spec:
  security:
    trustedCaCerts:
      - secretName: internal-ca
        key: ca.pem

Related: PR #111

---

Configurable canary image

The container image used for PIA and Workload Identity canary jobs is now
configurable via the canaryImage Helm value, CANARY_IMAGE environment
variable, or --canary-image CLI flag. Previously busybox:uclibc was
hardcoded, which fails in environments that cannot pull from Docker Hub.

canaryImage: my-registry.example.com/busybox:uclibc

The simplest approach is to mirror the default image:

docker pull busybox:uclibc
docker tag busybox:uclibc my-registry.example.com/busybox:uclibc
docker push my-registry.example.com/busybox:uclibc

If using a different image, it must provide cat, grep, and wget.

Related: Issue #94, PR #106

---

Bug Fixes

IPv6 dual-stack support

The operator now binds its HTTP server to [::] instead of 0.0.0.0,
supporting both IPv4 and IPv6 clusters. Previously, the readiness probe
failed on IPv6-only clusters because the operator only listened on IPv4.

Related: Issue #93, PR #107

---

Faster drain cleanup polling

When old deployment versions still have active invocations (draining), the
operator now requeues every 10 seconds instead of waiting for the default
5-minute reconcile interval. This means old versions are cleaned up within
seconds of drain completion rather than up to 5 minutes.

Related: PR #112

---

Upgrading

CRD Update Required: Helm does not upgrade CRDs on helm upgrade. Before upgrading the operator, manually apply the updated CRDs:

kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.4.0/restateclusters.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.4.0/restatedeployments.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.4.0/restatecloudenvironments.yaml

Then upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.4.0

v2.3.1

Restate Operator v2.3.1 Release Notes

This contains an important fix for a bug introduced v2.3.0. If you're using v2.3.0 you will need to upgrade to this version.

Highlights

• Fix: GCP Workload Identity now requires explicit opt-in via gcpWorkloadIdentity: true Helm value, fixing a 403 error loop on non-GCP clusters introduced in v2.3.0.

Bug Fixes

IAMPolicyMember cleanup causes 403 on non-GCP clusters

In v2.3.0, the operator unconditionally attempted to delete IAMPolicyMember
resources during reconciliation, even on non-GCP clusters where the RBAC rules
were not granted. This caused a 403 Forbidden error loop on every reconcile.

The operator now requires the gcpWorkloadIdentity Helm value to be explicitly
set before it will create or delete IAMPolicyMember resources. The
iam.gke.io/gcp-service-account annotation is ignored with a warning unless
the flag is enabled.

Impact on Users:

• Non-GCP clusters: The 403 reconcile loop is fixed. No action needed.
• GCP clusters using Workload Identity: You must now set
  gcpWorkloadIdentity: true in your Helm values.

Migration Guidance:

If you are using GCP Workload Identity with Config Connector, add to your Helm
values:

gcpWorkloadIdentity: true

Related: Issue #103, PR #104

---

Upgrading

Upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.3.1

No CRD changes in this release.

v2.3.0

Restate Operator v2.3.0

✨ New Features

• GCP Workload Identity via Config Connector — The operator now automatically creates IAMPolicyMember resources to bind Kubernetes service accounts to GCP service accounts via Workload Identity. This is triggered when a RestateCluster has iam.gke.io/gcp-service-account in serviceAccountAnnotations. The GCP project ID is extracted from the service account email, so no additional configuration is needed beyond the annotation. A canary job validates that credentials are available before the StatefulSet proceeds. This mirrors the existing AWS Pod Identity Association pattern and requires Config Connector to be installed on the GKE cluster.

• Configurable cluster DNS suffix — The operator now supports configuring the Kubernetes cluster DNS suffix via the --cluster-dns CLI flag, CLUSTER_DNS environment variable, or Helm clusterDns value. Previously cluster.local was hardcoded in all internal service URLs. This is needed for multi-cluster setups, federated environments, and clusters with custom DNS naming.

• Configurable drain delay — Added drainDelaySeconds to the RestateDeployment CRD's spec.restate section. This controls how long the operator waits after a deployment is drained before removing the old version. Default remains 300 seconds (5 minutes). (#96)

🐛 Bug Fixes

• Improved admin API error messages — When a deployment registration is rejected by the admin API (e.g. breaking changes without --force), the error message now includes the response from Restate and is logged and emitted as a Kubernetes event, making failures much easier to diagnose. (#100)

• Fixed canary job completion detection — Fixed a bug where a completed canary job was treated as still pending, causing the operator to loop indefinitely with a NotReady status condition. (#102)

⚙️ Configuration Changes

• New Helm value clusterDns for configuring the cluster DNS suffix (default: cluster.local)
• Conditional RBAC for IAMPolicyMember CRDs when GCP Workload Identity is enabled

---

⚠️ Upgrading Notes

CRD Update Required: Helm does not upgrade CRDs on helm upgrade. Before upgrading the operator, manually apply the updated CRDs:

kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.3.0/restateclusters.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.3.0/restatedeployments.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.3.0/restatecloudenvironments.yaml

Then upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.3.0

---

Full release notes: release-notes/v2.3.0.md

v2.2.0

Restate Operator v2.2.0

✨ New Features

• Knative Serving deployment mode — RestateDeployment now supports Knative Serving as an alternative to traditional ReplicaSets. This enables:

  • Scale-to-zero: Services automatically scale down when idle, saving resources
  • Automatic scaling: Replicas scale based on concurrent request load
  • In-place updates: Update service implementation without changing Restate deployment identity
  • Tag-based identity: Control versioning behavior with the tag field — same tag means in-place update, changed tag means versioned update, no tag means auto-versioning

  See the Knative Serving Mode documentation for details. (#64)

🐛 Bug Fixes

• Fix DNS network policy for NodeLocal DNSCache — The operator now creates DNS egress policies that work with both traditional kube-dns and NodeLocal DNSCache (169.254.20.10). This fixes DNS resolution issues on GKE Autopilot and other Kubernetes environments using node-local DNS caching. (#88)

⚙️ Configuration Changes

• Default partitions increased to 24 — The default number of partitions is now 24 (previously lower), providing better parallelism for most workloads. (#84)

📝 Documentation

• Added dedicated Knative Serving mode section to README with examples and tag-based versioning guide
• Added troubleshooting section for DNS resolution issues
• Updated RocksDB memory documentation (#82)

---

⚠️ Upgrading Notes

CRD Update Required: Helm does not upgrade CRDs on helm upgrade. Before upgrading the operator, manually apply the updated CRDs:

kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.2.0/restateclusters.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.2.0/restatedeployments.yaml
kubectl apply --server-side -f https://github.com/restatedev/restate-operator/releases/download/v2.2.0/restatecloudenvironments.yaml

Note: The restatedeployments CRD update is especially important for this release as it includes the new Knative Serving deployment mode fields.

Then upgrade the operator via Helm:

helm upgrade restate-operator restatedev/restate-operator --version 2.2.0

---

New Contributors

• @AhmedSoliman made their first contribution in #82

Full Changelog: v2.1.0...v2.2.0