Skip to content

feat: datumctl compute plugin — deploy and manage workloads from the CLI#113

Draft
scotwells wants to merge 97 commits into
mainfrom
feat/datumctl-compute-plugin
Draft

feat: datumctl compute plugin — deploy and manage workloads from the CLI#113
scotwells wants to merge 97 commits into
mainfrom
feat/datumctl-compute-plugin

Conversation

@scotwells

@scotwells scotwells commented May 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the datumctl compute plugin so developers can deploy and manage containerized workloads on Datum Cloud directly from the CLI.

Commands shipped:

  • deploy — push a container image as a workload with flags or a manifest file; waits for rollout
  • destroy — tear down a workload with a confirmation prompt
  • status — show workload health, per-city placement summary, and the active revision
  • instances — list all running instances across cities, with describe for full detail
  • scale — adjust minimum replica count across all placements
  • rollout — watch live rollout progress, browse revision history, and roll back to any prior revision
  • restart — trigger a rolling restart of a workload or a specific city
  • quota — inspect per-city instance usage and surface quota-exceeded messages
  • access — show or request Compute service access for the project (see below)

Revision history is stored as a ConfigMap per workload so rollout history and rollout undo work without server-side tracking.

Service activation & access UX

Compute is enabled per project through a ServiceEntitlement, and enabling it awaits a manual approval on the provider side. The plugin gates every data command on that enablement — and until now it told the user the wrong thing at every step: it created the request, waited a fixed 15 seconds, then printed Error: and "try again in a moment" even though the request had succeeded, and pointed at datumctl services … commands that don't exist.

This branch now derives the whole experience from the entitlement's own status, so the CLI is honest in every state.

Before (verbatim):

$ datumctl compute instances
Compute is not enabled for project "datum-cloud".
Would you like to request access? [y/N]: y
Requesting access to compute for project "datum-cloud"...

Access to compute for project "datum-cloud" has been requested.
Run your command again once it becomes active.

Check status with: datumctl services list
Error: compute access is not yet active — try again in a moment

After:

$ datumctl compute instances
Compute is not enabled for project "datum-cloud".
Requesting access sends an enablement request to the service provider for approval.
Would you like to request access? [y/N]: y
Requesting access to compute for project "datum-cloud"...
⠋ Waiting for the platform to process the request... (4s)

Your request to enable compute for project "datum-cloud" has been submitted.

  Status:  Pending approval — Waiting for the service provider to approve this request.
           Approval is a manual step by the service provider and may take a while.

Check progress with:   datumctl compute access
Wait for activation:   datumctl compute access request --wait

A submitted request no longer reads as an error, pending approval is a legible waiting state, denied/unavailable states carry the platform's own explanation plus a real recovery command, and every printed command exists. New gate-exempt verbs back this up:

  • datumctl compute access [-o json|yaml] — the current access state (with age and the server's message); always exits 0.
  • datumctl compute access request [--message] [--renew] [--wait] [--timeout] — the explicit request verb; safe in scripts, never prompts.

Scripts get a documented exit-code contract (10–13 for not-enabled / denied-or-revoked / pending / unavailable) instead of a blanket exit 1.

The gate logic lives in a shared, service-agnostic serviceactivation package in datumctl (see Dependencies), so the next plugin adopts the flow instead of forking it. Design and rationale: docs/compute/development/rfcs/cli-service-activation.md.

Dependencies

This is a three-repo dependency chain — compute → datumctl → service-catalog — and it must be merged and re-pinned in reverse order (service-catalog first, then datumctl, then compute), replacing each branch pseudo-version with a tagged/merged release before the next merges.

What's not included

  • logs — telemetry service not yet implemented
  • Tests — next step is adding envtest-based integration tests for each command
  • cities / instance-types resource listing commands

Related

Closes #98. Design proposal in #111. Activation UX design in docs/compute/development/rfcs/cli-service-activation.md. Shared SDK: datum-cloud/datumctl#242.

Workloads targeting a city location are now automatically routed to the
correct physical site via a Karmada-based federation layer. Each POP cell
operates independently, instance health is surfaced back to the control
plane in real time, and the platform remains available even when parts of
the control plane are temporarily unreachable.

Controllers added:
- WorkloadDeploymentFederator: replicates WDs into Karmada and manages
  PropagationPolicies per city code
- InstanceProjector: mirrors Instance write-backs from Karmada into the
  project namespace on the control plane

ResourceInterpreterCustomization deployed at config time teaches Karmada
how to aggregate replica counts and conditions across POP cells.

Operator flags --enable-management-controllers and --enable-cell-controllers
allow each deployment to opt into only the controllers it needs.

Includes a 6-test Chainsaw e2e suite covering federation, deletion cascade,
propagation policy lifecycle, instance projection, instance write-back, and
the full end-to-end chain.

Resolves #85

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scotwells and others added 28 commits May 26, 2026 15:04
…edge

Introduces management-plane and cell overlay paths to the compute
OCI artifact so the infra repo can deploy compute-manager in the
correct mode for each tier of the federation architecture.

The management-plane overlay deploys compute-manager with only
WorkloadDeploymentFederator and InstanceProjector enabled, connected
to the Karmada downstream control plane via projected ServiceAccount
token auth. The cell overlay deploys compute-manager with only
WorkloadDeploymentReconciler and InstanceReconciler enabled, with no
downstream connection or webhook server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ts for webhook TLS

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove the hardcoded datum-control-plane ClusterIssuer from the
csi-webhook-cert component. DNS names stay since they are fixed by the
service name and namespace. Each consuming overlay now supplies the issuer
via a strategic merge patch, allowing different environments to use
different cert issuers without forking the component.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each WorkloadDeployment is routed to exactly one cell cluster via its
PropagationPolicy, so aggregation across multiple members is not needed.
Replace the summing logic with a direct pass-through of the single member's
status.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cert issuer name is environment-specific configuration that belongs
in the infra repo, not the compute overlay. The infra repo's base manager
patch already owns the full webhook-server-tls volume definition including
the issuer. Consumers deploying outside infra must patch the issuer in their
own overlay.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…moval

dev: inline self-signed Issuer + Certificate for host.docker.internal,
replace kustomize replacements block with direct annotation patch, remove
Certificate-patching from webhook_patch.yaml, and clear webhookServer
secretRef from config.yaml.

single-cluster: replace cert-manager Certificate approach with the
csi-webhook-cert component, matching the main branch overlay.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The WorkloadReconciler watches networkingv1alpha.Network objects, which
requires the network-services-operator CRDs to be installed. Cell clusters
don't have those CRDs, causing the manager to crash on startup. Gate the
WorkloadReconciler behind enableManagementControllers so it only runs where
the Network CRDs are present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extracts server config file reading and decoding into a dedicated
loadServerConfig helper, reducing main's cyclomatic complexity from
31 to 29 to satisfy the gocyclo linter limit of 30.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Milo's authorization webhook uses Extra claims on the admission request
(iam.miloapis.com/parent-name, iam.miloapis.com/parent-type, etc.) to
resolve the correct project-scoped policy binding. Dropping them caused
the SAR to return Allowed=false even for users with networks.use, because
the authorizer couldn't locate the binding without the project context.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
metricRules belongs under spec.quota, not spec.billing. The field is
not declared in the ServiceBillingConfig schema, causing Flux dry-run
failures in staging with:

  .spec.billing.metricRules: field not declared in schema
Previously, InstanceReconciler wrote ResourceClaim objects against
the local deployment cluster via managementCluster.GetClient(). Those
claims were never seen by the Milo quota system, leaving every Instance
in QuotaGranted=Unknown indefinitely.

This change routes claim creation and deletion to the correct Milo
project control plane for each instance using a new
ProjectQuotaClientManager that builds per-project REST clients by
rewriting the host path — mirroring the URL construction already used
by the milomulticluster provider.

The management-cluster claim watch is replaced with a multicluster
Watches call so that grant/denial status changes in project control
planes re-trigger instance reconciles. Claims are stamped with a
source-cluster label (discovery.clusterName) so each edge controller
only reacts to the claims it created.

Co-Authored-By: Claude <claude@anthropic.com>
The admission webhook requires that all metrics referenced in
spec.quota.limits[].metric and spec.quota.metricRules[].metricCosts
match a name declared in spec.metrics[]. The four quota-tracking
metrics (workloads, instances, vcpus, memory) were missing from
spec.metrics[], causing the webhook to reject the resource.
…o cell setup

Controller flags --enable-management-controllers and --enable-cell-controllers
now default to false so kustomize components must explicitly opt in, rather than
both groups running by default. This prevented the management-plane deployment
from crashing when discovery.clusterName was unset — that field is only required
by the InstanceReconciler (a cell controller), so the validation now lives in
InstanceReconciler.SetupWithManager instead of initializeClusterDiscovery.

Also adds cell-controllers and management-controllers components to the
single-cluster overlay, which was silently running with no controllers enabled.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…scovery

The rebase during cherry-pick propagation introduced a mixed state where
cmd/main.go had the edgeClusterName/projectRestConfig return values partially
reverted. This cleans up the function signature and call sites to be consistent,
while keeping the validation removed from initializeClusterDiscovery (it belongs
in InstanceReconciler.SetupWithManager per the original fix intent).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… RBAC

The workload-deployment-federator calls ensureDownstreamNamespace before
federating WorkloadDeployment resources, but the compute-manager ClusterRole
was missing core-group namespace permissions, causing every reconcile to fail
with a forbidden error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Workload scheduling and admission now consult LocationBinding objects
(project-scoped, created by the service catalog) rather than the global
Location list. This ensures consumers only see locations that are both
healthy and available to their specific project.

Also upgrades network-services-operator and milo dependencies to
versions that introduce LocationBinding and address multicluster-runtime
v0.23 API changes (ClusterName type, ProviderRunnable Start lifecycle,
generic webhook builder).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ources

WorkloadDeploymentReconciler creates and owns NetworkBinding and SubnetClaim
resources, and watches Location, NetworkContext, and Subnet. InstanceReconciler
watches ResourceClaim for quota. Neither was granted the necessary ClusterRole
rules, causing watch failures on cell clusters.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
From the cell cluster's perspective, Karmada is upstream (the federation
control plane), not downstream. Rename the flag, env var, and related
variables throughout to reflect the actual relationship.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…viderRunnable fix

Points go.miloapis.com/milo to the feature branch commit that implements
multicluster.ProviderRunnable on the Milo provider, enabling the mc manager
to auto-call provider.Start() and set p.mcAware so project clusters can be
registered. Without this, p.mcAware was always nil and every project reconcile
logged "Multicluster manager not yet started" forever.

Also removes the & from ResourceRef in ResourceClaimSpec — the feature branch
has ResourceRef as a value type, not a pointer.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove non-existent QuotaRestConfig() call and fix SetupWithManager argument
count; pass nil quota config to skip quota enforcement for now. Single-tenant
cell mode uses namespace-as-project-id and the fixed 'single' cluster name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wires up Milo ResourceClaim-based quota accounting for cells running
in single-cell discovery mode (mode: single), where the multicluster
ClusterName is always "single" rather than the Milo project name.

Key changes:

- Add QuotaKubeconfigPath config field and QuotaRestConfig() method so
  quota REST config can be configured independently of discovery mode.
  Returns (nil, nil) when neither path is set, disabling quota rather
  than silently targeting the local apiserver.

- Add projectIDForInstance and clusterNameForProject func fields to
  InstanceReconciler. In single mode, project ID is derived from
  instance.Namespace; the watch map func always enqueues ClusterName
  "single" rather than the project namespace, avoiding ErrClusterNotFound
  on every quota-grant event.

- Guard ResourceClaim watch map func against claims with empty ResourceRef
  to prevent a nil-dereference panic when a label-matching claim from
  another actor has no ResourceRef set.

- Add TestReconcileQuotaSingleMode covering the full single-mode quota
  flow: project ID from namespace, watch re-enqueue to "single" cluster.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v2.1.5 was built with Go 1.24 and refuses to lint Go 1.25 modules.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tatus change

Write-back was only triggered inside the statusChanged||readyChanged block,
so instances stuck in a scheduling gate (no status transitions) were never
replicated to Karmada.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nged

Use apiequality.Semantic.DeepEqual to avoid unnecessary API calls to Karmada
on every reconcile when nothing has actually changed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
From the cell cluster's perspective, Karmada is upstream.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames the Instance "Running" status condition to "Available" (wire
value "Available") across the API types, controller, and CLI. An
instance can be available while not actively running a pod (e.g. scaled
to zero), so "Running" was a misleading serving/health signal.

API/controller: same constant renames as the backend branch
(InstanceRunning -> InstanceAvailable, InstanceRunningReason* ->
InstanceAvailableReason*, InstanceReadyReasonRunning ->
InstanceReadyReasonAvailable) plus the kubebuilder default marker and
regenerated Instance CRD.

CLI: the derived status now reports availability, never live runtime
state. `Ready=True` displays "Available" (was "Running"), failure
details read "Not available — …" (was "Not running — …"), and the
Available-condition-derived "Starting"/"Stopping" liveness states are
dropped — the CLI no longer indicates whether a process is actively
running at this instant. IsRunning -> IsAvailable.

BREAKING CHANGE: the on-the-wire Instance condition type changes from
"Running" to "Available".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@scotwells

scotwells commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Context: should rollout become a verb group?

Capturing the context so we can make this call deliberately — the decision is open, including doing it before this PR merges. Not asserting a deferral.

Today restart and rollout are flat siblings under compute, and rollout is a status-watcher (≈ kubectl rollout status). As we add more rollout lifecycle operations there's a natural pull toward the kubectl mental model, where rollout is a group of verbs:

compute rollout restart    # what `compute restart` does today
compute rollout status     # what `compute rollout` does today
compute rollout undo       # rollback
compute rollout history    # revisions
compute rollout pause / resume

The tradeoff

  • Moving to the grouped form is a breaking change to the surface (compute restartcompute rollout restart). Cheapest to do before this PR merges / before the plugin has real adoption — every release we wait raises the cost of moving users' muscle memory, scripts, and docs.
  • With only two rollout verbs today (restart + status), the grouping is mostly cosmetic; its clear payoff lands once we ship undo / history (revision tracking) and can design the whole verb set + shared flags (--to-revision, --watch, …) in one coherent pass.
  • So it's a real "now vs later" call: now = pay the design cost early but lock in the clean structure before adoption; later = avoid churn until the verbs justify it, at the cost of a harder migration.

Open design question — where does history come from?
rollout history and undo both depend on revision history of the resource, and we should decide where that lives before committing to these verbs:

  • Compute-specific — compute tracks its own Workload/template revisions (à la Deployment → ReplicaSet), owned and stored by this service; or
  • Generic platform capability — a shared resource-history / revisioning / audit-trail primitive that any resource type plugs into.

If the platform offers (or should offer) generic resource history, compute's rollout {history,undo} should be a thin view over that primitive, not a bespoke revision store — and that also shapes whether these verbs stay under compute or surface more generically across the CLI. This is worth resolving early because it drives both the data model behind rollout and the command surface we'd be locking in.

A signal we're already drifting toward grouping
workloads describe → "Next steps" already advertises datumctl compute rollout undo <wl>, which doesn't exist yet. Our own copy is implicitly assuming the grouped model.

If we do it now, keep restart as a hidden alias of rollout restart for a release or two so we don't break early scripts. If we defer, revisit when we pick up rollout history/undo.

scotwells added 3 commits July 9, 2026 17:55
Replace the bespoke enablement preflight with the shared serviceactivation
SDK. The old flow created a request, waited a fixed 15 seconds, then always
printed "Error: ... try again in a moment" — even though the request had
succeeded — and pointed at datumctl commands that don't exist. The gate now
derives its behavior from the entitlement's status: a submitted request
reads as submitted (not an error), pending approval is a legible waiting
state, and denied/unavailable states carry the platform's own explanation
plus a real recovery command.

Add `datumctl compute access` (status, -o json|yaml) and `datumctl compute
access request [--message] [--renew] [--wait] [--timeout]`, both exempt
from the gate. The plugin entry point maps the SDK's typed errors onto the
documented process exit codes (9-13) so scripts get deterministic results
instead of a blanket exit 1.
The compute plugin consumes the new go.datum.net/datumctl/serviceactivation
package. Pin datumctl to the pseudo-version that carries it so CI resolves
the dependency without a go.work workspace. This pulls datumctl's own
requirements forward, notably go.miloapis.com/milo 0.26 -> 0.29.3, and moves
service-catalog to an indirect dependency (now reached through the SDK).

Temporary: re-pin to a tagged or merged datumctl release before this merges.
The plugin branch had never had a full golangci-lint run in PR CI. Resolve
the findings so the gate is green:

- Extract the repeated instance-status strings ("Available", "Starting",
  "Pending", the "Failed (...)" and "Not available — ..." variants) into
  named constants shared by the list and describe views and their tests
  (goconst).
- Drop two unused controller constants (unused).
- Annotate the two list commands whose branching over flags, filters, and
  output formats is inherently above the cyclomatic-complexity threshold
  (gocyclo).
scotwells and others added 3 commits July 9, 2026 19:33
Bump go.datum.net/datumctl to the pseudo-version whose service-activation SDK
is backed by the generated service-catalog clientset. This pulls
go.miloapis.com/service-catalog forward to its clientset release (now an
indirect dependency, reached through the SDK).

Still temporary: the whole chain (compute -> datumctl -> service-catalog) must
be re-pinned to tagged/merged releases before this merges.
Replace the deprecated archives keys (archives.builds, archives.format,
archives.format_overrides.format) with their current equivalents
(archives.ids, archives.formats, archives.format_overrides.formats) so
plugin releases stop emitting deprecation warnings and keep working when
a future goreleaser major removes the old keys.

Verified with goreleaser 2.16.0: snapshot release produces the same six
platform archives with identical names and zero deprecation warnings.

Fixes #124

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ives-schema

chore: migrate .goreleaser-plugin.yaml to goreleaser v2 archives schema
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define the UX, DX, and AX for deploying and managing compute workloads

1 participant