feat: Direct lifecycle control of GCP instances#29
Merged
Conversation
These changes remove the use of managed instance groups, and instead directly provisions and manages the lifecycle of GCP instances. This move depends on changes in workload-operator to manage the lifecycle of `Instance` resources, and changes in network-services-operator to manage Subnet and Network resources. Instead of watching for `WorkloadDeployment` resources and creating managed instance groups, the provider will now watch for `Instance` resources and create and manage the underlying GCP instances. If the instance has any scheduling gates, it will not be processed. Most of the logic in the new instance controller is a port of logic in the previous workload deployment controller. While instances are reconciled directly, there are some resources such as service accounts and secrets that are managed at the workload level workload. In these cases, there is an additional controller to handle garbage collection when appropriate. Creation of SubnetClaims has been removed, and responsibility of creating those moved to workload-operator. Subnet provisioning has been moved to a dedicated controller, reacting to `Subnets` that are maintained by network-services-operator. Provisioning of GCP networks has been moved to a new controller that watches for `NetworkContexts`. One or more NetworkContexts may attach to the same Datum Network, which will be provisioned as a single VPC network at GCP. Other key changes: - Moved from leveraging k8s-config-connector to Crossplane for managing resources. The key benefit here is that a single deployment of Crossplane can support authenticating individual resources with different credentials. This significantly reduces per-project infrastructure overhead. The security posture remains the same, with the use of Workload Identity and no service account keys being provisioned for managing GCP resources. - Adoption of the multicluster-runtime library to support deploying an operator which can handle multiple clusters. - Added structured configuration with sane defaults, migrated some flags to config settings. - Added upstream, downstream, and deployment RBAC definitions. - Each new controller is responsible for maintaining a `Programmed` condition on relevant resources to communicate to other parts of the system whether or not the expected changes have been applied. - Provisioning of additional disk backed volumes is currently not supported. Callout: Test coverage for these changes is currently low. During development, a design pattern that we'd like to start leveraging was discovered, which will not only make testing simpler, but also bring greater visibility into the expected behavior of each controller. A followup PR will refactor the controllers to use the new pattern, and bring tests at that time.
joshlreese
force-pushed
the
feature/instance-lifecycle-control
branch
from
13913a7 to
749e563
Compare
joshlreese
changed the title
scotwells
previously approved these changes
Jun 11, 2025
scotwells
left a comment
scotwells
left a comment
Contributor
There was a problem hiding this comment.
Overall code looks good. Nothing major to call out, just some minor questions / clarifications.
…ed bug in default value for TLS.CertDir.
A new `managedResourceLabels` field has been added to the downstream resource management config options that will instruct the operator to add the labels to any downstream resources, and only observe downstream resources that have those labels. Only observing desired resources is accomplished by configuring the client cache with a default label selector based on the config settings. This feature is initially useful for ignoring Crossplane resources which are managed by other systems, but can be leveraged to allow multiple deployments of the operator to be attached to the same control plane, but managed a subset of resources.
scotwells
previously approved these changes
Jun 16, 2025
scotwells
approved these changes
Jun 18, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These changes remove the use of managed instance groups, and instead directly provisions and manages the lifecycle of GCP instances. This move depends on changes in workload-operator to manage the lifecycle of
Instanceresources, and changes in network-services-operator to manage Subnet and Network resources.Instead of watching for
WorkloadDeploymentresources and creating managed instance groups, the provider will now watch forInstanceresources and create and manage the underlying GCP instances. If the instance has any scheduling gates, it will not be processed. Most of the logic in the new instance controller is a port of logic in the previous workload deployment controller. While instances are reconciled directly, there are some resources such as service accounts and secrets that are managed at the workload level workload. In these cases, there is an additional controller to handle garbage collection when appropriate.Creation of SubnetClaims has been removed, and responsibility of creating those moved to workload-operator. Subnet provisioning has been moved to a dedicated controller, reacting to
Subnetsthat are maintained by network-services-operator.Provisioning of GCP networks has been moved to a new controller that watches for
NetworkContexts. One or more NetworkContexts may attach to the same Datum Network, which will be provisioned as a single VPC network at GCP.Other key changes:
Programmedcondition on relevant resources to communicate to other parts of the system whether or not the expected changes have been applied.Callout:
Test coverage for these changes is currently low. During development, a design pattern that we'd like to start leveraging was discovered, which will not only make testing simpler, but also bring greater visibility into the expected behavior of each controller. A followup PR will refactor the controllers to use the new pattern, and bring tests at that time.
Enhancement: datum-cloud/enhancements#28
Related work: