Epic: WYSIWYG Kubernetes Application Configuration

[bgrant0607]

Most Kubernetes users are interested in configuring applications. That's the primary original purpose of Kubernetes, running containerized workloads. Even cluster services / add-ons are applications. GitOps is primarily focused on deploying applications, as well. Obviously it's the core use case for Helm.

So, what do we need to address in order to be able to handle applications in kpt?

• Create a way to pass non-KRM files through function input/output #3118
• ConfigMap generation #3119
• Develop a way to handle application configuration #3210
• Figure out how to handle secrets #3125
• Figure out how to represent per-environment customizations, for dev and prod #3280
• We need to try set-namespace for this use case
• We'll want a way to capture the Kubernetes deployment context automatically -- I've called this "mini-kubeconfig". It's also similar to the ArgoCD ApplicationSet target generator.
• We need to figure out a reasonable variant-constructor pattern for the case of creating a specific application from a generic blueprint. Cluster services don't have this issue because there's just one per cluster usually.
• Make it easier to write value transformers #3155 (comment): set-image, set-labels, etc. need to be able to specify its source locations so that ApplyReplacements isn't needed to use them.
• More generally, we need a clear model for how to pass inputs to a package, especially deployable packages. Flesh out the input data model and patterns #3396
• We may want something similar to the Flux and ArgoCD image updaters to watch a container image repo and push updates for new images. I wonder if we could use or adapt one of those existing updaters.
• New resource types in the Backstage plugin: Deployment, Service, Ingress, Gateway, GatewayClass, HTTPRoute, PersistentVolumeClaim, StatefulSet, DaemonSet, HorizontalPodAutoscaler. Support for external secrets. Prometheus Operator types. Types relevant to cluster add-ons, such as CustomResourceDefinition, ClusterRole, ClusterRoleBinding, MutatingWebhook, and ValidatingWebhook. Maybe others as we dig into some specific applications. Possibly also Istio resources and/or Argo Rollouts.
• Support for commonly used recommended labels and well known annotations, such as kubectl.kubernetes.io/default-container
• Support for prometheus annotations and OpenTelemetry environment variables
• A function that generates RBAC Roles for the resource types in a package might be useful. Maybe there's something we can use from https://github.com/kubernetes-sigs/kubebuilder-declarative-pattern?
• Linkage between the config UI and a live-state UI, such as the Kubernetes dashboard or, ideally, a Kubernetes backstage plugin
• For cluster services / add-ons specifically, we'll likely want to use an app of apps pattern. In that case, we'd need to look at what we need to do to support RootSync and RepoSync in packages. For example, we'd probably want support for them in the backstage UI plugin. I can imagine needing a function to update pinned commits also.
• Best practices for off-the-shelf blueprints. For example, last-mile customizations that are just general Kubernetes resource attributes and not specific to blueprint components could be omitted from the blueprints. They'd be handled in general authoring logic, such as in the backstage plugin and/or functions.
• An approach for versioning off-the-shelf packages, similar to public helm charts. I insisted on sequential versioning in porch rather than semantic versioning in order to simplify the continuous deployment model. A concept of major version could be used to select the upstream blueprint revision stream. We'd eventually want to support rebase (kpt pkg rebase #2548).
• We will want to provision a namespace for the application prior to first deploying the application. That could be an interesting use case for dynamic dependencies, similar to crossplane provider package dependencies. As opposed to nested subpackages (Reconsider whether we need statically nested subpackages #3343). https://fidelity.github.io/kraan/docs/design/ supports dependencies by "layer". Flux supports package-level dependencies: https://fluxcd.io/docs/components/kustomize/kustomization/#kustomization-dependencies.
• A lifecycle annotation for enabling/disabling deployment of a resource, similar to local-config or the Config Sync manage annotation or a tombstone, but for the purpose of making blueprint resources optionally deployed, similar to the idea of disabling functions in the pipeline
• We may want to support skaffold.dev config as well.
• Document how to approach UX #3145: This is going to increase the surface area of the UI quite a bit. My guess is that some UX work will be required to make it less overwhelming.
  • Lots of resource types, some of which, like pod, have lots of attributes
  • Several resource cross references
  • Multiple components
  • Multiple deployments, such as for dev and prod
• We may want to revisit versioning for off-the-shelf packages, particularly if we're going to build in more versioning functionality, as discussed in Maintain a version field, similar to upstream info #2544.

My current opinion is that multi-cluster specialization and multi-cluster rollout is somewhat independent, but I may change that opinion as we dig into this more.

We plan to look at these common cluster services / add-ons as test cases:

• Cert Manager
• Nginx Ingress Controller
• External DNS
• Monitoring: Prometheus, AlertManager, Grafana, kube-state-metrics
• Logging: ElasticSearch, Fluentd, Kibana

We should also try deploying all our own components: porch server and controllers, config sync, resource group controller, backstage.

At some point, we should also try the ghost application (chart, rendered) we looked at previously. That involved multiple components, so that's another case for dependencies and/or app of apps or static subpackages or dynamic dependencies. It's kind of unusual in that it's an off-the-shelf app rather than a bespoke app or off-the-shelf cluster component or off-the-shelf app platform like knative, kubevela, spark, kubeflow, etc.

Once we figure out how to natively handle applications, we can look into automating helm chart import, rendering and patching helm charts, and so on.

@selfmanagingresource @justinsb @droot

[bgrant0607]

A few scenarios to try:

• Write application configuration from scratch for a specific application (bespoke and off the shelf)
• Publish the configuration for an off-the-shelf application
• Deploy an off-the-shelf application
• Deploy a group of off-the-shelf cluster add-ons
• Customize a bespoke application configuration for multiple environments
• Promote a new application image across environments
• Create a generic application blueprint for an org for a typical kind of application
• Customize a generic application blueprint for an org for a typical kind of application
• Per-cluster customization for a bespoke app in addition to per-environment customization
• Migrate from a helm chart, plain yaml, kustomize, cdk8s

[justinsb]

I agree that this is the direction that our users want us to go in, and I think this is a great list. We've started contributing some dogfooding PRs, which enable us to explore and prioritize the list of items you've identified here.

[bgrant0607]

Not that we're lacking for example applications to try, but here's another:
https://github.com/GoogleCloudPlatform/bank-of-anthos

It uses plain yaml, has a skaffold config, and has separate directories for prod and dev configs.

There's also:
https://github.com/GoogleCloudPlatform/microservices-demo

[bgrant0607]

An example from slack:
https://github.com/treactor/treactor-helm
https://github.com/treactor/treactor-kpt
https://github.com/treactor/treactor-kpt-functions

[bgrant0607]

I took another look at the kubernetes dashboard, portainer, k8syaml.com, k8od.io, lens, monokle, octant, the GKE UI, the Ambassador Labs clickops experience, and Humanitec.

If starting from scratch, starting with required fields makes sense to me. For a blueprint, we can stick with "example" for the names. In most cases other than the container image we could probably provide some defaults, such as labels, selectors, and ports.

If we want lots of defaults even in the blueprint authoring experience, I think the best way to provide those is with an upstream package.

After required fields, we should allow adding and editing of arbitrary fields, but we may want to group them by topic, such as scheduling or security, and sort them by frequency of use.

Dealing with multiple resources together should provide opportunities for autocompletion. For instance, the service selector could be the same as the deployment's selector by default, ports could be defaulted, etc., similar to kubectl run --expose.

Adding a ConfigMap could optionally mount it as a volume or inject the contents as environment variables. We might be able to guess which by looking at its contents. We may also want to be able to upload files and convert them to ConfigMaps using a function (#3119).

Our rule of thumb is that single values could just be edited in place. We'll need to think about how to direct users to those values that may need to be modified (#3145). We should identify cases where values need to be propagated to multiple places, which may suggest we need functions.

[bgrant0607]

I thought about just optimizing for starting blueprints from blueprints, but that would create a chicken and egg scenario of how to create base blueprints. We could potentially provide some, but it still feels unsatisfying. The k8syaml experience does feel like this, though. In combination with the ability to select resources to enable/disable, such as Service, Ingress, and HPA, it could enable selecting from base blueprints for stateless apps, stateful apps (StatefulSet, PVC, and headless Service), and daemons.

[bgrant0607]

The GKE Deploy experience starts with the container image, which it can autocomplete from GCR or AR.

It allows, but doesn't require, specification of the entry point and inline env vars.

Next it supports other configuration, which all has default values: name, namespace, labels. The labels are used for the pod template and selector.

It also creates a HPA, but not Service or Ingress, which I'd add options for. The Services and Ingresses page provides an option to select LoadBalancer Services to create an Ingress for.

From the details of a specific Deployment in the GKE UI, once it has already been created, there is a list of actions: autoscale (creates HPA), scale (edit replicas and resources), expose (create service), and update the image and update strategy knobs.

I'd also add advanced options, to expose the rest of the attributes, rather than just falling back to yaml editing. We possibly could do this via a generic form editor, which we will need to handle arbitrary CRDs: GoogleContainerTools/kpt-backstage-plugins#68. For organizing a large number of options, I like the way the GKE cluster creation page breaks down options into groups of related attributes. k8syaml.com does a flavor of this also.

Here's an Openshift UI example:
https://www.youtube.com/watch?v=jBDmX85IjLM

[bgrant0607]

Some projects have embraced the struct-constructor approach.

gimlet.io builds a UI on top of this chart (and any chart with jsonschema for the values):
https://github.com/gimlet-io/onechart/tree/master/charts/onechart/templates
https://gimlet.io/concepts/onechart-concepts/

The UI can support other charts that provide JSON Schema, though.

kapitan similarly takes a one-generator approach, but uses jsonnet rather than helm templates, and supports multiple components, kind of like helmfile:
https://github.com/kapicorp/kapitan-reference/blob/master/components/generators/kubernetes/README.md
https://github.com/kapicorp/kapitan-reference/tree/master/lib

Effectively these are both thin abstractions over Kubernetes types that enable representation of resources as maps of attributes.

Kapitan supports some kustomize-like features, such as setting common values ("global defaults") across components.

Kapitan normalizes the configuration experience across multiple different tools and generated artifacts, but AFAICT lacks an explicit schema for the inventory. It aims to simplify human authoring, but does not aim to support automation above Kapitan.

https://github.com/zalando-incubator/stackset-controller does support automation on top because it's implemented using CRDs. https://github.com/clastix/capsule takes a similar all-in-one resource approach for Namespace provisioning.

[johnbelamaric]

There are certain common operations that you may want to do on an application deployment but not build into the base blueprint. For example, creating a PodDisruptionBudget or HPA. It would be useful to have functions to generate these and other similar "decorations" on a deployment.

[johnbelamaric]

Just to gather things in one place, a couple other examples for package dependencies:

1. Cloud provider SAs, KSAs, and associated workload identity setup (see Fix the WI docs to include annotating the KSA #3388 (comment))
2. Cluster Proportional Autoscaler (https://github.com/kubernetes-sigs/cluster-proportional-autoscaler)

[johnbelamaric]

There are certain common operations that you may want to do on an application deployment but not build into the base blueprint. For example, creating a PodDisruptionBudget or HPA. It would be useful to have functions to generate these and other similar "decorations" on a deployment.

One issue will be providing guidance on when should we just expect the user to add a resource and not provide any particular help, and when we should use a function to generate the resource, and whether that function should just be run imperatively or should be a declarative function.