Skip to content

Commit 7fda928

Browse files
committed
Add AGENTS.md so agents can scaffold code for framework users
* also adds kustomization/overlay module to replace the old custom-manifests that was part of the to be retired catalog allowing platform features to be installed by agents based on clear rules in the AGENTS.md
1 parent b898094 commit 7fda928

27 files changed

Lines changed: 2779 additions & 23 deletions

AGENTS.md

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,23 @@ Before considering any task complete, check each row of the table below and appl
2828
| New required configuration attribute added to a module (guarded by a `lifecycle precondition`) | Add a commented-out example line for it in every affected quickstart file. Add a matching `sed` line to the **"Configure Kubestack"** step in `.github/workflows/main.yml`. See the Quickstart Placeholder and CI Injection Pattern section. |
2929
| New cloud provider added | Add provider-specific CLI build and dist targets to `oci/Dockerfile`, following the pattern of existing providers. Add auth instructions to the shared quickstart `README.md`. |
3030
| New quickstart added | Symlink shared files (`README.md`, `.gitignore`, `.user/`) to `quickstart/src/configurations/_shared/` instead of duplicating them. |
31+
| Any Markdown file edited or created under `quickstart/src/configurations/_shared/` | Apply the one-sentence-per-line rule to every prose paragraph touched. Do not reformat code blocks, tables, or list items. |
32+
33+
## Markdown Style
34+
35+
All Markdown files under `quickstart/src/configurations/_shared/` — currently `README.md` and `AGENTS.md` — MUST follow the one-sentence-per-line rule:
36+
37+
- Each sentence in a prose paragraph occupies its own line.
38+
- A blank line still separates paragraphs.
39+
- The rule does NOT apply to code blocks, tables, blockquotes, or list items — only to prose sentences inside regular paragraphs.
40+
41+
This makes sentence-level changes produce single-line diffs, which makes pull request reviews easier to read.
42+
43+
When editing either file, reformat any prose paragraph you touch to follow this rule, even if the surrounding paragraphs predate the rule and have not yet been updated.
3144

3245
## About Kubestack
3346

34-
Kubestack is an OpenTofu/Terraform framework for platform engineering teams building Kubernetes-based platforms. It provides re-usable modules to manage clusters, node pools, and platform services from a unified configuration using a GitOps workflow.
47+
Kubestack is an OpenTofu/Terraform framework for platform engineering teams building Kubernetes-based platforms. It provides re-usable modules to manage clusters, node pools, and platform features from a unified configuration using a GitOps workflow.
3548

3649
### Core Design Philosophy
3750

@@ -40,7 +53,7 @@ These principles govern all design decisions. Keep them in mind when resolving a
4053
- **Inheritance-based configuration** — all environments derive from a single base configuration, preventing drift.
4154
- **Separated infrastructure and application environments** — infrastructure changes never block application deployments.
4255
- **Shared-nothing architecture** — every environment has its own fully independent set of cluster and node pool resources. Nothing is shared between environments or across clusters.
43-
- **Platform unification** — provider-specific managed clusters become a consistent platform by installing platform services via Kubestack platform service modules, delivering the same capabilities regardless of the underlying cloud provider.
56+
- **Platform unification** — provider-specific managed clusters become a consistent platform by installing platform features via Kubestack platform feature modules, delivering the same capabilities regardless of the underlying cloud provider.
4457
- **GitOps workflow and automation** — proposed changes are previewed with `tofu plan` against any environment via pull request automation, then applied with `tofu apply` across environments in promotion order from least critical to most critical.
4558

4659
### Module Types
@@ -49,7 +62,7 @@ These principles govern all design decisions. Keep them in mind when resolving a
4962
|---|---|
5063
| **Cluster modules** | Provision a managed Kubernetes cluster from a cloud provider along with all required infrastructure (VPC, IAM, etc.). |
5164
| **Node-pool modules** | Configure and manage node pools attached to a cluster. Live as a submodule inside the cluster module directory. |
52-
| **Platform service modules** | Install cluster-level services that must exist before applications can be deployed. |
65+
| **Platform feature modules** | Install cluster-level services that must exist before applications can be deployed. |
5366

5467
## Design Principles
5568

@@ -90,13 +103,13 @@ When a provider does not support a required behaviour (e.g. private nodes), appl
90103

91104
- Provider-specific multi-region or multi-cloud features MUST be disabled.
92105
- Kubestack assumes exactly **one region and cloud provider per cluster**. Multi-region or multi-cloud in Kubestack means creating multiple clusters from the same infrastructure-as-code repository.
93-
- Cluster-level add-ons MUST be disabled by default, unless they are required to support another core design principle such as self-healing or auto-scaling. The preferred approach is to install platform services via Kubestack platform service modules, which allows platform builders to compose a consistent platform across cloud providers.
106+
- Cluster-level add-ons MUST be disabled by default, unless they are required to support another core design principle such as self-healing or auto-scaling. The preferred approach is to install platform features via Kubestack platform feature modules, which allows platform builders to compose a consistent platform across cloud providers.
94107
- Kubestack modules MAY expose opt-in configuration attributes to enable provider add-ons that are disabled by default. This allows users to enable them when needed without making them part of the opinionated baseline.
95108

96109
### Monitoring and Logging
97110

98111
- Cloud provider managed logging and monitoring MUST be enabled by default on every cluster module. This gives platform teams immediate observability into cluster and workload behaviour without any additional configuration.
99-
- Users MAY disable cloud provider logging and monitoring via configuration, using a configuration attribute whose name mirrors the upstream provider resource argument (e.g. `enabled_cluster_log_types` for EKS, `logging_config` / `monitoring_config` for GKE, `enable_log_analytics` for AKS). The expected use case for disabling these is when the platform team intends to deploy a unified, provider-independent observability stack via a Kubestack platform service module instead.
112+
- Users MAY disable cloud provider logging and monitoring via configuration, using a configuration attribute whose name mirrors the upstream provider resource argument (e.g. `enabled_cluster_log_types` for EKS, `logging_config` / `monitoring_config` for GKE, `enable_log_analytics` for AKS). The expected use case for disabling these is when the platform team intends to deploy a unified, provider-independent observability stack via a Kubestack platform feature module instead.
100113
- Where the provider exposes a choice of components to enable (e.g. GKE's `enable_components` list), the Kubestack default MUST include at minimum system-level components (control plane and node agent). Workload-level log collection SHOULD also be enabled by default.
101114
- Kubestack modules MUST NOT deploy additional log forwarding agents, sink configurations, or third-party monitoring agents beyond what is natively available on the cluster resource itself.
102115

@@ -109,7 +122,7 @@ When a provider does not support a required behaviour (e.g. private nodes), appl
109122
- Nodes MUST be configured with private IPs only by default. Cluster egress MUST be routed through NAT gateways (or the provider-equivalent) so that nodes can reach the internet without being directly reachable from it. Users MAY opt out by enabling public IPs on nodes via configuration.
110123
- The cluster control plane API endpoint MUST be publicly accessible by default, so that users can interact with the cluster without requiring VPN or private connectivity. Modules MUST expose a configuration option for users to restrict or fully privatise the API endpoint where the provider supports it.
111124
- Modules MUST NOT implement any cloud provider-specific network connectivity features such as VPC peering or dedicated interconnects.
112-
- Cross-cluster and external connectivity SHOULD be implemented by installing a service mesh via a Kubestack platform service module, preferring a provider-independent solution over provider-proprietary options.
125+
- Cross-cluster and external connectivity SHOULD be implemented by installing a service mesh via a Kubestack platform feature module, preferring a provider-independent solution over provider-proprietary options.
113126

114127
#### CIDR Defaults
115128

@@ -129,8 +142,8 @@ When a provider does not support a required behaviour (e.g. private nodes), appl
129142
#### Human User Authentication
130143

131144
- Kubestack modules do not manage human user kubeconfigs. Human users authenticate by running the cloud provider's CLI to obtain a kubeconfig via IAM. Whether this IAM authentication approach is used only for admin break-glass access or for all platform users is a decision for the platform builder.
132-
- Modules MUST NOT provision any cluster-level RBAC bindings for named human users. User identity and RBAC are the responsibility of platform service modules or the platform operator.
133-
- For platform builders who want to unify end-user authentication across providers behind a single IdP, installing an authentication proxy (e.g. TremoloSecurity/kube-oidc-proxy, vmware/pinniped) as a Kubestack platform service module is the recommended approach. This is optional and at the platform builder's discretion.
145+
- Modules MUST NOT provision any cluster-level RBAC bindings for named human users. User identity and RBAC are the responsibility of platform feature modules or the platform operator.
146+
- For platform builders who want to unify end-user authentication across providers behind a single IdP, installing an authentication proxy (e.g. TremoloSecurity/kube-oidc-proxy, vmware/pinniped) as a Kubestack platform feature module is the recommended approach. This is optional and at the platform builder's discretion.
134147

135148
#### Workload Identity
136149

DEVIATIONS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ When a divergence is resolved, remove its entry from this file.
3939
4040
> **Divergence — `aws/cluster/node-pool`:** Tagging and Labelling — the node-pool `labels` configuration attribute is not applied to the Kubernetes node object's `.metadata.labels`; EKS node group labels are set on the cloud API only, violating the rule that node labels MUST be applied to both the cloud API and the Kubernetes node object. Planned resolution: apply labels to both APIs.
4141
42-
> **Divergence — `aws/cluster`:** Human User Authentication — the `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` configuration attributes allow arbitrary IAM-to-RBAC mappings to be injected into the `aws-auth` ConfigMap via the module, making the module a conduit for cluster-level RBAC bindings for named human users, which AGENTS.md reserves for platform service modules or the platform operator. Planned resolution: investigate if we can remove `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` from the module interface - this depends on whether AWS offers an alternative now; an alternative is preferrable because the in-module kubernetes provider for this configmap is an anti-pattern and it would be overall beneficial to clean this up.
42+
> **Divergence — `aws/cluster`:** Human User Authentication — the `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` configuration attributes allow arbitrary IAM-to-RBAC mappings to be injected into the `aws-auth` ConfigMap via the module, making the module a conduit for cluster-level RBAC bindings for named human users, which AGENTS.md reserves for platform feature modules or the platform operator. Planned resolution: investigate if we can remove `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` from the module interface - this depends on whether AWS offers an alternative now; an alternative is preferrable because the in-module kubernetes provider for this configmap is an anti-pattern and it would be overall beneficial to clean this up.
4343
4444
> **Divergence — `google/cluster`:** Networking — `google_compute_network` is created with `auto_create_subnetworks = true`, which auto-creates unmanaged subnets across all GCP regions rather than the module provisioning its own explicit, Terraform-managed subnet resources. Planned resolution: set `auto_create_subnetworks = false` and add explicit `google_compute_subnetwork` resources in `network.tf`; to do this in a backwards compatbiel way, it may be necessary to make this conditional on a variable so existing clusters can keep using the current subnetworks.
4545

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ If you have any questions while following the tutorial, join the [#kubestack](ht
5757
## Contributing
5858

5959
This repository holds Terraform modules in directories matching the respective provider name, e.g. [`aws`](./aws), [`azurerm`](./azurerm), [`google`](./google). Additionally [`common`](./common) holds the modules that are used for all providers.
60-
Most notably the [`metadata`](./common/metadata) module that ensures a consistent naming scheme and the `cluster_services` module which integrates Kustomize into the Terraform apply.
60+
Most notably the [`metadata`](./common/metadata) module that ensures a consistent naming scheme and the `kustomization/overlay` module which integrates Kustomize into the Terraform apply for platform features.
6161

6262
Each cloud provider specific module directory always has a `cluster` and `_modules` directories.
6363
The cluster module is user facing and once Kubestack is out of beta the goal is to not change the module interface unless the major version changes.
@@ -86,4 +86,4 @@ One super simple way to contribute to the success of this project is to give it
8686
* [kbst/terraform-provider-kustomization](https://github.com/kbst/terraform-provider-kustomization)
8787
* Kustomize Terraform Provider - A Kubestack maintained Terraform provider for Kustomize, available in the [Terraform registry](https://registry.terraform.io/providers/kbst/kustomization/latest).
8888
* [kbst/catalog](https://github.com/kbst/catalog)
89-
* Catalog of Terraform modules for Kubernetes platforms - The [Kubestack Terraform modules](https://www.kubestack.com/catalog/) make it easy for platform engineering teams to deliver common platform features in production ready configurations.
89+
* Catalog of Terraform modules for Kubernetes platforms - The [Kubestack Terraform modules](https://www.kubestack.com/catalog/) make it easy for platform engineering teams to deliver common platform features in production-ready configurations.

kustomization/overlay/.terraform.lock.hcl

Lines changed: 24 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
module "configuration" {
2+
source = "../../common/configuration"
3+
configuration = var.configuration
4+
base_key = var.configuration_base_key
5+
}
6+
7+
locals {
8+
# current workspace config
9+
cfg = lookup(module.configuration.merged, terraform.workspace)
10+
}

0 commit comments

Comments
 (0)