From 797194251fc23cdde04458d9cde67490445b4a23 Mon Sep 17 00:00:00 2001 From: Philipp Strube Date: Fri, 27 Feb 2026 15:16:14 +0100 Subject: [PATCH] Refine and extend rules in AGENTS.md * shared nothing architecture, unified platforms and gitops design philosophy points * private clusters * clarify add-ons * clarify quickstart purposes and requirements * configuration inheritance and required values implementation * automation and user authentication to kubernetes api * workload identity federation * start tracking deviations in dedicated file --- AGENTS.md | 195 +++++++++++++++++++++++++++++++++----------------- DEVIATIONS.md | 114 +++++++++++++++++++++++++++++ 2 files changed, 244 insertions(+), 65 deletions(-) create mode 100644 DEVIATIONS.md diff --git a/AGENTS.md b/AGENTS.md index 10a64928..07322d58 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,8 +3,6 @@ > **This file governs framework development.** > For framework _usage_, see the AGENTS.md included with each quickstart. ---- - ## Critical Constraints — Read First These rules have no exceptions and MUST be applied before taking any other action: @@ -14,39 +12,60 @@ These rules have no exceptions and MUST be applied before taking any other actio - **ALWAYS** run `make validate` after making any change to module code or test configuration. It is safe to run at any time. - **ALWAYS** update `tests/` when any module interface changes (new variables, removed variables, changed types). - **ALWAYS** update the relevant quickstart examples when any module interface changes. +- **ALWAYS** update `DEVIATIONS.md` when adding a provider-specific exception or documenting a divergence between implementation and rules. See the Tracking Divergences and Exceptions section. + +## Mandatory Update Obligations ---- +Before considering any task complete, check each row of the table below and apply every obligation that matches what you changed. + +| What changed | Obligations | +|---|---| +| Any module interface (variable added, removed, renamed, or type changed) | Update `tests/` to reflect the new interface. Update all affected quickstart examples. Run `make validate`. | +| `common/configuration` or `common/metadata` | Run the module's unit tests (`make validate` does not cover these). See the Unit Tests for Common Modules section. | +| Any module code or `tests/` configuration | Run `make validate`. | +| Any divergence between implementation and rules discovered or resolved | Add or remove the entry in `DEVIATIONS.md`. | +| Any permanent exception to the standard module contract added or removed | Add or remove the entry in `DEVIATIONS.md`. | +| New cloud provider added | Add provider-specific CLI build and dist targets to `oci/Dockerfile`, following the pattern of existing providers. Add auth instructions to the shared quickstart `README.md`. | +| New quickstart added | Symlink shared files (`README.md`, `.gitignore`, `.user/`) to `quickstart/src/configurations/_shared/` instead of duplicating them. | ## About Kubestack Kubestack is an OpenTofu/Terraform framework for platform engineering teams building Kubernetes-based platforms. It provides re-usable modules to manage clusters, node pools, and platform services from a unified configuration using a GitOps workflow. -### Module Types +### Core Design Philosophy -| Type | Purpose | -|---|---| -| **Cluster modules** | Provision managed Kubernetes offerings from cloud providers along with all required infrastructure (VPC, IAM, etc.) | -| **Node-pool modules** | Configure and manage node pools attached to a cluster | -| **Platform service modules** | Install cluster-level services that must exist before applications can be deployed | +These principles govern all design decisions. Keep them in mind when resolving ambiguities not covered by a specific rule: -### Core Design Philosophy +- **Inheritance-based configuration** — all environments derive from a single base configuration, preventing drift. +- **Separated infrastructure and application environments** — infrastructure changes never block application deployments. +- **Shared-nothing architecture** — every environment has its own fully independent set of cluster and node pool resources. Nothing is shared between environments or across clusters. +- **Platform unification** — provider-specific managed clusters become a consistent platform by installing platform services via Kubestack platform service modules, delivering the same capabilities regardless of the underlying cloud provider. +- **GitOps workflow and automation** — proposed changes are previewed with `tofu plan` against any environment via pull request automation, then applied with `tofu apply` across environments in promotion order from least critical to most critical. -- **Inheritance-based configuration** prevents drift between environments by deriving all environments from a single base configuration. -- **Separated infrastructure and application environments** prevent infrastructure changes from blocking application deployments. +### Module Types ---- +| Type | Purpose | +|---|---| +| **Cluster modules** | Provision a managed Kubernetes cluster from a cloud provider along with all required infrastructure (VPC, IAM, etc.). | +| **Node-pool modules** | Configure and manage node pools attached to a cluster. Live as a submodule inside the cluster module directory. | +| **Platform service modules** | Install cluster-level services that must exist before applications can be deployed. | ## Design Principles All Kubestack modules MUST adhere to every principle in this section. +When a provider does not support a required behaviour (e.g. private nodes), apply the most restrictive option the provider does support, document the gap in a `# TODO:` comment directly above the relevant resource argument, and note it in the PR description. + ### Configuration Inheritance -- All environments inherit their configuration from the base environment. The base environment key defaults to `apps` (set via `configuration_base_key`). An environment-specific value always overrides the inherited base value. The inheritance is implemented in the `common/configuration` module, which all Kubestack modules MUST use. +- All environments inherit their configuration from the base environment. The base environment key defaults to `apps` (set via `configuration_base_key`). An environment-specific value always overrides the inherited base value. +- The inheritance logic is implemented in the `common/configuration` module. All Kubestack modules MUST use it. See the Variables and Outputs section for the required usage pattern. +- Because all `configuration` object attributes must be marked `optional(...)` to support inheritance, values that have no module default and must always be user-provided (region, zones, instance type, and `min`/`max` node counts — see Multi-Zone Resilience and Node Pool Lifecycle sections) cannot be enforced by the type system alone. For each such attribute, modules MUST add a `lifecycle` `precondition` block on the primary resource (the cluster resource in a cluster module, the node-pool resource in a node-pool module). The condition MUST check that `local.cfg.` is not `null`. The error message MUST name the attribute and state that it is required, e.g.: `" must be set — Kubestack does not provide a default for this value."`. ### Multi-Zone Resilience -- Node pools MUST be distributed across a minimum of three availability zones by default. The user MUST explicitly specify the zones and MAY specify fewer than three. +- Users MUST explicitly specify a region and availability zones for every cluster and node pool. Modules MUST NOT set any default for region or zones. Node pools MUST be placed in the same region as their cluster. Different node pools within the same environment MAY specify different zones independently. Users MAY specify different zones per environment for a node pool via its configuration inheritance mechanism. +- Node pool module implementations MUST distribute nodes across the specified zones. - Load balancers MUST be distributed across the same zones as the nodes. ### Kubernetes Version Management @@ -56,52 +75,82 @@ All Kubestack modules MUST adhere to every principle in this section. 1. **Explicit**: specify major and minor version (e.g., `1.30`). 2. **Channel**: select a release channel (e.g., `STABLE`). - If both an explicit version and a release channel are specified, the explicit version MUST take precedence. -- Users MAY configure a maintenance window if the provider support it. When a default is required, it SHOULD target 3:00 AM UTC. +- Users MAY configure a maintenance window where the provider supports it. When a Kubestack default is required, it MUST target 3:00 AM UTC. ### Node Pool Lifecycle - **Self-healing** MUST be enabled by default. Users MAY disable it via configuration. +- **Instance type** (the provider-specific attribute for node size, e.g. `instance_type`, `vm_size`, `node_type`) MUST always be specified by the user. Modules MUST NOT set any default. - **Auto-scaling** MUST be enabled by default. Users MAY disable it via configuration. - - `min` and `max` node counts MUST always be user-configurable. The default `min` SHOULD be one per zone and the default `max` SHOULD be two times `min`. + - `min` and `max` node counts MUST always be specified by the user. Modules MUST NOT set any default for these values. - `current` / `desired` / `initial` node count (where the provider exposes it) MUST be owned by the autoscaler and MUST NOT be managed by OpenTofu/Terraform state. Implement this by adding a `lifecycle { ignore_changes }` block on the node-group or node-pool resource, targeting the provider's desired/initial count argument. ### Cloud Provider Isolation - Provider-specific multi-region or multi-cloud features MUST be disabled. -- Kubestack assumes exactly **one cluster per region per cloud provider**. +- Kubestack assumes exactly **one region and cloud provider per cluster**. Multi-region or multi-cloud in Kubestack means creating multiple clusters from the same infrastructure-as-code repository. +- Cluster-level add-ons MUST be disabled by default, unless they are required to support another core design principle such as self-healing or auto-scaling. The preferred approach is to install platform services via Kubestack platform service modules, which allows platform builders to compose a consistent platform across cloud providers. +- Kubestack modules MAY expose opt-in configuration attributes to enable provider add-ons that are disabled by default. This allows users to enable them when needed without making them part of the opinionated baseline. ### Networking -- Every cluster module MUST create its own dedicated network resources (VPC, subnets, route tables, gateways, and any other provider-required constructs). Sharing or accepting externally-created network resources is NOT supported. Place all network resources in a dedicated `network.tf` file inside the cluster module. -- The CNI plugin defaults to the provider's recommended option. Where the provider exposes a CNI choice, it MUST be user-configurable via a `cni` (or equivalent) configuration attribute. +- Every cluster module MUST create its own dedicated network resources (VPC, subnets, route tables, gateways, and any other provider-required constructs). Sharing or accepting externally-created network resources is NOT supported. +- All network resources for a cluster module MUST be placed in a file named `network.tf` inside that cluster module. This applies to all providers, regardless of provider-conventional naming (e.g. use `network.tf`, not `vpc.tf`). +- The CNI plugin MUST default to the provider's recommended option. Where the provider exposes a CNI choice, it MUST be user-configurable via a `cni` (or equivalent) configuration attribute. - Network policies MUST be enabled by default where the provider or CNI supports enforcement at the cluster level. Users MAY disable them where the provider exposes that option. -- The Kubestack-preferred approach for cross-cluster and external connectivity is installing a service mesh via a platform service module. A provider-independent solution is preferred over provider-proprietary options. +- Nodes MUST be configured with private IPs only by default. Cluster egress MUST be routed through NAT gateways (or the provider-equivalent) so that nodes can reach the internet without being directly reachable from it. Users MAY opt out by enabling public IPs on nodes via configuration. +- The cluster control plane API endpoint MUST be publicly accessible by default, so that users can interact with the cluster without requiring VPN or private connectivity. Modules MUST expose a configuration option for users to restrict or fully privatise the API endpoint where the provider supports it. +- Modules MUST NOT implement any cloud provider-specific network connectivity features such as VPC peering or dedicated interconnects. +- Cross-cluster and external connectivity SHOULD be implemented by installing a service mesh via a Kubestack platform service module, preferring a provider-independent solution over provider-proprietary options. #### CIDR Defaults -Kubestack defers to the upstream defaults of each cloud provider. Where a provider accepts CIDR arguments but does not require them, MUST NOT set values — pass `null` so the provider applies its own defaults. Where a provider requires explicit CIDR values to be set, use the defaults from that provider's official documentation. Users MAY override any CIDR via configuration to integrate with existing infrastructure. +- Where a provider accepts CIDR arguments but does not require them, modules MUST NOT set values — pass `null` so the provider applies its own defaults. +- Where a provider requires explicit CIDR values, modules MUST use the defaults from that provider's official documentation. +- Users MAY override any CIDR via configuration to integrate with existing infrastructure. ### Identity and Access -- Support for OpenID Connect (OIDC) tokens for a workload's service account MUST be enabled by default. Users MAY disable it via configuration. +#### Cluster API Authentication (Terraform and CI/CD) + +- The `kubeconfig` output MUST be constructed from the host and credentials returned directly by the cluster resource. These credentials are valid for the duration of the Terraform run and are used by any Terraform provider that configures itself from the kubeconfig output (e.g. the `kbst/kustomization` or `kubernetes` provider). No additional credential exchange or IAM assumption step is performed by the module itself — CI/CD pipelines are expected to handle role assumption or workload identity federation before invoking Terraform. +- Where a provider exposes a control to disable static local accounts or long-lived credential issuance on the cluster resource (e.g. `local_account_disabled` on AKS, `issue_client_certificate = false` on GKE), that control MUST be set to disable static credentials. + +#### Human User Authentication + +- Kubestack modules do not manage human user kubeconfigs. Human users authenticate by running the cloud provider's CLI to obtain a kubeconfig via IAM. Whether this IAM authentication approach is used only for admin break-glass access or for all platform users is a decision for the platform builder. +- Modules MUST NOT provision any cluster-level RBAC bindings for named human users. User identity and RBAC are the responsibility of platform service modules or the platform operator. +- For platform builders who want to unify end-user authentication across providers behind a single IdP, installing an authentication proxy (e.g. TremoloSecurity/kube-oidc-proxy, vmware/pinniped) as a Kubestack platform service module is the recommended approach. This is optional and at the platform builder's discretion. + +#### Workload Identity + +- Workload identity federation (OpenID Connect tokens for workload service accounts) MUST be enabled by default on all providers. Users MAY disable it via configuration. ### Tagging and Labelling - All cloud and Kubernetes resources MUST be tagged/labelled using the output from the `common/metadata` module. -- Users MUST be able to add extra tags/labels for cloud resources and Kubernetes resources **independently** via separate configuration attributes. +- Users MUST be able to add extra labels/tags to all cloud and Kubernetes resources via a configuration attribute whose name mirrors the upstream provider resource argument (e.g. `tags` for AWS and Azure, `resource_labels` for GKE cluster resources). When merging, the following precedence applies, from lowest to highest: + 1. Kubestack metadata labels (from `common/metadata`). + 2. User-supplied labels — user values take precedence over metadata labels on key collision. + 3. Mandatory provider labels (e.g. the EKS `kubernetes.io/cluster/=shared` tag) — always applied last and MUST NOT be suppressible by user configuration. + +#### Node Labels + +Node-pool modules MUST additionally expose a configuration attribute for Kubernetes node labels whose name mirrors the upstream provider resource argument (e.g. `labels` for GKE and EKS, `node_labels` for AKS). Labels specified via this attribute MUST be applied to both the cloud API (the node/instance resource) and the Kubernetes API (the node object's `.metadata.labels`) for every node in the pool. + +- This node-label attribute MUST be passed through from the cluster module's default node pool configuration to the node-pool submodule call in `node_pool.tf`, so that the default node pool is configurable in the same way as any additional node pool. +- The full precedence order on node resources is: metadata labels → user-supplied cloud resource labels → user-supplied node labels → mandatory provider labels. ### Node Taints -- Users MUST be able to specify node taints on both the default node pool and any additional node pools. +- Users MUST be able to specify node taints on both the default node pool and any additional node pools matching the provider specific attribute name and variable type/format. ### Cross-Provider Developer Experience -- Key configuration (zones, min/max node counts, instance types) MUST be available for any module, using provider-specific variable names. +- Region, zones, instance type, and `min`/`max` node counts MUST be exposed as required configuration attributes in every module, using provider-specific attribute names. Modules MUST NOT set defaults for these values. - All configuration attribute names SHOULD mirror the respective provider resource attribute names. - Provider resource blocks containing nested arguments SHOULD be reflected as nested configuration object attributes. ---- - ## Repository and Module Layout ### Directory Structure @@ -116,7 +165,7 @@ Kubestack defers to the upstream defaults of each cloud provider. Where a provid ### Required Files -Every module MUST use the following file layout. Do not consolidate these files. +Every module MUST use the following file layout. Do NOT consolidate these files or add content to a file that does not match its stated purpose. | File | Required In | Purpose | |---|---|---| @@ -125,9 +174,10 @@ Every module MUST use the following file layout. Do not consolidate these files. | `outputs.tf` | All modules | All output definitions. | | `moved.tf` | All modules | All `moved` blocks. No other content. | | `versions.tf` | All modules | Provider requirements and minimum OpenTofu/Terraform version constraint. | -| `kubeconfig.tf` | Cluster modules | Kubeconfig local value. See kubeconfig rules below. | +| `network.tf` | Cluster modules | All network resources (VPC, subnets, route tables, gateways, and any other provider-required network constructs). | +| `kubeconfig.tf` | Cluster modules | Kubeconfig local value. See Kubeconfig Generation section. | | `ingress.tf` | Cluster modules | Default ingress feature resources. | -| `node_pool.tf` | Cluster modules | Default node pool (see decision tree below for exceptions). | +| `node_pool.tf` | Cluster modules | Default node pool call (see Default Node Pool Decision Tree below for exceptions). | | `data_sources.tf` | When required | Data sources shared across resources or containing conditional logic (see rule below). | **Data source placement rule:** @@ -135,7 +185,15 @@ Every module MUST use the following file layout. Do not consolidate these files. - A data source used by **exactly one resource** → place it directly before that resource in the same file. - A data source used by **multiple resources** OR containing **conditional logic** → place it in `data_sources.tf`. -**Additional files** (e.g., `vpc.tf`, `roles.tf`, `launch_template.tf`, `network.tf`) MAY be created when grouping resources into a dedicated file meaningfully improves readability. +**Additional files** (e.g., `roles.tf`, `launch_template.tf`) MAY be created when grouping resources into a dedicated file meaningfully improves readability. + +**`moved` block rule:** + +When a resource is renamed or moved, add a `moved` block to `moved.tf`. MUST NOT run `tofu state mv` (or `terraform state mv`) — Kubestack releases modules, not managed infrastructure, so any state operation would only affect `tests/` and would not be portable to downstream module users who manage their own state. + +If a rename or move cannot be expressed as a declarative `moved` block, it is a breaking change. Past the v1 release, breaking changes MUST be avoided. When a pre-v1 breaking change is unavoidable, document the required manual `tofu state mv` command as an explicit migration step in the release notes. + +`moved.tf` MUST contain only `moved` blocks — no resources, data sources, or locals. ### Default Node Pool — Decision Tree @@ -150,8 +208,6 @@ Every cluster MUST have a default node pool. Determine the implementation approa 3. **All other providers** (e.g., EKS / `aws`): → Provision the default node pool by calling the cluster's node-pool submodule in `node_pool.tf`. ---- - ## Variables and Outputs ### Configuration Inheritance Variables (All Module Types) @@ -166,7 +222,7 @@ variable "configuration" { variable "configuration_base_key" { type = string - default = "apps" # optional for module users; but MUST be defined in all modules + default = "apps" # optional for module users; MUST be defined in all modules nullable = false } ``` @@ -178,11 +234,9 @@ Rules for the `configuration` object type: ### Using Configuration Inside a Module -Inside every module, apply the following pattern: +Inside every module, apply the following pattern exactly: -1. Call the `common/configuration` module in `main.tf`, passing `var.configuration` as `configuration` and `var.configuration_base_key` as `base_key`. -2. Expose the merged result as `local.cfg`, keyed on `terraform.workspace`. -3. Reference `local.cfg.` everywhere a resource needs a configuration value. +**Step 1.** Call the `common/configuration` module in `main.tf`: ```hcl module "configuration" { @@ -190,25 +244,32 @@ module "configuration" { configuration = var.configuration base_key = var.configuration_base_key } +``` +**Step 2.** Expose the merged result as `local.cfg` in the same `main.tf`: + +```hcl locals { cfg = module.configuration.merged[terraform.workspace] } ``` +**Step 3.** Reference `local.cfg.` everywhere a resource needs a configuration value. Never reference `var.configuration` directly in resource arguments. + ### Setting Resource Argument Values Kubestack prefers upstream provider defaults. Apply the following rules when assigning `local.cfg` values to resource arguments: | Situation | Pattern | |---|---| -| No Kubestack-level default is needed | Pass `null` directly: `argument = local.cfg.some_attribute` | +| No Kubestack-level default is needed | `argument = local.cfg.some_attribute` | | A Kubestack-level default is required | `argument = try(coalesce(local.cfg.some_attribute, null), )` | -**Explanation of `try(coalesce(local.cfg., null), )`:** +**How `try(coalesce(local.cfg., null), )` works:** -- `coalesce(local.cfg., null)` returns the configured value if set, or `null` if the attribute was not configured. -- `try(..., )` catches both nested-key errors and the `null` from `coalesce`, falling through to the default. This ensures the default value is specified in exactly one place. +- `coalesce(local.cfg., null)` returns the configured value when set, or `null` when the attribute was not configured. +- `try(..., )` catches both nested-key access errors and the `null` from `coalesce`, falling through to the Kubestack default. +- This ensures the default value is specified in exactly one place — never inside `optional()` in `variables.tf`. **Example** (from `aws/cluster/main.tf`): @@ -261,7 +322,10 @@ locals { } ``` -The authentication field inside `users[].user` is provider-specific: use a `token` for providers that issue short-lived tokens (like EKS, GKE), or `client-certificate-data` / `client-key-data` for providers that issue client certificates (like AKS). +The authentication field inside `users[].user` is provider-specific: + +- Use `token` for providers that issue short-lived tokens (e.g. EKS, GKE). +- Use `client-certificate-data` / `client-key-data` for providers that issue client certificates (e.g. AKS). ### Node-Pool Module Variables and Outputs @@ -278,51 +342,52 @@ Node-pool modules MUST define the following output: |---|---|---| | `current_config` | `local.cfg` | The merged configuration for the active workspace. | -Do NOT add extra variables or outputs without a compelling reason. +Do NOT add extra variables or outputs without a compelling reason. When an exception is necessary, record it in `DEVIATIONS.md` following the instructions in the Tracking Divergences and Exceptions section below. -> **Documented exception:** `aws/cluster/node-pool` defines two additional provider-specific variables — `cluster_default_node_pool_name` and `cluster_default_node_pool_subnet_ids` — because the node-pool module for extra node pools determines subnets from the default node pool's subnet IDs by default. When provisioning the default node pool itself, however, these variables are necessary to break the circular dependency. +### Tracking Divergences and Exceptions ---- +All known gaps between the rules in this file and the current implementation, and all permanent justified exceptions to the standard module contract, are tracked in `DEVIATIONS.md`. Keep that file current — do not leave divergences or exceptions undocumented. -## Testing +**Divergences** are temporary: places where the implementation does not yet comply with a rule, with a planned resolution. Add a divergence entry when you discover or introduce such a gap. Remove the entry once it is resolved. -### Safety Rules +**Exceptions** are permanent: justified deviations from the standard contract that cannot be resolved without breaking something fundamental (e.g. a circular dependency). Add an exception entry when a module must deviate from the standard variable or output contract. Remove the entry only if a refactor makes it unnecessary. -> **NEVER run `make test` or `make cleanup` unless the user explicitly instructs you to.** -> **NEVER run OpenTofu/Terraform commands directly.** Always use the Makefile targets. +For the entry formats and all current entries, see `DEVIATIONS.md`. + +## Testing -These tests provision and destroy real cloud infrastructure across multiple providers and incur real costs. +See Critical Constraints at the top of this file for the safety rules that govern all test-related commands. The rules in this section cover how and when to run tests. ### Validation — Run After Every Change +Run after every change to module code or `tests/` configuration: + ``` make validate ``` -This target is safe to run at any time. It MUST be run to confirm that all changes are syntactically and structurally valid before considering a task complete. +This target is always safe to run. A task is not complete until `make validate` passes. ### Unit Tests for Common Modules -After making any change to `common/configuration` or `common/metadata`, run the module's unit tests: +`make validate` does not cover the `common/` modules. After any change to `common/configuration` or `common/metadata`, run the affected module's unit tests directly: ``` -cd common/configuration && tofu test -cd common/metadata && tofu test +make -C common/configuration test +make -C common/metadata test ``` ### Integration Test Configuration - The multi-cloud test platform configuration lives in `tests/`. -- When any module interface changes (added/removed/renamed variables, changed types), `tests/` MUST be updated to reflect the new interface before running `make validate`. - ---- +- When any module interface changes (added/removed/renamed variables, changed types), update `tests/` to reflect the new interface before running `make validate`. ## Dist Assets -The Kubestack framework is released as three asset types. When making changes, apply the corresponding update obligations: +The Kubestack framework is released as three asset types. Refer to the Mandatory Update Obligations section at the top of this file for a complete list of when each asset must be updated. -| Asset | Description | Update Obligation | -|---|---|---| -| **Versioned modules** | Consumed via `module` blocks using GitHub URLs | Update `tests/` and quickstarts to reflect interface changes. | -| **Container image** | Base image for CI/CD pipelines and manual tasks (`oci/Dockerfile`) | When adding a new cloud provider, add provider-specific CLI build and dist targets to `oci/Dockerfile`, following the pattern of existing providers. | -| **Quickstarts** | Example directory layouts for bootstrapping new repositories | MUST be updated to reflect any module interface or usage changes. Common files (`README.md`, `.gitignore`, `.user/`) are shared via symlinks pointing to `quickstart/src/configurations/_shared/`. When adding a new quickstart, symlink these files instead of duplicating them. When adding a new provider, add its auth instructions to the shared `README.md`. | +| Asset | Description | +|---|---| +| **Versioned modules** | Consumed via `module` blocks using GitHub URLs. | +| **Container image** | Base image for CI/CD pipelines and manual tasks, bundling cloud provider CLIs for IAM authentication and debugging, and Kustomize, OpenTofu/Terraform binaries at the correct version (`oci/Dockerfile`). | +| **Quickstarts** | Example directory layouts for bootstrapping new user repositories. Quickstart examples MUST include `name_prefix`, `base_domain`, region, zones, instance type, and autoscaling `min`/`max` — all values that have no module default and must always be user-provided. Leave all other values absent to rely on module and provider defaults. Common files (`README.md`, `.gitignore`, `.user/`) are shared via symlinks pointing to `quickstart/src/configurations/_shared/`. | diff --git a/DEVIATIONS.md b/DEVIATIONS.md new file mode 100644 index 00000000..88d4fba3 --- /dev/null +++ b/DEVIATIONS.md @@ -0,0 +1,114 @@ +# Kubestack DEVIATIONS.md — Known Divergences and Exceptions + +> This file is referenced by `AGENTS.md`. +> When adding a new divergence or exception, follow the formats defined in each section below. + +## Current Divergences + +Divergences are places where the current module implementation does not yet comply with a rule in AGENTS.md. Every divergence has a planned resolution. Do not work around a divergence silently — if a task requires touching code covered by a divergence, note it in the PR description. + +When a divergence is resolved, remove its entry from this file. + +### Format + +> **Divergence — `` (or "All modules"):** `` — ``. Planned resolution: ``. + +### Entries + +> **Divergence — All cluster and node-pool modules:** Node Pool Lifecycle — instance type and `min`/`max` node counts — existing modules set hardcoded defaults for instance type and `min`/`max` node counts, violating the rule that these must always be user-provided with no module default. Planned resolution: defaults will be removed in the v1 release. + +> **Divergence — `azurerm/cluster`:** Cluster API Authentication — the kubeconfig output uses a client certificate issued via `kube_admin_config` rather than a short-lived Azure AD / Entra ID token, violating the rule that static long-lived credentials must be disabled. Planned resolution: switch to Azure AD / Entra ID token-based authentication in the v1 release. + +> **Divergence — `azurerm/cluster`:** Networking — network resources are placed in `vnet.tf` instead of the required `network.tf`. Planned resolution: rename `vnet.tf` to `network.tf` in the v1 release. + +> **Divergence — `azurerm/cluster`:** Networking — the VNet and subnet are only created when `network_plugin = "azure"`; when the default `"kubenet"` plugin is used no dedicated network resources are provisioned by the module, violating the rule that every cluster module MUST create its own dedicated network resources. Planned resolution: always provision a VNet and subnet in the v1 release. + +> **Divergence — `azurerm/cluster`:** Networking — CIDR Defaults — `service_cidr`, `dns_service_ip`, and `pod_cidr` are given hardcoded Kubestack defaults (`10.0.0.0/16`, `10.0.0.10`, `10.244.0.0/16`) even though AKS accepts but does not require these arguments; the rule requires passing `null` to let the provider apply its own defaults. Planned resolution: remove the hardcoded defaults and pass `null` in the v1 release. + +> **Divergence — `azurerm/cluster`:** Cross-Provider Developer Experience — there is no `region` configuration attribute; the Azure region is derived from `data.azurerm_resource_group.current.location` rather than being exposed as a required configuration attribute with a `precondition`. Planned resolution: add a `region` configuration attribute and precondition in the v1 release. + +> **Divergence — `azurerm/cluster`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `azurerm_kubernetes_cluster` for required attributes (`resource_group`, `availability_zones`, `vm_size`, `min_count`, `max_count`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `azurerm/cluster/node-pool`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `azurerm_kubernetes_cluster_node_pool` for required attributes (`availability_zones`, `vm_size`, `min_count`, `max_count`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `azurerm/cluster`:** Tagging and Labelling — the cluster module exposes `additional_metadata_labels` for user-supplied tags on cloud resources; the upstream `azurerm_kubernetes_cluster` resource argument is `tags`, so the configuration attribute should be named `tags` to mirror it. Planned resolution: rename to `tags` in the v1 release. + +> **Divergence — `azurerm/cluster/node-pool`:** Tagging and Labelling — `azurerm_kubernetes_cluster_node_pool` has no `tags` block applying metadata labels, violating the rule that all cloud resources MUST be tagged using the metadata module output. Planned resolution: add `cluster_metadata` variable and a `tags` argument referencing its labels in the v1 release. + +> **Divergence — `azurerm/cluster/node-pool`:** Node-Pool Module Variables — the node-pool module does not define a `cluster_metadata` input variable, violating the standard node-pool contract which requires both `cluster` and `cluster_metadata` variables. Planned resolution: add `cluster_metadata` variable and use it for resource tagging in the v1 release. + +> **Divergence — `azurerm/cluster`:** Cloud Provider Isolation — `azurerm_log_analytics_workspace` and `azurerm_log_analytics_solution` are provisioned and the OMS agent is enabled by default (`enable_log_analytics` defaults to `true`), violating the rule that cluster-level add-ons MUST be disabled by default unless required for self-healing or auto-scaling. Planned resolution: flip the default to `false` in the v1 release. + +> **Divergence — `azurerm/cluster`:** Cluster Module Outputs — the cluster module exposes `aks_vnet` and `default_ingress_ip` outputs beyond the four required outputs, without compelling provider-specific justification. Planned resolution: evaluate for removal in the v1 release. + +> **Divergence — `aws/cluster`:** Networking — network resources are placed in `vpc.tf` instead of the required `network.tf`. Planned resolution: rename `vpc.tf` to `network.tf` in the v1 release. + +> **Divergence — `aws/cluster/node-pool`:** Networking — network resources are placed in `vpc.tf` instead of the required `network.tf`. Planned resolution: rename `vpc.tf` to `network.tf` in the v1 release. + +> **Divergence — `aws/cluster`:** Networking — nodes are assigned public IPs by default (`map_public_ip_on_launch = true` is the default when `cluster_vpc_subnet_map_public_ip` is unset), violating the rule that nodes MUST be configured with private IPs only by default and egress MUST route through NAT gateways. Planned resolution: flip the default to private nodes with NAT gateway egress in the v1 release. + +> **Divergence — `aws/cluster`:** Cross-Provider Developer Experience — there is no `region` configuration attribute; the AWS region is sourced entirely from the provider configuration rather than being exposed as a required configuration attribute with a `precondition`. Planned resolution: add a `region` configuration attribute and precondition in the v1 release. + +> **Divergence — `aws/cluster`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `aws_eks_cluster` for required attributes (`cluster_availability_zones`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `aws/cluster/node-pool`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `aws_eks_node_group` for required attributes (`availability_zones`, `instance_types`, `min_size`, `max_size`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `aws/cluster`:** Tagging and Labelling — user-supplied cloud resource tags are only configurable via `additional_node_tags` nested inside `default_node_pool`, meaning there is no top-level `tags` attribute on the cluster configuration object and extra-node-pool resources (VPC, subnets, security groups, etc.) cannot receive user-supplied tags. Planned resolution: add a top-level `tags` attribute to the cluster configuration in the v1 release. + +> **Divergence — `aws/cluster/node-pool`:** Tagging and Labelling — the node-pool `labels` configuration attribute is not applied to the Kubernetes node object's `.metadata.labels`; EKS node group labels are set on the cloud API only, violating the rule that node labels MUST be applied to both the cloud API and the Kubernetes node object. Planned resolution: apply labels to both APIs in the v1 release. + +> **Divergence — `aws/cluster`:** Human User Authentication — the `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` configuration attributes allow arbitrary IAM-to-RBAC mappings to be injected into the `aws-auth` ConfigMap via the module, making the module a conduit for cluster-level RBAC bindings for named human users, which AGENTS.md reserves for platform service modules or the platform operator. Planned resolution: remove `cluster_aws_auth_map_roles` and `cluster_aws_auth_map_users` from the module interface in the v1 release. + +> **Divergence — `google/cluster`:** Networking — `google_compute_network` is created with `auto_create_subnetworks = true`, which auto-creates unmanaged subnets across all GCP regions rather than the module provisioning its own explicit, Terraform-managed subnet resources. Planned resolution: set `auto_create_subnetworks = false` and add explicit `google_compute_subnetwork` resources in `network.tf` in the v1 release. + +> **Divergence — `google/cluster`:** Tagging and Labelling — `google_container_cluster` has no `resource_labels` block, and `google_compute_network`, `google_compute_router`, `google_compute_address`, and `google_dns_managed_zone` resources in `network.tf` and `ingress.tf` have no labels from the `common/metadata` output, violating the rule that all cloud resources MUST be tagged/labelled using the metadata module output. Planned resolution: add `resource_labels` (or equivalent) referencing `module.cluster_metadata.labels` to all cloud resources in the v1 release. + +> **Divergence — `google/cluster/node-pool`:** Tagging and Labelling — the merge order in `node_config.labels` is `merge(cfg.labels, cluster_metadata.labels)`, placing metadata labels last so they override user-supplied labels, which is the inverse of the required precedence (metadata labels → user-supplied cloud resource labels → user-supplied node labels → mandatory provider labels). Planned resolution: fix the merge order in the v1 release. + +> **Divergence — `google/cluster`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `google_container_cluster` for required attributes (`region`, `project_id`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `google/cluster/node-pool`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `google_container_node_pool` for required attributes (`location`, `machine_type`, `min_node_count`, `max_node_count`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `google/cluster/node-pool`:** Node Pool Lifecycle — `google_container_node_pool` sets `initial_node_count` but has no `lifecycle { ignore_changes = [initial_node_count] }` block, violating the rule that the desired/initial node count must be owned by the autoscaler and not managed by OpenTofu/Terraform state. Planned resolution: add the `lifecycle` ignore block in the v1 release. + +> **Divergence — `google/cluster/node-pool`:** Node Pool Lifecycle — `machine_type` defaults to `""` (empty string) via `try(coalesce(local.cfg.machine_type, null), "")`, which the GCP provider interprets as a provider-level default, violating the rule that instance type must always be user-provided with no module default. Planned resolution: remove the empty-string default in the v1 release (covered by the existing "All cluster and node-pool modules" divergence but noted here for the specific empty-string mechanism). + +> **Divergence — `google/cluster`:** Cluster Module Outputs — the cluster module exposes a `default_ingress_ip` output beyond the four required outputs, without a compelling provider-specific justification. Planned resolution: evaluate for removal in the v1 release. + +> **Divergence — `google/cluster/node-pool`:** Node-Pool Module Outputs — the node-pool module exposes an `id` output beyond the single required `current_config` output, without a compelling provider-specific justification. Planned resolution: evaluate for removal in the v1 release. + +> **Divergence — `google/cluster`:** Required File Layout — `provider.tf` contains the data source `data.google_client_config.default`, which is used by both `kubeconfig.tf` and the kubernetes provider alias block in the same file; per the data source placement rule, a data source used by multiple resources belongs in `data_sources.tf`, not co-located with a provider block. Planned resolution: move the data source to `data_sources.tf` in the v1 release. + +> **Divergence — `scaleway/cluster` and `scaleway/cluster/node-pool`:** Required File Layout — neither module contains a `moved.tf` file, violating the rule that every module MUST have a `moved.tf`. Planned resolution: add empty `moved.tf` files in the v1 release. + +> **Divergence — `scaleway/cluster`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `scaleway_k8s_cluster` for required attributes (`region`, `cluster_version`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `scaleway/cluster/node-pool`:** Configuration Inheritance — `precondition` lifecycle blocks are absent from `scaleway_k8s_pool` for required attributes (`zones`, `node_type`, `min_size`, `max_size`), violating the rule that required attributes with no module default must be guarded by a precondition on the primary resource. Planned resolution: add preconditions in the v1 release. + +> **Divergence — `scaleway/cluster`:** Networking — nodes are assigned public IPs by default (`public_ip_disabled` defaults to `false` in `node_pool.tf`), violating the rule that nodes MUST be configured with private IPs only by default. Planned resolution: flip the default to `true` in the v1 release. + +> **Divergence — `scaleway/cluster/node-pool`:** Tagging and Labelling — the node-pool module exposes no configuration attribute for Kubernetes node labels; node labels can only be approximated by encoding them as Scaleway tag strings, with no guarantee that they are applied to the Kubernetes node object's `.metadata.labels`, violating the rule that node labels MUST be applied to both the cloud API and the Kubernetes node object. Planned resolution: add a dedicated node label attribute once the Scaleway provider exposes first-class Kubernetes node label support. + +> **Divergence — `scaleway/cluster`:** Cluster API Authentication — `scaleway_k8s_cluster.current.kubeconfig[0].token` is a long-lived static admin token issued by the cluster resource; Scaleway does not expose a control to disable static credential issuance on the cluster resource, so this cannot currently be resolved, but it diverges from the rule that static long-lived credentials must be disabled where the provider exposes such a control. Planned resolution: revisit when the Scaleway provider exposes a control to disable static token issuance. + +## Provider-Specific Exceptions + +Exceptions are permanent, justified departures from the standard module contract. Unlike divergences, exceptions are not planned to be resolved — they exist because the standard rule cannot be applied to a specific module without breaking something fundamental (e.g. a circular dependency). + +When adding an exception, update the `AGENTS.md` Critical Constraints to remind future agents that this file must be kept current. + +When an exception becomes unnecessary (e.g. a refactor removes the underlying constraint), remove its entry from this file. + +### Format + +> **Exception — ``:** `` — ``. + +### Entries + +> **Exception — `aws/cluster/node-pool`:** `cluster_default_node_pool_name` and `cluster_default_node_pool_subnet_ids` — the node-pool module for extra node pools determines subnets from the default node pool's subnet IDs by default; when provisioning the default node pool itself these variables are required to break the circular dependency. + +> **Exception — `azurerm/cluster`:** `provider.tf` — the AKS cluster module requires a `kubernetes` provider alias configured with static client certificate credentials from `kube_config`; because provider blocks cannot be placed in any of the standard required files, a dedicated `provider.tf` is the only viable location for this alias configuration. + +> **Exception — `google/cluster`:** `provider.tf` — the GKE cluster module requires a `kubernetes` provider alias configured with a short-lived GCP access token obtained from `data.google_client_config.default`; because provider blocks cannot be placed in any of the standard required files, a dedicated `provider.tf` is the only viable location for this alias configuration. + +> **Exception — `google/cluster`:** `cluster_role_binding.tf` — on GKE, a freshly-created cluster grants no Kubernetes RBAC permissions to any GCP IAM identity by default; the `kubernetes_cluster_role_binding` that grants `cluster-admin` to the currently-authenticated GCP service account (the CI/CD automation identity) is required to bootstrap RBAC so that the `kubernetes` and `kbst/kustomization` providers can operate against the cluster during the same Terraform run, directly supporting the Cluster API Authentication requirement.