|
| 1 | +# Kubestack AGENTS.md file |
| 2 | + |
| 3 | +Kubestack is an OpenTofu/Terraform framework for platform engineering teams building Kubernetes-based platforms. |
| 4 | +It provides re-usable modules to manage clusters, node-pools and platform services from a unified configuration using a GitOps workflow. |
| 5 | + |
| 6 | + * cluster modules provision managed Kubernetes offerings from cloud providers and all their respective infrastructure requirements |
| 7 | + * node-pool modules are used to configure the node pools attached to the respective cluster |
| 8 | + * platform service modules install services that need to exist on clusters before applications can be deployed |
| 9 | + |
| 10 | +Inheritance-based configuration to avoid drift between environments, and keeping infrastructure and application environments separate to avoid blockers between infrastructure and application changes are cornerstones of Kubestack's design. |
| 11 | + |
| 12 | +**This file documents contribution guidelines for the Kubestack framework itself, not for users of the framework.** |
| 13 | + |
| 14 | +## Repository and module layout |
| 15 | + |
| 16 | + * the repository root has directories for `common/` modules shared between all providers, and provider-specific module directories |
| 17 | + * provider-specific module directory names follow the OpenTofu/Terraform provider name for that cloud, e.g. `aws`, `azurerm` or `google` |
| 18 | + * in each provider-specific module directory there is a `cluster/` directory holding the cluster module and its node-pool submodule in the `cluster/node-pool` directory |
| 19 | + * the configuration and metadata module calls, as well as the main resource for the respective module (cluster resource for cluster modules, node-pool resource for node-pool modules) must always be in the file `main.tf` |
| 20 | + * every cluster provisioned by Kubestack must have a default node pool; the default node pool must be provisioned by calling the cluster's node-pool submodule in a `node_pool.tf` file, with the following exceptions: |
| 21 | + * if the cloud provider mandates a built-in default node pool that cannot be disabled (e.g. AKS), use the provider's built-in default node pool instead; `node_pool.tf` is not required in this case |
| 22 | + * if the cloud provider includes a default node pool that can be disabled (e.g. GKE), it must be disabled and the node-pool submodule must still be used in `node_pool.tf` |
| 23 | + * kubeconfig and default ingress related code must be in `kubeconfig.tf` and `ingress.tf` for all cluster modules |
| 24 | + * outputs must be defined in `outputs.tf` |
| 25 | + * moved blocks must be in `moved.tf` |
| 26 | + * variables must be defined in `variables.tf` |
| 27 | + * a data source used by only one resource must be placed directly in front of that resource; if a data source is used by multiple resources or supports conditional logic, it must be placed in a separate `data_sources.tf` file |
| 28 | + * additional files like `vpc.tf`, `roles.tf`, `launch_template.tf` etc. may be created if grouping resources into separate files improves maintainability |
| 29 | + |
| 30 | +### Variables and Outputs |
| 31 | + * all Kubestack modules share a common set of variables and outputs per module type |
| 32 | + * configuration inheritance variables: |
| 33 | + * every module written for the Kubestack framework must define both a `configuration` variable of type `map(object)` and a `configuration_base_key` variable of type `string`; the `configuration_base_key` variable defaults to `apps` so it is optional for users of the module, but both variables must always be defined when writing framework modules |
| 34 | + * the map key of `configuration` is the environment name and the object defines the module and provider-specific configuration attributes |
| 35 | + * all configuration object attributes must be marked optional and must not define a default, because the inheritance implementation relies on unset attributes being null to determine whether to inherit or overwrite an attribute for the current environment |
| 36 | + * inside the module the configuration module must be called, passing in `var.configuration` and `var.configuration_base_key`; the resulting merged config for the current environment (`terraform.workspace`) must be made available as `local.cfg` |
| 37 | + * `local.cfg.<key>` must be used as the input to any resource that requires a value from the module's configuration |
| 38 | + * Kubestack prefers upstream defaults by passing `null` to resource arguments |
| 39 | + * if a default must be set, use the `try(coalesce(<key>, null), <default>)` pattern; this ensures that any nested key error or a null value from coalesce is caught by try, so the default only has to be specified once |
| 40 | + * cluster module outputs: |
| 41 | + * `cluster`: all attributes the cluster resource returns |
| 42 | + * `current_config`: the merged output of the configuration module (`local.cfg`) |
| 43 | + * `current_metadata`: the entire output of the metadata module |
| 44 | + * `kubeconfig`: the yaml encoded kubeconfig for the provisioned cluster; the output is used to configure the provider for platform service modules to be installed on the cluster |
| 45 | + * do not add additional outputs unless there is no alternative; exceptions must not be taken lightly to avoid module user experience drifting apart between providers |
| 46 | + * node-pool module variables and outputs: |
| 47 | + * in addition to the common `configuration` and `configuration_base_key` variables, node pools also require `cluster` and `cluster_metadata` variables that expect the data from the cluster's corresponding `cluster` and `current_metadata` outputs |
| 48 | + * do not add additional variables or outputs unless there is no alternative; the `aws/cluster` and `aws/cluster/node-pool` modules are an example of a case where additional variables were necessary |
| 49 | + |
| 50 | +## Testing instructions |
| 51 | + * testing the Kubestack framework is expensive because real cloud infrastructure has to be created and cleanly deprovisioned as part of the integration tests |
| 52 | + * the multi-cloud platform configuration for the tests can be found in the `tests/` directory |
| 53 | + * a Makefile provides common development targets to run locally and ensures they are run in a container with the correct dependencies and correct working directory |
| 54 | + * never run OpenTofu/Terraform commands manually outside one of the specified make targets |
| 55 | + * `make validate` is safe to run at any time and must be run to validate any changes |
| 56 | + * never run `make test` or `make cleanup` unless explicitly asked to by the user |
| 57 | + * the `common/configuration` and `common/metadata` modules have unit tests; after making any changes to these modules, run `tofu test` inside the respective module directory |
0 commit comments