|
2 | 2 | <a href=https://prudnitskiy.pro/en/ title=Blog>Blog</a></nav></header><main id=main tabindex=-1><article class="post h-entry"><div class=post-header><header><h1 class="p-name post-title">Cleaning up terraform repos with layers</h1><span><ul class=post-translations><li>EN</li><li><a href=/ru/post/2026-05-26-terraform-layers/>RU</a></li></ul></span></header><div class="post-info noselect"><div class="post-date dt-published"><time datetime=2026-05-26>2026-05-26</time></div><a class="post-hidden-url u-url" href=/en/post/2026-05-26-terraform-layers/>/en/post/2026-05-26-terraform-layers/</a> |
3 | 3 | <a href=https://prudnitskiy.pro/ class="p-name p-author post-hidden-author h-card" rel=me>map[name:Paul Rudnitskiy]</a><div class=post-taxonomies><ul class=post-tags><li><a href=/tags/clouds>#clouds</a></li><li><a href=/tags/terraform>#terraform</a></li></ul></div></div></div><details class="toc noselect"><summary>Table of Contents</summary><div class=inner><nav id=TableOfContents><ul><li><a href=#the-problem>The problem</a><ul><li><a href=#terraform-state>Terraform state</a></li><li><a href=#blast-radius>Blast radius</a></li><li><a href=#bonus-track-security>Bonus-track: security</a></li></ul></li><li><a href=#solution-layers>Solution: layers</a><ul><li><a href=#in-practice>In practice</a></li></ul></li><li><a href=#outcome>Outcome</a></li></ul></nav></div></details><script>var toc=document.querySelector(".toc");toc&&toc.addEventListener("click",function(){event.target.tagName!=="A"&&(event.preventDefault(),this.open?(this.open=!1,this.classList.remove("expanded")):(this.open=!0,this.classList.add("expanded")))})</script><div class="content e-content"><p>Terraform is the most popular infrastructure-as-code framework. I wasn’t able to find a good source of information on Terraform’s market share. The closest source is a report from 6Sense that says Terraform has 37% of the market. This market share doesn’t include OpenTofu, which is a fork of Terraform. It also compares Terraform to Ansible (30%), which is a bit of a weird comparison. Anyway, Terraform has a majority of the IaC market. In most of the cloud projects you are currently working on, there will be Terraform one way or another.</p><h2 id=the-problem><div><a href=#the-problem># |
4 | 4 | </a>The problem</div></h2><p>Terraform is not the most convenient tool, even if it is the most popular. One of the advantages is Terraform utilizes providers. This approach allows you to extend Terraform easily by adding new types of resources. You are still using HCL and relatively the same code to manage resources for different providers. It also allows you to transfer data from one kind of resource to another one. For example, you have an application in a Kubernetes cluster that utilizes OAuth2. To use OAuth2, you need to create an application in your OAuth2 provider (such as Okta or Azure EntraID). Then you need to pass the client ID, tenant ID, and client secret (or PKCE URL) to your application. You need to create a secret in Kubernetes so application can use those credentials to start utilizing the OAuth client. Terraform allows you to do it because it operates one large state. All variables in the code are globally visible, so you can use it for your convenience.</p><h3 id=terraform-state><div><a href=#terraform-state>## |
5 | | -</a>Terraform state</div></h3><p>Everything Terraform works with it stores in the state file (tfstate). It is an enormous JSON document that Terraform unmarshals to the Go structure in memory on start. The first problem of the state is its size. The size of the state is roughly equal to the size of the object multiplied by the amount of objects your Terraform operates. The size of the exact object is defined by the provider and, through it, by the API that contains all possible properties of the object. For example, Google is very careful with the API design; the resources only contain properties absolutely required. On the other hand, Microsoft or Oracle are not that meticulous in the API. Just have a look on (<a href=https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/core_instance.html%29[oci_core_instance] target=_blank rel="noopener noreferrer">https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/core_instance.html)[oci_core_instance]</a> to see how many unclear and non-obvious the setup can be. All the properties of the object must be in the state. That causes state inflation, the more objects you have, the bigger the state will be. Large state consumes memory, because you cannot read the part of the state. You have to read the whole file to unmarshall it and place it in the memory as it is. I’ve seen infrastructures that required 24Gbs of RAM just to run <code>terraform plan</code>. Otherwise, terraform job fails due to out-of-memory error. The bonus problem hides in the terraform reconciliation. Terraform refreshes objects in state as a part of run cycle. The more objects you have the longer it takes. Some public APIs have rate limits and other security measures to avoid DoS attacks. Some of the public APIs may be simply slow (infamous Azure Entra ID, I’m talking to you). Sometimes you have to wait more than 10 minutes to have everything in order.</p><h3 id=blast-radius><div><a href=#blast-radius>## |
| 5 | +</a>Terraform state</div></h3><p>Everything Terraform works with it stores in the state file (tfstate). It is an enormous JSON document that Terraform unmarshals to the Go structure in memory on start. The first problem of the state is its size. The size of the state is roughly equal to the size of the object multiplied by the amount of objects your Terraform operates. The size of the exact object is defined by the provider and, through it, by the API that contains all possible properties of the object. For example, Google is very careful with the API design; the resources only contain properties absolutely required. On the other hand, Microsoft or Oracle are not that meticulous in the API. Just have a look on <a href=https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/core_instance.html target=_blank rel="noopener noreferrer">oci_core_instance</a> to see how many unclear and non-obvious the setup can be. All the properties of the object must be in the state. That causes state inflation, the more objects you have, the bigger the state will be. Large state consumes memory, because you cannot read the part of the state. You have to read the whole file to unmarshall it and place it in the memory as it is. I’ve seen infrastructures that required 24Gbs of RAM just to run <code>terraform plan</code>. Otherwise, terraform job fails due to out-of-memory error. The bonus problem hides in the terraform reconciliation. Terraform refreshes objects in state as a part of run cycle. The more objects you have the longer it takes. Some public APIs have rate limits and other security measures to avoid DoS attacks. Some of the public APIs may be simply slow (infamous Azure Entra ID, I’m talking to you). Sometimes you have to wait more than 10 minutes to have everything in order.</p><h3 id=blast-radius><div><a href=#blast-radius>## |
6 | 6 | </a>Blast radius</div></h3><p>Terraform initially designed to be used by individuals, not teams. The HCL (HashiCorp Language used by Terraform to describe state you want to achieve) doesn’t have internal protection mechanisms common to normal languages. It doesn’t have proper encapsulation, all variables are global and may be re-defined or changed anywhere in the system. A small change may affect unpredictable places, causing significant infrastructure damage. Of course there is a <code>terraform plan</code> shows you the changes terraform going to apply. However, it is easy to miss a critical change. Especially in the case of many object changes same time or if you use AI code review. People tend to trust other people’s judgement and not paying enough attention to the tool output. And the result may be catastrophic.</p><p>Terraform doesn’t perform changes by itself, it uses providers. Each provider has the configuration and performs the authentication. Of course, you must grant the provider with the permissions necessary to perform the tasks you use terraform for. By default, you have a provider per kind of resource (for example, to configure Azure or Google Cloud). Any mistake done with this provider is done with elevated credentials of the provider. It is possible to implement least privilege approach by creating multiple providers and specify the provider for a specific group of resources. However, it is not really useful – people tend to use the simplest approach possible and in the end everything will be done by one privileged provider. You can’t remove default provider. You can’t prohibit something to highly-qualified people (and your colleagues, I’m sure, are highly qualified). Smart people will find their way one way or another.</p><h3 id=bonus-track-security><div><a href=#bonus-track-security>## |
7 | 7 | </a>Bonus-track: security</div></h3><p>Terraform was not designed to implement security. The state file contains any object terraform operates. All this data is not encrypted. Any time you generate encryption keys, pass API credentials, certificates or any kind of secret information using dependency or <code>data</code> you save this information in state. This information is not encrypted. Any person with state access can pull any data from state by using <code>terraform state pull</code> or just copying the file. What is worse there is no way to limit access to a specific area of the state. The operator (the terraform) requires the whole state to work with because there is no way to read the state piece by piece.</p><h2 id=solution-layers><div><a href=#solution-layers># |
8 | 8 | </a>Solution: layers</div></h2><p>Firstly, there is no perfect, 100% working solution fits everyone and works everywhere. The layered approach is the one potential option I see interesting. I saw it in different teams invented independently so I think it is a good option to consider. The general idea is to split code in layers. Each layer represents a logic group of resources provisioning some state of the environment. The state of the layer is mostly independent and only explicit dependencies passed via <code>data</code> or <code>remote_state</code> terraform primitive. Those layers integrated vertically. The data passed from “top” layers to “bottom” but never in opposite or lateral direction. This setup creates a natural resource hierarchy. For example, we have resources in Google Kubernetes cluster deployed in Google Cloud project. This setup may be represented in three layers:</p><ul><li>Layer 1 defines the project itself, its properties and billing policies.</li><li>Layer 2 represents cluster deployment, node pools and other in-project resources</li><li>Layer 3 represents resources inside of GKE cloud</li></ul><p>Every layer has its own specific responsibility. Any layer can use a specific set of permissions narrowed to the requirements of this specific layer. Any information passed between layers explicitly. We are using <code>output</code> on the upper level as an outbound and <code>remote_state</code> as an inbound for the information we pass. For multiple groups on the same level, you can use multiple folders (or separate repos) if needed. It helps with the isolation of concern and through this reduces blast radius as well as a current state size. This approach requires some mind shift, but it works quite well for big repositories with complex resource graph. This approach works best for teams because each part of the infrastructure you manage becomes mostly independent. A very important note is to avoid lateral data transfer. Any data passes only top to bottom. Without it, you can easily get a dependency loop: service A depends on service B while service B depends on service A. Terraform tracks resource graph in monorepos but for layered approach it is not possible because you are not directly passing resources between layers, only their values. This risk increases with the amount of services you manage. It will be impossible to avoid such problem in large infrastructures with thousands and thousands of resources. Layered approach gives a set of advantages:</p><ul><li>Each layer is compact and small. It is simple to observe and fix. Compact code produces a compact plan, and any potential problem in the plan looks much more visible.</li><li>Data access management is much more… Manageable as you pass data explicitly and know what and where do you pass.</li><li>Reduced resource usage and faster reconciliation thanks to the smaller code and state.</li><li>The layered design approach naturally improves code discipline. It requires careful design and coding. It reduces sloppiness with ad-hoc solutions full of <code>#@TODO:</code> and <code>#@FIXMIE:</code> as well as required to think of code as of module with API as a contract. It makes code more reliable.</li><li>Layered approach reduces blast radius. For the worst case possible the damage isolated to the current layer and its descendants keeping parallel and upper layers unaffected.</li><li>As any layer has its own area of responsibility, you can design explicit permissions minimized to a specific layer’s needs. This approach reduces blast radius and makes security team happy.</li><li>In fact layered approach encapsulates code in layers, creating a nearly accurate OOP with terraform, like a grown up language do. It helps to change the code in a specific layer without affecting other layers. You only need to keep the output data format compatible to previous revision. The output data format becomes public contract. It helps with the big teams with many people works with the same infrastructure. You can also update the version seamlessly by creating multiple versions of the output and change the consumer gradually. It moves infrastructure-as-a-code code to nearly enterprise coding patterns.</li></ul><h3 id=in-practice><div><a href=#in-practice>## |
9 | | -</a>In practice</div></h3><p>One picture works better than a thousand words. Let’s try it in practice. We have a simple setup built of 3 layers:</p><ul><li>Layer 0 sets project in the cloud</li><li>Layer 1 creates Kubernetes cluster in this cloud. It deploys VMs, builds LoadBalancers and disk and so on. This layer also manages OAuth credentials in the authentication provider.</li><li>Layer 2 manages resources inside of Kubernetes cluster deployed in Kubernetes</li></ul><p>Honestly, it is not the best approach as Layer-1 has many roles in the same time. However, it is an example from my personal lab, and I’m the only person manages this setup so we can consider it manageable. The main reason we use layers is to simplify our life. Layers design is not a dogma, and you need to find a balance between amount of layers and complexity of data you pass from layer to another. As layers applied one-by-one, many layers will slow down the pipeline even if each layer works faster.</p><p>I use monorepo in this example, with all code placed in the same place. However, it is up to you to split it to multiple repositories. Here’s an example repository structure:</p><pre tabindex=0><code>. |
| 9 | +</a>In practice</div></h3><p>One picture works better than a thousand words. Let’s try it in practice. We have a simple setup built of 3 layers:</p><ul><li>Layer 0 sets project in the cloud</li><li>Layer 1 creates Kubernetes cluster in this cloud. It deploys VMs, builds LoadBalancers and disk and so on. This layer also manages OAuth credentials in the authentication provider.</li><li>Layer 2 manages resources inside of Kubernetes cluster deployed in Kubernetes. As our applications uses OAuth2 to authenticate users with SSO we need to have keyapirs from previous layer on this layer.</li></ul><p>Honestly, it is not the best approach as Layer-1 has many roles in the same time. However, it is an example from my personal lab, and I’m the only person manages this setup so we can consider it manageable. The main reason we use layers is to simplify our life. Layers design is not a dogma, and you need to find a balance between amount of layers and complexity of data you pass from layer to another. As layers applied one-by-one, many layers will slow down the pipeline even if each layer works faster.</p><p>I use monorepo in this example, with all code placed in the same place. However, it is up to you to split it to multiple repositories. Here’s an example repository structure:</p><pre tabindex=0><code>. |
10 | 10 | └── layer-0-project |
11 | 11 | └── layer-1-cloud |
12 | 12 | ├── layer-2-front |
|
0 commit comments