From 8858d226a6b1408ce5486cf760e1231f58d1f2c4 Mon Sep 17 00:00:00 2001 From: Anton Tayanovskyy Date: Tue, 29 Apr 2025 11:50:19 -0400 Subject: [PATCH 1/4] WIP --- .../announcing-direct-tf-modules/index.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 content/blog/announcing-direct-tf-modules/index.md diff --git a/content/blog/announcing-direct-tf-modules/index.md b/content/blog/announcing-direct-tf-modules/index.md new file mode 100644 index 000000000000..3d10e55e5520 --- /dev/null +++ b/content/blog/announcing-direct-tf-modules/index.md @@ -0,0 +1,143 @@ +--- +title: "Introducing Pulumi Support for Executing Terraform Modules" +allow_long_title: true +date: 2024-06-05T00:00:00-04:00 +draft: false +meta_desc: "Pulumi can now execute Terraform modules directly" +meta_image: "meta.png" +authors: + - anton-tayanovskyy +tags: + - terraform + - features +--- + +We are excited to announce that Pulumi can now execute [Terraform modules +directly](https://github.com/pulumi/pulumi-terraform-module). This new capability unlocks a great option for users +contemplating migrating a large Terraform installation to Pulumi: when dealing with a complicated Terraform module, you +can now bypass translating its sources while still quickly moving its state over to Pulumi and cross-linking its inputs +and outputs with Pulumi code. Additionally, all Pulumi users can now more easily benefit from the existing awesome +modules in the Terraform registry. + + + +As organizations optimize their infrastructure management, many teams are exploring transitioning from Terraform to +Pulumi. However, migrating existing, complex Terraform configurations to Pulumi has often been cited as challenging, +especially when intricate Terraform modules are involved. Our customers tell us that Pulumi's [pulumi convert --from +terraform](https://www.pulumi.com/blog/converting-full-terraform-programs-to-pulumi/) is very useful for small to +medium programs, but runs into challenges on more complex projects, especially involving modules. Even when sources +convert successfully, full migration still requires meticulous validation to ensure production infrastructure +continuity is not affected by small differences in Pulumi and Terraform behavior. + +To address this feedback, Pulumi is excited to announce new support for executing Terraform modules directly within +Pulumi. This new feature enables teams to continue utilizing their existing Terraform modules without source +modifications. Modules execute under their exact Terraform semantics powered by [OpenTofu](https://opentofu.org). Their +inputs and outputs are exposed in a type-safe manner to your favorite Pulumi programming language, to be freely +composed with other Pulumi components. Finally, Terraform state is automatically [stored in +Pulumi](https://www.pulumi.com/docs/iac/concepts/state-and-backends/) and takes full advantage of proper [secret +encryption](https://www.pulumi.com/docs/iac/concepts/secrets/). + +Our hope is that this approach becomes an important part of the migration toolbox to be applied to selectively to those +parts of the codebase that have the highest migration risk and/or source complexity. + +## Walkthrough + +To get started, run the following command in an existing Pulumi directory, linking a module as a package and giving it +a friendly name "vpcmod": + + +``` +pulumi package add terraform-module terraform-aws-modules/vpc/aws 5.18.1 vpcmod +``` + +While the example uses a well-known module from the registry, but a local path will also work to point to an in-house +or locally maintained module. + +// What will this do. + +// Make a simpler example. + +// Cool if we could configure AWS Provider? + +```typescript + +import * as vpcmod from '@pulumi/vpcmod'; +import * as std from '@pulumi/std'; + +const prefix = cfg.get("prefix") ?? pulumi.getStack(); + +const vpc = new vpcmod.Module("test-vpc", { + azs: azs, + name: `test-vpc-${prefix}`, + cidr, + private_subnets: azs.apply(azs => azs.map((_, i) => + std.cidrsubnetOutput({input: cidr, newbits: 8, netnum: i+1}).result + return getCidrSubnet(cidr, 8, i+1); + })), + public_subnets: azs.apply(azs => azs.map((_, i) => { + return getCidrSubnet(cidr, 8, i+1+4); + })), + intra_subnets: azs.apply(azs => azs.map((_, i) => { + return getCidrSubnet(cidr, 8, i+1 + 8); + })), + + + enable_nat_gateway: true, + single_nat_gateway: true, + + public_subnet_tags: { + 'kubernetes.io/role/elb': '1', + }, + private_subnet_tags: { + 'kubernetes.io/role/internal-elb': '1', + }, +}); + +// export private subnets vpc.private_subnets.apply(subnets => subnets![0]), +``` + +// Show pulumi preview and pulumi up here. + +// Note that pulumi preview and up explain what is changing inside the module. + + +## Supported Features + +All the regular operations are supported: + +- Pulumi executes preview/update/refresh as expected +- The entire resource tree being constructed is made visible +- Updates detail which concrete module-managed resources are changing +- Pulumi generates typed input and output accessors to interop with the module +- Pulumi handles first-class secrets and unknown values seamlessly +- Pulumi supports configuring the Terraform providers powering the module execution + +## Limitations + +As of today some Pulumi features will not work as expected with resources managed by a directly executed Terraform +module: + +- using the [transformations](https://www.pulumi.com/docs/iac/concepts/options/transformations/) resource option +- using targeted operations such as `pulumi up --target` +- the [protect](https://www.pulumi.com/docs/iac/concepts/options/protect/) resource option + +Pulumi's capability to infer accurate types for module inptus and outputs is also currently limited and will sometimes +infer sub-optimal or even unusable types. See [README](https://github.com/pulumi/pulumi-terraform-module) for +configuration options. + +## What's next + +As part of hardening this feature Pulumi will be looking at removing the limitations and improving error handling and +usability of the directly executed modules. Our team is currently excited about these possibilities: + +- extending `pulumi convert --from terraform` to put the users in control of which modules are converted recursively to + Pulumi source code, and which are instead converted into directly executed modules + +- enhancing state import from Terraform into Pulumi to seamlessly work with directly executed modules + +## Get Started + +Support for modules is available as of today. Download the latest Pulumi CLI and it a try. If you run into any issues +or have suggestions and feedback, please [let us +know](https://github.com/pulumi/pulumi-terraform-module/issues/new/choose) or reach out in the [Pulumi Community +Slack](https://slack.pulumi.com/). From 3429facdc9960cd29da68e326d04c5f265fc03f0 Mon Sep 17 00:00:00 2001 From: Anton Tayanovskyy Date: Wed, 14 May 2025 16:50:56 -0400 Subject: [PATCH 2/4] Edits with more worked code examples --- .../announcing-direct-tf-modules/index.md | 179 ++++++++++++++---- 1 file changed, 141 insertions(+), 38 deletions(-) diff --git a/content/blog/announcing-direct-tf-modules/index.md b/content/blog/announcing-direct-tf-modules/index.md index 3d10e55e5520..b67171f23f4f 100644 --- a/content/blog/announcing-direct-tf-modules/index.md +++ b/content/blog/announcing-direct-tf-modules/index.md @@ -45,64 +45,167 @@ parts of the codebase that have the highest migration risk and/or source complex To get started, run the following command in an existing Pulumi directory, linking a module as a package and giving it a friendly name "vpcmod": +``` +$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 5.18.1 vpcmod +Successfully generated a Nodejs SDK for the vpcmod package at /Users/anton/tmp/2025-05-14/blog/sdks/vpcmod +``` + +Notice that Pulumi generated a local SDK for the module: ``` -pulumi package add terraform-module terraform-aws-modules/vpc/aws 5.18.1 vpcmod +$ ls sdks/vpcmod +README.md index.ts node_modules provider.ts tsconfig.json utilities.ts +bin module.ts package.json scripts types ``` -While the example uses a well-known module from the registry, but a local path will also work to point to an in-house -or locally maintained module. +And linked it into your project in `package.json`: -// What will this do. -// Make a simpler example. +``` + "dependencies": { + ..., + "@pulumi/vpcmod": "file:sdks/vpcmod" + } +``` -// Cool if we could configure AWS Provider? +You can now reference the module from your code with full code completion support. For example: ```typescript - +import * as pulumi from "@pulumi/pulumi"; import * as vpcmod from '@pulumi/vpcmod'; -import * as std from '@pulumi/std'; - -const prefix = cfg.get("prefix") ?? pulumi.getStack(); const vpc = new vpcmod.Module("test-vpc", { - azs: azs, - name: `test-vpc-${prefix}`, - cidr, - private_subnets: azs.apply(azs => azs.map((_, i) => - std.cidrsubnetOutput({input: cidr, newbits: 8, netnum: i+1}).result - return getCidrSubnet(cidr, 8, i+1); - })), - public_subnets: azs.apply(azs => azs.map((_, i) => { - return getCidrSubnet(cidr, 8, i+1+4); - })), - intra_subnets: azs.apply(azs => azs.map((_, i) => { - return getCidrSubnet(cidr, 8, i+1 + 8); - })), - - - enable_nat_gateway: true, - single_nat_gateway: true, - - public_subnet_tags: { - 'kubernetes.io/role/elb': '1', - }, - private_subnet_tags: { - 'kubernetes.io/role/internal-elb': '1', - }, + azs: ["us-west-2a", "us-west-2b"], + name: `test-vpc-${pulumi.getStack()}`, + cidr: "10.0.0.0/16", + public_subnets: [ + "10.0.1.0/24", + "10.0.2.0/24", + ], + private_subnets: [ + "10.0.3.0/24", + "10.0.4.0/24", + ], + enable_nat_gateway: true, + single_nat_gateway: true, }); -// export private subnets vpc.private_subnets.apply(subnets => subnets![0]), +export const publicSubnets = vpc.public_subnets; +export const privateSubnets = vpc.private_subnets; +``` + +If you have AWS credentials set up, you can now do `pulumi up` and it will show all the resources being created: + ``` -// Show pulumi preview and pulumi up here. +Previewing update (dev) + +View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/anton-blog/dev/previews/5851682a-a3e3-4f47-a50e-3d7b4f4e6c34 + + Type Name Plan + + pulumi:pulumi:Stack anton-blog-dev create + + └─ vpcmod:index:Module test-vpc create + + ├─ vpcmod:index:ModuleState test-vpc-state create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[1] create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[1] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[0] create + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.private[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[1] create + + ├─ vpcmod:tf:aws_nat_gateway module.test-vpc.aws_nat_gateway.this[0] create + + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.public_internet_gateway[0] create + + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.private_nat_gateway[0] create + + ├─ vpcmod:tf:aws_default_network_acl module.test-vpc.aws_default_network_acl.this[0] create + + ├─ vpcmod:tf:aws_default_route_table module.test-vpc.aws_default_route_table.default[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[0] create + + ├─ vpcmod:tf:aws_internet_gateway module.test-vpc.aws_internet_gateway.this[0] create + + ├─ vpcmod:tf:aws_vpc module.test-vpc.aws_vpc.this[0] create + + ├─ vpcmod:tf:aws_default_security_group module.test-vpc.aws_default_security_group.this[0] create + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.public[0] create + + ├─ vpcmod:tf:aws_eip module.test-vpc.aws_eip.nat[0] create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[0] create + + └─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[1] create + +Resources: + + 22 to create + +Previewing update (dev) + +View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/anton-blog/dev/previews/faab8c4d-3a86-42f6-9685-ef36ece68652 + + Type Name Plan + + pulumi:pulumi:Stack anton-blog-dev create + + └─ vpcmod:index:Module test-vpc create + + ├─ vpcmod:index:ModuleState test-vpc-state create + + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.private_nat_gateway[0] create + + ├─ vpcmod:tf:aws_default_route_table module.test-vpc.aws_default_route_table.default[0] create + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.public[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[1] create + + ├─ vpcmod:tf:aws_default_network_acl module.test-vpc.aws_default_network_acl.this[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[0] create + + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.public_internet_gateway[0] create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[0] create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[0] create + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.private[0] create + + ├─ vpcmod:tf:aws_eip module.test-vpc.aws_eip.nat[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[1] create + + ├─ vpcmod:tf:aws_internet_gateway module.test-vpc.aws_internet_gateway.this[0] create + + ├─ vpcmod:tf:aws_default_security_group module.test-vpc.aws_default_security_group.this[0] create + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[1] create + + ├─ vpcmod:tf:aws_vpc module.test-vpc.aws_vpc.this[0] create + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[0] create + + ├─ vpcmod:tf:aws_nat_gateway module.test-vpc.aws_nat_gateway.this[0] create + + └─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[1] create + +Resources: + + 22 to create + +Updating (dev) + +View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/anton-blog/dev/updates/1 + + Type Name Status + + pulumi:pulumi:Stack anton-blog-dev created (145s) + + └─ vpcmod:index:Module test-vpc created (143s) + + ├─ vpcmod:index:ModuleState test-vpc-state created (133s) + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.private[0] created (1s) + + ├─ vpcmod:tf:aws_internet_gateway module.test-vpc.aws_internet_gateway.this[0] created (0.58s) + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[1] created (0.76s) + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[1] created (1.00s) + + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.public[0] created (1s) + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[1] created (2s) + + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.public_internet_gateway[0] created (1s) + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[0] created (3s) + + ├─ vpcmod:tf:aws_default_security_group module.test-vpc.aws_default_security_group.this[0] created (4s) + + ├─ vpcmod:tf:aws_default_route_table module.test-vpc.aws_default_route_table.default[0] created (4s) + + ├─ vpcmod:tf:aws_default_network_acl module.test-vpc.aws_default_network_acl.this[0] created (2s) + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[0] created (2s) + + ├─ vpcmod:tf:aws_eip module.test-vpc.aws_eip.nat[0] created (3s) + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[1] created (3s) + + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[0] created (1s) + + ├─ vpcmod:tf:aws_nat_gateway module.test-vpc.aws_nat_gateway.this[0] created (1s) + + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[0] created (2s) + + ├─ vpcmod:tf:aws_vpc module.test-vpc.aws_vpc.this[0] created (2s) + + └─ vpcmod:tf:aws_route module.test-vpc.aws_route.private_nat_gateway[0] created (3s) + +Resources: + + 22 created + +Duration: 2m26s +``` -// Note that pulumi preview and up explain what is changing inside the module. +The infrastructure has now provisioned and the corresponding Terraform state is stored securely inside the Pulumi +state, which can be verified with `pulumi stack export`. +The above program is very simple. To take it further, check out +[examples](https://github.com/pulumi/pulumi-terraform-module/tree/main/examples) for more realistic programs showcasing +features such as computing subnets dynamically with Pulumi `aws.getAvailabilityZonesOutput` function or passing the +results of the VPC module to an EKS module. ## Supported Features +The power of Pulumi is that all components can be composed seamlessly with modules, including chaining and wrapping. + All the regular operations are supported: - Pulumi executes preview/update/refresh as expected @@ -121,7 +224,7 @@ module: - using targeted operations such as `pulumi up --target` - the [protect](https://www.pulumi.com/docs/iac/concepts/options/protect/) resource option -Pulumi's capability to infer accurate types for module inptus and outputs is also currently limited and will sometimes +Pulumi's capability to infer accurate types for module inputs and outputs is also currently limited and will sometimes infer sub-optimal or even unusable types. See [README](https://github.com/pulumi/pulumi-terraform-module) for configuration options. From 0e332fddf5eed7f7cdb8821db6f259cc540beda7 Mon Sep 17 00:00:00 2001 From: Anton Tayanovskyy Date: Wed, 14 May 2025 16:51:22 -0400 Subject: [PATCH 3/4] TODO on cross-configuring a provider --- content/blog/announcing-direct-tf-modules/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/blog/announcing-direct-tf-modules/index.md b/content/blog/announcing-direct-tf-modules/index.md index b67171f23f4f..0b773d5c6596 100644 --- a/content/blog/announcing-direct-tf-modules/index.md +++ b/content/blog/announcing-direct-tf-modules/index.md @@ -202,6 +202,8 @@ The above program is very simple. To take it further, check out features such as computing subnets dynamically with Pulumi `aws.getAvailabilityZonesOutput` function or passing the results of the VPC module to an EKS module. +// TODO show-case cross-configuring the provider with a given AWS region + ## Supported Features The power of Pulumi is that all components can be composed seamlessly with modules, including chaining and wrapping. From 7b51ff01f0451616cba18931b638eb1bf69ec3b9 Mon Sep 17 00:00:00 2001 From: Anton Tayanovskyy Date: Wed, 14 May 2025 16:52:57 -0400 Subject: [PATCH 4/4] TODO showcase converting programs --- content/blog/announcing-direct-tf-modules/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/blog/announcing-direct-tf-modules/index.md b/content/blog/announcing-direct-tf-modules/index.md index 0b773d5c6596..eb17471c81fc 100644 --- a/content/blog/announcing-direct-tf-modules/index.md +++ b/content/blog/announcing-direct-tf-modules/index.md @@ -203,6 +203,7 @@ features such as computing subnets dynamically with Pulumi `aws.getAvailabilityZ results of the VPC module to an EKS module. // TODO show-case cross-configuring the provider with a given AWS region +// TODO show-case converting TF programs with @sandbox annotation ## Supported Features