Runs-on instances private subnet

[jochem725]

I'm looking at some failures in our CI system (postgres user management) and it appears that the runs-on instances from our reference architecture run in a public subnet by default (therefore being unable to access other accounts through the transit gateway.)

I've tried some things such as (looking at the documentation)

• Setting Private to true in the parameters of the runs-on component in the catalog
• Passing private subnets to subnet_ids
• Requesting private=true in the Github action

But none of this seems to lead to a valid (working) configuration that spawns instances in the private subnet.

What is the correct way to configure this within the reference architecture?

[milldr]

RunsOn runners can choose either private or public subnets, depending on the labels. See the private label: https://runs-on.com/configuration/job-labels/#private

However in order to connect to the Transit Gateway private network, you will need to add the a few components to connect the RunsOn VPC to Transit Gateway and to update the VPC subnet table routes.

First add or update a few components in your stacks:
(In the example below, the namespace is acme and the primary region is us-east-1, so of course change this to your namespace and primary region.)

core-auto stack

 # stacks/orgs/acme/core/auto/us-east-1/foundation.yaml
 import:
   - orgs/acme/core/auto/_defaults
   - mixins/region/us-east-1
...
   - catalog/tgw/attachment/defaults
   - catalog/vpc-routes/defaults
   # GitHub Self-Hosted Runners
   - catalog/runs-on/defaults

 components:
   terraform:
...
+    tgw/attachment/runs-on:
+      metadata:
+        component: tgw/attachment
+        inherits:
+          - tgw/attachment
+      vars:
+        enabled: true
+        vpc_id: !terraform.state runs-on vpc_id
+        private_subnet_ids: !terraform.state runs-on private_subnet_ids

     vpc/routes/private:
       metadata:
         component: vpc-routes
         inherits:
           - vpc/routes/defaults
       vars:
         route_table_ids: !terraform.state vpc private_route_table_ids
         routes:
           # Route traffic to core-network
           - destination:
               cidr_block: !terraform.state vpc core-use1-network vpc_cidr
             target:
               type: transit_gateway_id
               value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          # Route traffic to core-auto for RunsOn
+          - destination:
+              cidr_block: !terraform.state runs-on core-use1-auto vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
           # Route traffic to all plat account
           - destination:
               cidr_block: !terraform.state vpc plat-use1-sandbox vpc_cidr
             target:
               type: transit_gateway_id
               value: !terraform.state tgw/hub core-use1-network transit_gateway_id
           - destination:
               cidr_block: !terraform.state vpc plat-use1-dev vpc_cidr
             target:
               type: transit_gateway_id
               value: !terraform.state tgw/hub core-use1-network transit_gateway_id
           - destination:
               cidr_block: !terraform.state vpc plat-use1-staging vpc_cidr
             target:
               type: transit_gateway_id
               value: !terraform.state tgw/hub core-use1-network transit_gateway_id
           - destination:
               cidr_block: !terraform.state vpc plat-use1-prod vpc_cidr
             target:
               type: transit_gateway_id
               value: !terraform.state tgw/hub core-use1-network transit_gateway_id

+    vpc/routes/runs-on:
+      metadata:
+        component: vpc-routes
+        inherits:
+          - vpc/routes/defaults
+      vars:
+        route_table_ids: !terraform.state runs-on private_route_table_ids
+        routes:
+          # Route traffic to core-network
+          - destination:
+              cidr_block: !terraform.state vpc core-use1-network vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          # Route traffic to core-auto default VPC
+          - destination:
+              cidr_block: !terraform.state vpc core-use1-auto vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          # Route traffic to all plat account
+          - destination:
+              cidr_block: !terraform.state vpc plat-use1-sandbox vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          - destination:
+              cidr_block: !terraform.state vpc plat-use1-dev vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          - destination:
+              cidr_block: !terraform.state vpc plat-use1-staging vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+          - destination:
+              cidr_block: !terraform.state vpc plat-use1-prod vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id

core-network stack

# stacks/orgs/acme/core/network/us-east-1/foundation.yaml
import:
  - orgs/acme/core/network/_defaults
  - mixins/region/us-east-1
  - catalog/vpc/defaults
  - catalog/vpc-flow-logs-bucket/defaults
  - catalog/ec2-client-vpn/defaults
  - catalog/tgw/hub/defaults
  - catalog/tgw/attachment/defaults
  - catalog/tgw/routes/defaults
  - catalog/vpc-routes/defaults
  - catalog/bastion/defaults

components:
  terraform:
    vpc:
      metadata:
        component: vpc
        inherits:
          - vpc/defaults
      vars:
        ipv4_primary_cidr_block: 10.9.16.0/20

    tgw/routes:
      vars:
        transit_gateway_route_tables:
          - transit_gateway_route_table_id: !terraform.state tgw/hub core-use1-network transit_gateway_route_table_id
            routes:
              # Local VPC TGW Attachment (this account)
              - attachment_id: !terraform.state tgw/attachment transit_gateway_vpc_attachment_id
              # core-auto
              - attachment_id: !terraform.state tgw/attachment core-use1-auto transit_gateway_vpc_attachment_id
+             # core-auto RunsOn VPC
+             - attachment_id: !terraform.state tgw/attachment/runs-on core-use1-auto transit_gateway_vpc_attachment_id
              # plat accounts
              - attachment_id: !terraform.state tgw/attachment plat-use1-sandbox transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use1-dev transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use1-staging transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use1-prod transit_gateway_vpc_attachment_id

    vpc/routes/private:
      metadata:
        component: vpc-routes
        inherits:
          - vpc/routes/defaults
      vars:
        route_table_ids: !terraform.state vpc core-use1-network private_route_table_ids
        routes:
          # Route traffic to core-auto
          - destination:
              cidr_block: !terraform.state vpc core-use1-auto vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+         - destination:
+              cidr_block: !terraform.state runs-on core-use1-auto vpc_cidr
+            target:
+              type: transit_gateway_id
+              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
          # Route traffic to all plat accounts
          - destination:
              cidr_block: !terraform.state vpc plat-use1-sandbox vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
          - destination:
              cidr_block: !terraform.state vpc plat-use1-dev vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
          - destination:
              cidr_block: !terraform.state vpc plat-use1-staging vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
          - destination:
              cidr_block: !terraform.state vpc plat-use1-prod vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id

and for all `plat` accounts using the shared configure

# stacks/catalog/vpc-routes/plat.yaml
import:
  - catalog/vpc-routes/defaults

components:
  terraform:
    vpc/routes/private:
      metadata:
        component: vpc-routes
        inherits:
          - vpc/routes/defaults
      vars:
        route_table_ids: !terraform.state vpc private_route_table_ids
        routes:
          # Route traffic to core-network
          - destination:
              cidr_block: !terraform.state vpc core-use1-network vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
          # Route traffic to core-auto
          - destination:
              cidr_block: !terraform.state vpc core-use1-auto vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use1-network transit_gateway_id
+         # Route traffic to core-auto RunsOn VPC
+         - destination:
+             cidr_block: !terraform.state runs-on core-use1-auto vpc_cidr
+           target:
+             type: transit_gateway_id
+             value: !terraform.state tgw/hub core-use1-network transit_gateway_id

Then deploy the components:

1. Deploy the Transit Gateway attachment for the RunsOn VPC:

atmos terraform apply tgw/attachment/runs-on -s core-use1-auto

2. Add the new Transit Gateway route:

atmos terraform apply tgw/routes -s core-use1-network

3. Deploy the VPC routes for the RunsOn VPC itself:

atmos terraform apply vpc/routes/runs-on -s core-use1-auto

4. Then deploy the VPC routes for all other VPCs to allow RunsOn VPC to connect:

atmos terraform apply vpc/routes/private -s core-use1-network
atmos terraform apply vpc/routes/private -s core-use1-auto
atmos terraform apply vpc/routes/private -s plat-use1-dev
atmos terraform apply vpc/routes/private -s plat-use1-staging
atmos terraform apply vpc/routes/private -s plat-use1-prod

[jochem725]

Thank you @milldr - I was able to verify that this was already mostly in the ref arch.

Ultimately the problem was that we require both public/private workers and there was no way to pass both public and private subnets when the networking_stack is set to external (so it uses the existing VPC of core auto).

I've opened a pull request in the runs-on component to allow passing both public and private subnets.

Now I can pass the subnets as follows and everything works as expected.

private_subnet_ids: !terraform.output vpc core-use1-auto private_subnet_ids
public_subnet_ids: !terraform.output vpc core-use1-auto public_subnet_ids

[goruha]

@jochem725 thanks for your contribution.
I think we should merge the PR after you address the changes request in the PR cloudposse-terraform-components/aws-runs-on#43 (review)

[GabisCampana]

Hi @jochem725,

Thanks again for digging into this and for opening the PR to support both public and private subnets.

At this point, the remaining blocker is the review feedback Igor left on the PR (cloudposse-terraform-components/aws-runs-on#43). Once those requested changes are addressed and the PR is merged, this should fully resolve the limitation you ran into.

Please let us know if you need any clarification on the review comments or want help validating the final configuration.

Thanks.