VPC creation should be more customizable

[inductor]

Why do you want this feature?
Currently you can create a VPC with the CIDR block, but you cannot customize the subnet CIDR range and AZs that they belong to.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: foo-cluster
  region: ap-northeast-1

vpc:
  cidr: "10.0.0.0/16"

What feature/behavior/change do you want?
Without giving a subnet ID, it should be able to create new subnets on the AZs

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: foo-cluster
  region: ap-northeast-1

vpc:
  cidr: "10.0.0.0/16"
  subnets:
    private:
      ap-northeast-1a:
        cidr: "10.0.0.0/24"
      ap-northeast-1b:
        cidr: "10.0.1.0/24"
    public:
      ap-northeast-1a:
        cidr: "10.0.2.0/24"
      ap-northeast-1b:
        cidr: "10.0.3.0/24"

[inductor]

This is useful when you want to deploy other AWS resources on your VPC/subnets.
For example, when you want to create prd/stg/dev environment with EKS + RDS in your PRD/STG/DEV environment, you need to set different subnet IDs on each environment and it has to be specified without using any sort of alias.

If this feature is implemented, all you need is just create a cluster with eksctl first, then you use the CloudFormation output from eksctl to refer which subnets it has to be and create RDS environment on top of it.

[sekka1]

I agree, this is a useful feature. I would go even further and allow someone to pass in a VPC ID for it to do what the other people said above. From a perspective of an infrastructure operator, Kubernetes is just another application and I have many applications in my VPC that has to work together in an environment. I mostly don't want my Kubernetes tool to be building my VPC. What if later I dont want to use this tool anymore? I am then stuck with it managing my VPC state.

Here is my usual workflow:

1. Launch VPC with default routing via Terraform
2. Do necessary peering via Terraform
3. Start launching apps
4. Launch a Kubernetes cluster in the VPC - I want this to build it's own world that it needs in my VPC (subnets, routing, etc). This would allow me to perform CRUD operations on it without affecting anything else.
5. Launch RDS in the VPC - I want this to build it's own world that it needs in my VPC (subnets, routing, etc). This would allow me to perform CRUD operations on it without affecting anything else.
6. Launch other apps in a similar way

I get why the current deployment model is in place. It is the first pass at this eksctl tool. Kops is another kubernetes tool and that has a semantics to do the above and I think eksctl should also.

[morrissimo]

From the docs, I assumed this capability already existed! Have exactly the same use case as the OP: existing VPC, want eksctl to create new subnets with specified CIDRs in specified AZs

[darnone]

This would definitely be a very useful feature. I was just searching if there was a way to do this and come across this. The above feature would generate the necessary networking for you rather than having to use CloudFormation or Terraform to first create the infrastructure and then use eksctl to redeploy in the existing VPC. It is like putting both steps together. So is this not yet available?

[Himangini]

Need to investigate but I think we already support this. @aclevername @nikimanoledaki @cPu1 can anyone of you take a look at this, please.

[aclevername]

Need to investigate but I think we already support this. @aclevername @nikimanoledaki @cPu1 can anyone of you take a look at this, please.

This is not currently supported:

$ cat cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: jk-test
  region: us-west-2

vpc:
  cidr: "10.0.0.0/16"
  subnets:
    private:
      us-west-2a:
        cidr: "10.0.0.0/24"
      us-west-2b:
        cidr: "10.0.1.0/24"
    public:
      us-west-2a:
        cidr: "10.0.2.0/24"
      us-west-2b:
        cidr: "10.0.3.0/24"

$ eksctl create cluster -f cluster.yaml
2021-11-17 12:13:20 [ℹ]  eksctl version 0.75.0-dev+664ee7cc.2021-11-17T12:12:20Z
2021-11-17 12:13:20 [ℹ]  using region us-west-2
Error: can't describe subnet by CIDR without VPC id

[Himangini]

We need to Spike on this first to come up with a nice UX for supporting the subnets settings since we already use this syntax to specify subnet topology.

We'll add the findings here on this ticket on how much work is involved

Timebox: 1-2 days

[TiberiuGC]

I'll start breaking this down but first explaining how things are working at the moment.

Scenario 1 - using existing VPC

If the VPC is imported (i.e. VPC ID is specified) then we try to import the subnets as-well, thus subnets must already exist within the imported VPC. We do so by calling EKS API on DescribeSubnets endpoint. Parameters like subnet ID, AZ and CIDR are being used as filters in this search. In case multiple subnets are returned after filtering, one is selected at random. If a subnet is not being found, we return an error.

Scenario 2 - using new VPC

If the VPC is created with the cluster (i.e. VPC ID is not specified) then one shouldn't specify subnets topology, as the subnets will be created by eksctl in either user specified or randomly selected AZs. The subnet CIDR allocation rules are reflected in the code snippets below.

https://github.com/weaveworks/eksctl/blob/098db7655c14a7edc465a9d76088f4195b1f2620/pkg/vpc/vpc.go#L32-L45

Basically, eksctl creates one public and one private subnet for each defined/selected AZ. Thus, it splits the VPC CIDR into 2*AZ CIDRs of equal lengths and assigns one to each subnet.

https://github.com/weaveworks/eksctl/blob/098db7655c14a7edc465a9d76088f4195b1f2620/pkg/vpc/vpc.go#L56-L77

What needs to change

With this feature, we want to allow subnets to also be created and not only imported, accordingly to the user defined subnets topology, in both scenarios presented above.

For scenario one - we could still try at first to import the subnets from the existing VPC, basically doing the same as before. However if any of the subnets cannot be found (i.e. don't already exist), we should not return an error anymore, but rather create them as per the configuration provided.

For scenario two - if subnets topology does not contain CIDRs, we can still rely on the strategy described above. However, if the topology specifies CIDRs, then I suggest we drop the existing allocation strategy and use the user defined CIDRs as is. We probably don't want to have it both ways (i.e. some subnets having user defined CIDRs while others relying on eksctl for allocation), as this would substantially complicate the way in which we break down VPC CIDR into individual subnet CIDR and it's probably not worth the effort. Thus, we should enforce a validation that if a user provides a desired CIDR for one subnet, then they should be doing the same for all.

Validations

We need to enforce a few validations on the user defined CIDRs

• they are not overlapping
• they are contained within VPC's CIDR
• they must be set for either all or none of the subnets

Other considerations

In terms of UX, I think we can stick to the config file format we're using right now. With the proposed approach, eksctl will be able to decide, behind the scenes, which subnets need to be created and which already exist and will be used as is. As long as the configuration ends up as the user defines it in the config file, they don't need to be bothered by the exact implementation details.

As this feature involves a considerable logic refactoring, we need to ensure thorough testing is being conducted (unit tests, integration tests and manual tests).

[bwagner5]

Any updates? Would love to have this feature!