EKS control plane ENIs are placed in secondary (CNI/pod) subnets — makeVpcConfig doesn't apply FilterNonCni() #6172

/kind bug

What steps did you take and what happened:

In an unmanaged VPC with secondary subnets tagged
sigs.k8s.io/cluster-api-provider-aws/association=secondary (per the
pod networking docs),
CAPA passes all subnets from the AWSManagedControlPlane spec — including
the CNI/pod subnets — to EKS CreateCluster. EKS then arbitrarily picks 2+
AZs for its cross-account control plane ENIs, which can (and did, in our
case) land in the pod subnets.

What did you expect to happen:

Control plane ENIs should only be placed in non-CNI subnets, consistent with
how CAPA already handles nodegroup and EC2 instance placement.

Why it matters:

Secondary CIDRs are commonly non-routable CGNAT ranges (e.g. 100.64.0.0/16)
reused per-VPC. When control plane ENIs land there, the private endpoint
becomes unreachable from outside the VPC — including from the CAPA
management cluster itself, which then fails all workload-cluster
reconciliation (ReconcileCNI, ENIConfig creation, vpcCni env vars) with
connection timeouts. The failure is silent and hard to trace back to subnet
placement.

Analysis:

#4800 established that CNI-tagged subnets are not intended to host
infrastructure, and introduced FilterNonCni(), applied to EKS nodegroups
and EC2 instance placement. makeVpcConfig
(pkg/cloud/services/eks/cluster.go) is the remaining unfiltered path — it
still carries a // TODO: Do we need to just add the private subnets?.
restrictPrivateSubnets doesn't help since secondary subnets are private.

Proposed fix:

Apply subnets.FilterNonCni() at the top of makeVpcConfig, consistent
with #4800. (A longer-term alternative: a dedicated controlPlaneSubnets
field, similar to the Terraform EKS module's control_plane_subnet_ids.)

Workaround:

Post-creation
aws eks update-cluster-config --resources-vpc-config subnetIds=<non-CNI subnets>
(CAPA doesn't reconcile SubnetIds, so it isn't reverted).

Environment:

• cluster-api-provider-aws version: v2.12.1
• AWSManagedControlPlane, unmanaged (static) VPC, EKS 1.34