fix: use one big vnet and attach AKS clusters to it to avoid creating bastion multiple times

Author: awesomenix

e2e/README.md

@@ -20,38 +20,130 @@ From a high-level, for each scenario,
 To write an E2E scenario,
 
 - choose a testing cluster. There are a few defined
-  in [cluster.go](https://github.com/Azure/AgentBaker/blob/dev/e2e/cluster.go), e.g,
-    - ClusterKubenetAirgap
-    - ClusterAzureNetwork
+  in [cache.go](cache.go), e.g,
     - ClusterKubenet
+    - ClusterAzureNetwork
+    - ClusterAzureOverlayNetwork
+    - ClusterCiliumNetwork
 - use `NodeBootstrappingConfiugration` (`nbc`) to setup your scenario. it is used to invoke the primary
   node-bootstrapping
   API [GetLatestNodeBootstrapping](https://github.com/Azure/AgentBaker/blob/2e730b5a498c5be9b082d912fd08ac9346582db9/pkg/agent/bakerapi.go#L14).
   to modify agentpool properties, usually you need to set both`nbc.containerService.properties.AgentPoolProfiles[0].xxx`
   as well as `nbc.agentPoolProfile`. It is because when RP invokes AgentBaker, it will set the properties in this way
   and in e2e we follow the pattern.
 - use `VMConfigMutator` to set VMSS properties such as SKU when needed.
-  Check [vmss](https://github.com/Azure/AgentBaker/blob/dev/e2e/vmss.go) for other configs.
+  Check [vmss](vmss.go) for other configs.
   it is necessary to set `nbc.agentPoolProfile.VMSize` to match the VMSS SKU if you choose to change.
 - use `Validator` to include your own verification of the VM's live state, such as file existsnce, sysctl settings, etc.
 
+## Infrastructure Architecture
+
+All E2E clusters share a single VNet and Azure Bastion in the `abe2e-{location}` resource group. This
+avoids creating a per-cluster Bastion (~10 min each) and ensures all clusters are reachable from a
+single SSH entry point.
+
+```mermaid
+graph TB
+    subgraph RG["abe2e-{location} Resource Group"]
+        subgraph VNET["abe2e-shared-vnet (10.0.0.0/8)"]
+            BASTION_SUBNET["AzureBastionSubnet<br/>10.0.0.0/26"]
+            FW_SUBNET["AzureFirewallSubnet<br/>10.0.1.0/24"]
+            KUBENET_SUBNET["aks-subnet-abe2e-kubenet-v5<br/>10.1.0.0/20"]
+            AZNET_SUBNET["aks-subnet-abe2e-azure-network-v4<br/>10.1.16.0/20"]
+            OVERLAY_SUBNET["aks-subnet-abe2e-azure-overlay-...<br/>10.1.32.0/20"]
+            MORE_SUBNETS["... more cluster subnets"]
+        end
+        BASTION["abe2e-shared-bastion<br/>(Standard SKU, Tunneling)"]
+        FIREWALL["abe2e-fw<br/>(Azure Firewall)"]
+    end
+
+    subgraph MC_KUBENET["MC_abe2e-kubenet-v5 Resource Group"]
+        VMSS_K["VMSS (system pool)"]
+        VMSS_K_TEST["VMSS (test VMs)"]
+        RT_K["Route Table<br/>(pod routes + firewall)"]
+    end
+
+    subgraph MC_AZNET["MC_abe2e-azure-network-v4 Resource Group"]
+        VMSS_A["VMSS (system pool)"]
+        VMSS_A_TEST["VMSS (test VMs)"]
+    end
+
+    BASTION --> BASTION_SUBNET
+    FIREWALL --> FW_SUBNET
+    VMSS_K --> KUBENET_SUBNET
+    VMSS_K_TEST --> KUBENET_SUBNET
+    RT_K -.->|associated| KUBENET_SUBNET
+    VMSS_A --> AZNET_SUBNET
+    VMSS_A_TEST --> AZNET_SUBNET
+
+    DEV["Developer / CI"]
+    DEV -->|SSH via tunnel| BASTION
+    BASTION -->|"connects to any VM<br/>in shared VNet"| VMSS_K_TEST
+    BASTION -->|"connects to any VM<br/>in shared VNet"| VMSS_A_TEST
+```
+
+### Shared Infrastructure Setup
+
+The shared infrastructure is created **automatically** on first test run via cached idempotent
+functions — no separate setup script is needed.
+
+| Resource | Name | Details |
+|----------|------|---------|
+| VNet | `abe2e-shared-vnet` | `10.0.0.0/8` — supports ~4096 `/20` cluster subnets |
+| Bastion | `abe2e-shared-bastion` | Standard SKU with tunneling enabled for native SSH |
+| Bastion Subnet | `AzureBastionSubnet` | `10.0.0.0/26` (required by Azure Bastion) |
+| Firewall Subnet | `AzureFirewallSubnet` | `10.0.1.0/24` (created by shared infra, firewall on-demand) |
+
+Each AKS cluster gets its own `/20` subnet (4091 usable IPs) in the shared VNet. The subnet is
+named `aks-subnet-{clusterName}`.
+
+### How It Works
+
+1. **`CachedEnsureSharedInfra`** — runs once per location per test run. Creates/verifies the shared
+   VNet, Bastion, and Firewall subnet.
+2. **`CachedEnsureClusterSubnet`** — runs once per cluster. Creates/verifies the cluster's dedicated
+   subnet in the shared VNet.
+3. Each cluster model sets `VnetSubnetID` on the agent pool profile (BYOV — Bring Your Own VNet).
+4. AKS creates VMSS and route tables in the `MC_` resource group, but uses the shared VNet's subnet.
+5. SSH to test VMs goes through the shared Bastion, which can reach any VM in the VNet.
+
+### Test Flow
+
 ```mermaid
 sequenceDiagram
-    E2E->>+ARM: Get or Create AKS Cluster
-    ARM-->>-E2E: Cluster details
-    E2E->>+AgentBakerCode: Fetch VM Configuration (include CSE)
-    AgentBakerCode-->>-E2E: VM Configuration
-    E2E->>+ARM: Create VM using fetched VM Config in cluster network
-    ARM-->>-E2E: VM instance
-    E2E->>+Bastion: Create SSH Tunnel
-    Bastion->>+VM: Forward SSH Connection
-    E2E->>VM: Healthcheck via SSH Tunnel
-    VM-->>E2E: Healthcheck OK
-    E2E->>+KubeAPI: Verify Node Ready
-    KubeAPI-->>-E2E: Node Ready
-    E2E->>VM: Execute test validators via SSH Tunnel
-    VM-->>-E2E: Test results
-    Bastion-->>-E2E: Close SSH Tunnel
+    participant CI as Developer / CI
+    participant Infra as Shared Infra (cached)
+    participant ARM as Azure Resource Manager
+    participant AB as AgentBaker API
+    participant Bastion as Shared Bastion
+    participant VM as Test VM
+    participant K8s as Kube API Server
+
+    CI->>Infra: Ensure shared VNet + Bastion
+    Infra-->>CI: Ready (cached after first run)
+
+    CI->>Infra: Ensure cluster subnet
+    Infra-->>CI: Subnet ID
+
+    CI->>ARM: Create/Get AKS cluster (BYOV subnet)
+    ARM-->>CI: Cluster details
+
+    CI->>AB: Generate CSE + CustomData
+    AB-->>CI: VM configuration
+
+    CI->>ARM: Create VMSS in cluster subnet
+    ARM-->>CI: VM instance
+
+    CI->>Bastion: SSH tunnel to VM private IP
+    Bastion->>VM: Forward SSH connection
+
+    CI->>VM: Run health checks + validators
+    VM-->>CI: Results
+
+    CI->>K8s: Verify node ready
+    K8s-->>CI: Node ready ✓
+
+    Bastion-->>CI: Close tunnel
 ```
 
 ## Running Locally