Open
Description
Describe the bug
I am trying to use AKO with NSX-T cloud in a dedicated non-admin tenant. I have deployed all the required infrastructure in NSX-T and the AVI tenant. A manually created VS works fine. But AKO cannot set up static routes on VRF because it tries to look for them in the admin tenant (note the words tenant admin):
ako-0 ako 2024-03-19T15:23:37.717Z INFO api/api.go:52 Setting route for GET /api/status
ako-0 ako 2024-03-19T15:23:37.718Z INFO ako-main/main.go:77 AKO is running with version: v1.11.3
ako-0 ako 2024-03-19T15:23:37.718Z INFO api/api.go:110 Starting API server at :8080
ako-0 ako 2024-03-19T15:23:37.718Z INFO ako-main/main.go:87 We are running inside kubernetes cluster. Won't use kubeconfig files.
ako-0 ako 2024-03-19T15:23:37.718Z INFO lib/control_config.go:314 Setting the client version to AVI Max supported version 22.1.3
ako-0 ako 2024-03-19T15:23:37.730Z INFO ako-main/main.go:174 Kubernetes cluster apiserver version 1.27
ako-0 ako 2024-03-19T15:23:37.736Z INFO utils/utils.go:171 Initializing configmap informer in avi-system
ako-0 ako 2024-03-19T15:23:39.674Z INFO cache/controller_obj_cache.go:2359 Avi cluster state is CLUSTER_UP_HA_ACTIVE
ako-0 ako 2024-03-19T15:23:39.978Z INFO cache/controller_obj_cache.go:3131 Setting cloud vType: CLOUD_NSXT
ako-0 ako 2024-03-19T15:23:39.978Z INFO cache/controller_obj_cache.go:3134 Setting cloud uuid: cloud-77594459-fc93-467c-9588-5821fb8d9360
ako-0 ako 2024-03-19T15:23:39.978Z INFO lib/lib.go:301 Setting AKOUser: ako-albVS-ORG0001-k8s for Avi Objects
ako-0 ako 2024-03-19T15:23:40.085Z INFO cache/controller_obj_cache.go:3413 Skipping the check for Node Network
ako-0 ako 2024-03-19T15:23:40.365Z INFO cache/controller_obj_cache.go:3583 Setting VRF T1_ORG0001-001-DATA found that matches the T1Lr /infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f
ako-0 ako 2024-03-19T15:23:40.366Z INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"avi-system", Name:"ako-0", UID:"7dad8c70-4398-4a51-9066-8675a2753ee5", APIVersion:"v1", ResourceVersion:"53128292", FieldPath:""}): type: 'Normal' reason: 'ValidatedUserInput' User input validation completed.
ako-0 ako 2024-03-19T15:23:40.370Z INFO lib/lib.go:240 Setting Disable Sync to: false
ako-0 ako 2024-03-19T15:23:40.373Z INFO k8s/ako_init.go:276 avi k8s configmap created
ako-0 ako 2024-03-19T15:23:41.725Z WARN lib/avi_api.go:65 msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.816Z WARN lib/avi_api.go:65 msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z WARN lib/avi_api.go:65 msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z WARN cache/controller_obj_cache.go:2521 DNSProperty Get uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 returned err msg: AviGet retried 3 times, aborting
ako-0 ako 2024-03-19T15:23:41.907Z WARN cache/controller_obj_cache.go:2466 Cloud: albC-ORG0001 does not have a dns provider configured
ako-0 ako 2024-03-19T15:23:43.059Z WARN nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/nginx/nginx, msg: nothing to delete for route: nginx
ako-0 ako 2024-03-19T15:23:43.059Z WARN nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/oauth/dex, msg: nothing to delete for route: dex
ako-0 ako 2024-03-19T15:23:43.059Z WARN nodes/avi_model_l7_translator.go:253 key: Ingress/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found
ako-0 ako 2024-03-19T15:23:43.071Z WARN status/svc_status.go:39 key: syncstatus, msg: Service hostname not found for service [nginx-ingress-controller/nginx-ingress-controller] status update
ako-0 ako 2024-03-19T15:23:43.100Z WARN rest/dequeue_nodes.go:65 key: ORG0001/DummyVSForStaleData, msg: no model found for the key
ako-0 ako 2024-03-19T15:23:43.406Z WARN rest/rest_operation.go:304 key: ORG0001/T1_ORG0001-001-DATA, msg: RestOp method PUT path /api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71 tenant admin Obj {"_last_modified":"1710833802833745","attrs":[{"key":"tier1path","value":"/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f"}],"cloud_ref":"https://nsx-alb.tld.de/api/cloud/cloud-77594459-fc93-467c-9588-5821fb8d9360","lldp_enable":true,"name":"T1_ORG0001-001-DATA","static_routes":[{"next_hop":{"addr":"10.70.27.33","type":"V4"},"prefix":{"ip_addr":{"addr":"0.0.0.0","type":"V4"},"mask":0},"route_id":"1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.35","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.1.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.34","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.2.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-2"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.36","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.0.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-3"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.40","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.4.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-4"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.41","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.5.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-5"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.42","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.6.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-6"}],"system_default":false,"tenant_ref":"https://nsx-alb.tld.de/api/tenant/tenant-1e7619a4-a24f-4e9c-baa7-edd2c2dab69f","url":"https://nsx-alb.tld.de/api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","uuid":"vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71"} returned err {"code":0,"message":"map[error:VrfContext object not found!]","Verb":"PUT","Url":"https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","HttpStatusCode":404} with response null
ako-0 ako 2024-03-19T15:23:43.406Z WARN rest/dequeue_nodes.go:624 key: ORG0001/T1_ORG0001-001-DATA, msg: there was an error sending the macro Error during PUT: Encountered an error on PUT request to URL https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71: HTTP code: 404; error from Avi: map[error:VrfContext object not found!]
ako-0 ako 2024-03-19T15:23:43.406Z WARN rest/dequeue_nodes.go:659 key: ORG0001/T1_ORG0001-001-DATA, msg: Avi model not set, possibly a DELETE call
ako-0 ako 2024-03-19T15:23:44.098Z WARN nodes/avi_model_l7_translator.go:253 key: Endpoints/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found
Reproduction steps
- Create all required networks and firewall rules in NSX-T manager
- Create tenant, user, cloud, IPAM profile, SEG, network profiles in NSXALB manager (VRFs are auto-created during cloud creation)
- Deploy AKO with nodeNetworkList and ClusterIP mode
AKOSettings:
clusterName: 'k8stest' # A unique identifier for the kubernetes cluster, that helps distinguish the objects for this cluster in the avi controller. // MUST-EDIT
cniPlugin: 'cilium' # Set the string if your CNI is calico or openshift or ovn-kubernetes. For Cilium CNI, set the string as cilium only when using Cluster Scope mode for IPAM and leave it empty if using Kubernetes Host Scope mode for IPAM. enum: calico|canal|flannel|openshift|antrea|ncp|ovn-kubernetes|cilium
disableStaticRouteSync: 'false' # If the POD networks are reachable from the Avi SE, set this knob to true.
### This section outlines the network settings for virtualservices.
NetworkSettings:
nsxtT1LR: '/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f' # Unique ID (note: not display name) of the T1 Logical Router for Service Engine connectivity. Only applies to NSX-T cloud.
# Network information of the VIP network. Multiple networks allowed only for AWS Cloud.
# Either networkName or networkUUID should be specified.
# If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
vipNetworkList:
- networkName: 'segO-T1_ORG0001-001-VIP-0001'
## This list of network and cidrs are used in pool placement network for vcenter cloud.
## Node Network details are not needed when in nodeport mode / static routes are disabled / non vcenter clouds.
## Either networkName or networkUUID should be specified.
## If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
nodeNetworkList:
- networkName: "segO-T1_ORG0001-001-0001"
cidrs:
- 10.244.0.0/16
L4Settings:
defaultDomain: '' # If multiple sub-domains are configured in the cloud, use this knob to set the default sub-domain to use for L4 VSes.
autoFQDN: disabled # ENUM: default(<svc>.<ns>.<subdomain>), flat (<svc>-<ns>.<subdomain>), "disabled" If the value is disabled then the FQDN generation is disabled.
#L7Settings:
# serviceType: NodePort # enum NodePort|ClusterIP|NodePortLocal
### This section outlines settings on the Avi controller that affects AKO's functionality.
ControllerSettings:
serviceEngineGroupName: 'albSEG-ORG0001' # Name of the ServiceEngine Group.
cloudName: 'albC-ORG0001' # The configured cloud name on the Avi controller.
controllerHost: 'nsx-alb.tld.de' # IP address or Hostname of Avi Controller
controllerVersion: '22.1.5' # The controller API version
tenantsPerCluster: true # If set to true, AKO will map each k8s cluster uniquely to a tenant in AVI.
tenantName: 'ORG0001' # Name of the tenant where all the AKO objects will be created in AVI.
avicredentials:
username: 'ako-ORG0001'
password: ''
authtoken: 'redacted'
certificateAuthorityData: |-
-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----
- See that AKO is unable to set up static routes on VRF in non-admin tenant
Expected behavior
AKO is able to set up static routes on VRF in non-admin tenant
Additional context
AVI Controller Version: 22.1.5
NSX-T Version: 4.1.2.3