TF Provider tries to change order auf VTI Default Rules in nsxt_policy_predefined_gateway_policy

[benzander]

Describe the bug

I use nsxt_policy_predefined_gateway_policy to change the default rule action from ALLOW to Drop in all gateway Firewalls.
For every Route Based VPN the NSX Manager creates a default rule. So I use the same ressource to set those rules to DROP as well.

Unfortunately I run into an issue in case there are multiple Route Based VPNs in the Same T1 or T0.
The TF Providers tries to change the order of those default rules. That is how the Change looks like with 4 Route Based VPNs:

Terraform will perform the following actions:

  # module.vrf_On_CustomerT0_t0V-001.nsxt_policy_predefined_gateway_policy.gwFW-predefinedGatewayPolicy will be updated in-place
  ~ resource "nsxt_policy_predefined_gateway_policy" "gwFW-predefinedGatewayPolicy" {
        id          = "Policy_Default_Infra-tier0-df1f3954-0683-43db-89a0-f5bfe52fe142"
        # (3 unchanged attributes hidden)

      ~ default_rule {
          ~ scope           = "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/84f05f4a-e89c-4d1e-a1bc-106a08a8e724" -> "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/50c5e8d2-42b9-48d4-b705-640ffbe224b7"
            # (8 unchanged attributes hidden)
        }
      ~ default_rule {
          ~ scope           = "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/8ed8467a-deed-4d01-84eb-7623d6cd8814" -> "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/84f05f4a-e89c-4d1e-a1bc-106a08a8e724"
            # (8 unchanged attributes hidden)
        }
      ~ default_rule {
          ~ scope           = "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/f71d6083-d34d-456c-847c-142156b0983d" -> "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/8ed8467a-deed-4d01-84eb-7623d6cd8814"
            # (8 unchanged attributes hidden)
        }
      ~ default_rule {
          ~ scope           = "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/50c5e8d2-42b9-48d4-b705-640ffbe224b7" -> "/infra/tier-0s/df1f3954-0683-43db-89a0-f5bfe52fe142/ipsec-vpn-services/8241df91-878f-45f9-8902-8d16a39d511e/sessions/f71d6083-d34d-456c-847c-142156b0983d"
            # (8 unchanged attributes hidden)
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

That is how the TF Code looks like:

resource "nsxt_policy_predefined_gateway_policy" "gwFW-predefinedGatewayPolicy" {
  path = "/infra/domains/default/gateway-policies/Policy_Default_Infra-tier0-${nsxt_policy_tier0_gateway.t0V.id}"

  dynamic "default_rule" {

    for_each = toset(concat(length(keys(var.vpnConfiguration)) > 0 ? [for session in module.VpnTunnels["1"].vpnSessions : session.path if session.vpn_type == "RouteBased"] : []))

    content {
      scope     = default_rule.value
      logged    = true
      log_label = local.fwRuleLogLabel
      action    = "DROP"
    }
  }

  default_rule {
    scope     = nsxt_policy_tier0_gateway.t0V.path
    logged    = true
    log_label = local.fwRuleLogLabel
    action    = "DROP"
  }
}

I can apply those changes without an error message but during the next tf plan the same change is detected again.
I would sort the input for my for_each statement into the same order as the NSX Manager is sorting the VTI Default Rules, but I don't know what the correct order is.

Or do you have the possibility to fix that somehow in the nsx tf provider?

Reproduction steps

See description

Expected behavior

See description

Additional context

Terraform Version: 1.8.5
NSX TF Provider: v3.8.0
NSX Manager: 4.1.2.4.0.23786742

[annakhm]

Hi @benzander, thanks for raising this. The order of rules has significance in NSX, and therefore in provider (looks like in this particular use case the order is not important, however we have to respect the general use case). You can check the actual order on NSX via API or UI - I expect it to be consistent given that the order of creation of VPN tunnels is consistent. Hopefully this could be a workaround for short term?
One idea to solve this more elegantly could be to create a separate nsxt_policy_predefined_gateway_policy_rule resource. We'll discuss this and potential other solutions.

[benzander]

Hi @annakhm,
thanks for the fast response.
I agree that the order of firewall rules have significance. For the default rules the NSX manager doesn't provide the option to change the order.
In played around a bit and the order is indeed based on the creation of the tunnel. So a rule for a new tunnel is added over the existing default rules.

I tried to work around by using one nsxt_policy_predefined_gateway_policy per default rule, but that doesn't work either. The TF Provider wants to change the order here too. I also saw other site effects. When deleting tunnels it left the first default rule on the default action (allow). Another tf apply was required to change it back zo drop. So doesn't seem to be a good idea to use this resource multiple times for the same gateway.

[annakhm]

I have another question - do you create VPN tunnels using terraform as well? You might not be able to predict the order of creation if those are parallelized by terraform.
Indeed, creating multiple nsxt_policy_predefined_gateway_policy resources for the same policy is not a good idea since all those instances would try to update the same object.

[benzander]

Yes, we are using Terraform to create the VPN Tunnels. We are doing the creation of the VPN Tunnels in a Terraform Module which also makes sure that the default firewall rule will be changed from allow to drop.
I also tried to predict the order and noticed that this is not possible.
We are running scheduled terraform plans for all customers to make sure the desired config matches the configured config. And that causes "config drift notifications". It would be very helpful if you could improved that in the TF Provider.