Changes on T0 VRFs in nsxt_policy_tier0_gateway ressource causes error during apply

[benzander]

Describe the bug

It is not possibile to do changes on a T0 VRF.

I get this error:
Error: Failed to update Tier0 99da836b-3011-4529-9f00-d78b4f61a613: Error while creating objects of type:Tier0 (code 500157) │ Related errors: │ Forwarding up-timer for VRF tier-0 /infra/tier-0s/99da836b-3011-4529-9f00-d78b4f61a613 cannot be modified. It is inherited from parent tier-0. (code 528061) │ │ with module.vrf_On_CustomerT0_t0V-001.nsxt_policy_tier0_gateway.t0V, │ on .terraform/modules/vrf_On_CustomerT0_t0V-001/030_t0V.tf line 37, in resource "nsxt_policy_tier0_gateway" "t0V": │ 37: resource "nsxt_policy_tier0_gateway" "t0V" {

We rarely do changes on VRFs. We usually create or destroy them. That is why I don't know since which version this issue exists.
You can see this issue e.g. if you set a description and then try to change it.

Terraform v1.8.5

• provider registry.terraform.io/vmware/nsxt v3.8.0
  NSX Manager: 4.2.1.3.0.24533887

Reproduction steps

.

Expected behavior

.

Additional context

No response

[annakhm]

Hi @benzander, could you please provide reproduction steps? I've tried a few configs (beyond our regular acceptance tests) and I'm not able to reproduce this.

[benzander]

Hi @annakhm,
I can reproduce it that way:

resource "nsxt_policy_tier0_gateway" "t0V" {
  display_name         = "Test VRF"
  description          = "Testing for https://github.com/vmware/terraform-provider-nsxt/issues/1555"
  failover_mode        = "PREEMPTIVE"
  default_rule_logging = true
  enable_firewall      = true
  ha_mode              = "ACTIVE_STANDBY"

  locale_service {
    edge_cluster_path    = "/infra/sites/default/enforcement-points/default/edge-clusters/b78d8e4e-3de5-44b0-932f-9e8f98d933dd"
    preferred_edge_paths = ["/infra/sites/default/enforcement-points/default/edge-clusters/b78d8e4e-3de5-44b0-932f-9e8f98d933dd/edge-nodes/0"]
  }

  vrf_config {
    gateway_path = "/infra/tier-0s/b80a93b7-66ce-4e2b-b5f4-1843e7257f13"
  }

  tag {
    scope = "ben"
    tag   = "test"
  }
}

Then I try to change the description:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # nsxt_policy_tier0_gateway.t0V will be updated in-place
  ~ resource "nsxt_policy_tier0_gateway" "t0V" {
      ~ description              = "Testing for https://github.com/vmware/terraform-provider-nsxt/issues/1555" -> "Testing for https://github.com/vmware/terraform-provider-nsxt/issues/1555 - change"
        id                       = "851c87d0-0fb8-44cb-9289-f2daba3b554b"
        # (16 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

nsxt_policy_tier0_gateway.t0V: Modifying... [id=851c87d0-0fb8-44cb-9289-f2daba3b554b]
nsxt_policy_tier0_gateway.t0V: Still modifying... [id=851c87d0-0fb8-44cb-9289-f2daba3b554b, 10s elapsed]
╷
│ Error:  Failed to update Tier0 851c87d0-0fb8-44cb-9289-f2daba3b554b: Error while creating objects of type:Tier0 (code 500157)
│ Related errors:
│ Forwarding up-timer for VRF tier-0 /infra/tier-0s/851c87d0-0fb8-44cb-9289-f2daba3b554b cannot be modified. It is inherited from parent tier-0. (code 528061) 
│ 
│   with nsxt_policy_tier0_gateway.t0V,
│   on 100_t0-001.tf line 80, in resource "nsxt_policy_tier0_gateway" "t0V":
│   80: resource "nsxt_policy_tier0_gateway" "t0V" {
│ 
╵

Have you tested against a 4.2.x NSX Manager Version? There was a change in the API:

Since Version 4.2 the default value of forwarding_up_timer was changed from 0 to 5. Our NSX Managers were upgraded from previous 4.2 and obviously the upgrade process has not changed this value. We still have forwarding_up_timer 0 in all our T0s.

[annakhm]

Thank you @benzander, this is very helpful. This looks like an upgrade issue, following default change which is not backward compatible. Terraform does not support this timer setting, which would cause NSX to grab the default value and detect a conflict.

From the provider side, the solution is to offer support for advanced_config. I'll also check with NSX team.

[annakhm]

Suggested workaround for the time being - In case you don't mind changing timer value, manually setting both values to 5 should fix this issue.

[benzander]

@annakhm:
Actually I was not aware about this timer value until we experienced that terraform issue. But I guess 5 is the better default the 0. So we are going to change that value anyway.

I've tried your suggested workaround. So I've changed the time to 5 (over the NSX Manager GUI. The Policy GUI doesn't support this value) for the parent t0.
Changing that to 5 for a vrf is not supportet. The GUI says "[Routing] Forwarding up timer in routing config cannot be modified on VRF. (Error code: 10192)"
The GUI shows 0 for all VRFs unrelated to what the parentT0 has set.
But that change didn't work... TF shows the same error during a change.

Strange is that I've created a new parent T0 and a new VRF. So that is one which was not upgraded from before NSX 4.2.
The Parent T0 is deployed with the new default 5. The new VRF shows still 0.
Updates of the VRF via TF work.
If I change this timer value from 5 to 0 for the new Parent T0 changes for the VRF are still possible.

So all VRFs running on a ParentT0 that were upgraded cannot be changed by TF.
For all VRFs on new T0s that were deployed after the NSX 4.2 upgrade can be changed and I cannot break that.

[annakhm]

Unfortunately Manager GUI doesn't change the same object, so this would not help..
I'm curious - if you try to update description in the upgraded VRF via UI, would that work?