`r/vapp_entity`: fails with: `...object 'vim.VirtualMachine:resgroup-xxxxxxx' has already been deleted or has not been completely created`

[erSitzt]

Terraform Version

v0.14.5

vSphere Provider Version

v1.24.3

Affected Resource(s)

• vsphere_vapp_entity

Terraform Configuration Files

resource "vsphere_vapp_container" "vapp_container_cluster" {
  name                    = var.cluster_name
  parent_folder_id        = data.vsphere_folder.folder.id
  parent_resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
}

resource "vsphere_vapp_container" "vapp_container_controlplane" {
  name                    = "${var.cluster_name}-controlplane"
  parent_folder_id        = vsphere_vapp_container.vapp_container_cluster.id
  parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}

resource "vsphere_vapp_container" "vapp_container_worker" {
  name                    = "${var.cluster_name}-worker"
  parent_folder_id        = vsphere_vapp_container.vapp_container_cluster.id
  parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}

resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {

  target_id    = vsphere_vapp_container.vapp_container_controlplane.id
  container_id = vsphere_vapp_container.vapp_container_cluster.id
  start_order = 1
  start_action = "powerOn"
  stop_action = "guestShutdown"
}

resource "vsphere_vapp_entity" "vapp_entity_vappworker" {

  target_id    = vsphere_vapp_container.vapp_container_worker.id
  container_id = vsphere_vapp_container.vapp_container_cluster.id
  start_order = 2
  start_action = "powerOn"
  stop_action = "guestShutdown"
}

I have ommitted the other vsphere resources as they are all default stuff... i can add them if necessary :)

Debug Output

I'll add debug output after cleaning up all sensitive data

Panic Output

Expected Behavior

Settings for nested vApps should have been updated

Actual Behavior

Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415598' has already been deleted or has not been completely created

Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415599' has already been deleted or has not been completely created

Steps to Reproduce

created a nested vApp config like above with settings for

• start_order
• start_action
• stop_action

Important Factoids

References

• #0000

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[erSitzt]

Another thing i noticed.. sometimes if there were like 3 vms in a vApp, the settings where applied to one of them in the first run and to the others in the second run of terraform

[erSitzt]

Still a problem with

❯ terraform version
Terraform v0.15.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/vsphere v1.26.0
+ provider registry.terraform.io/rancher/rke v1.0.1

Can anyone verify the problems with nested vApps ?

[tenthirtyam]

Are you still seeing issue or was it resolved since originally posted?

Ryan

[erSitzt]

I'll try to reproduce this and post an update or close the issue

[tenthirtyam]

Thanks for the update @erSitzt. Happy to help if it's still a concern.

Ryan

[erSitzt]

Seems this is still not working...

❯ terraform version
Terraform v1.1.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/vsphere v2.0.2

On first run it fails with there errors:

vsphere_vapp_container.vapp_container_cluster: Creating...
vsphere_vapp_container.vapp_container_cluster: Creation complete after 1s [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creation complete after 0s [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Creating...
vsphere_vapp_container.vapp_container_worker: Creation complete after 1s [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappworker: Creating...
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│   with vsphere_vapp_entity.vapp_entity_vappcontrolplane,
│   on main.tf line 40, in resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane":
│   40: resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│   with vsphere_vapp_entity.vapp_entity_vappworker,
│   on main.tf line 49, in resource "vsphere_vapp_entity" "vapp_entity_vappworker":
│   49: resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
│ 
╵

Running apply again, does not fix it

❯ terraform apply
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # vsphere_vapp_container.vapp_container_cluster has changed
  ~ resource "vsphere_vapp_container" "vapp_container_cluster" {
        id                      = "resgroup-v522179"
        name                    = "test-vapp-issue"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_container.vapp_container_controlplane has changed
  ~ resource "vsphere_vapp_container" "vapp_container_controlplane" {
        id                      = "resgroup-v522180"
        name                    = "test-controlplane"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_container.vapp_container_worker has changed
  ~ resource "vsphere_vapp_container" "vapp_container_worker" {
        id                      = "resgroup-v522181"
        name                    = "test-worker"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane has changed
  ~ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
        id             = "resgroup-v522179:resgroup-v522180"
      ~ stop_action    = "guestShutdown" -> "powerOff"
        # (7 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappworker has changed
  ~ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
        id             = "resgroup-v522179:resgroup-v522181"
      ~ stop_action    = "guestShutdown" -> "powerOff"
        # (7 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
      ~ id             = "resgroup-v522179:resgroup-v522180" -> (known after apply)
      ~ stop_action    = "powerOff" -> "guestShutdown"
        # (7 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappworker is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
      ~ id             = "resgroup-v522179:resgroup-v522181" -> (known after apply)
      ~ stop_action    = "powerOff" -> "guestShutdown"
        # (7 unchanged attributes hidden)
    }

Plan: 2 to add, 0 to change, 2 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│ 
╵

And destroy will fail too...

❯ terraform destroy
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # vsphere_vapp_container.vapp_container_cluster will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_cluster" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522179" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-vapp-issue" -> null
      - parent_folder_id        = "group-v3" -> null
      - parent_resource_pool_id = "resgroup-8" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_container.vapp_container_controlplane will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_controlplane" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522180" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-controlplane" -> null
      - parent_folder_id        = "resgroup-v522179" -> null
      - parent_resource_pool_id = "resgroup-v522179" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_container.vapp_container_worker will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_worker" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522181" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-worker" -> null
      - parent_folder_id        = "resgroup-v522179" -> null
      - parent_resource_pool_id = "resgroup-v522179" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane will be destroyed
  - resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
      - container_id   = "resgroup-v522179" -> null
      - id             = "resgroup-v522179:resgroup-v522180" -> null
      - start_action   = "powerOn" -> null
      - start_delay    = 120 -> null
      - start_order    = 1 -> null
      - stop_action    = "powerOff" -> null
      - stop_delay     = 120 -> null
      - target_id      = "resgroup-v522180" -> null
      - wait_for_guest = false -> null
    }

  # vsphere_vapp_entity.vapp_entity_vappworker will be destroyed
  - resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
      - container_id   = "resgroup-v522179" -> null
      - id             = "resgroup-v522179:resgroup-v522181" -> null
      - start_action   = "powerOn" -> null
      - start_delay    = 120 -> null
      - start_order    = 2 -> null
      - stop_action    = "powerOff" -> null
      - stop_delay     = 120 -> null
      - target_id      = "resgroup-v522181" -> null
      - wait_for_guest = false -> null
    }

Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│ 
╵

[erSitzt]

Btw this is the resulting vApp

[erSitzt]

And this the state file after all this

terraform.tfstate

{
  "version": 4,
  "terraform_version": "1.1.5",
  "serial": 9,
  "lineage": "4f7eef30-b7a8-8da2-31ea-7375f11405c7",
  "outputs": {},
  "resources": [
    {
      "mode": "data",
      "type": "vsphere_compute_cluster",
      "name": "cluster",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "datacenter_id": "datacenter-2",
            "id": "domain-c7",
            "name": "Cluster1",
            "resource_pool_id": "resgroup-8"
          },
          "sensitive_attributes": []
        }
      ]
    },
    {
      "mode": "data",
      "type": "vsphere_datacenter",
      "name": "dc",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "datacenter-2",
            "name": "DC1"
          },
          "sensitive_attributes": []
        }
      ]
    },
    {
      "mode": "data",
      "type": "vsphere_folder",
      "name": "folder",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "group-v3",
            "path": "/DC1/vm"
          },
          "sensitive_attributes": []
        }
      ]
    },
    {
      "mode": "managed",
      "type": "vsphere_vapp_container",
      "name": "vapp_container_cluster",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "cpu_expandable": true,
            "cpu_limit": -1,
            "cpu_reservation": 0,
            "cpu_share_level": "normal",
            "cpu_shares": 4000,
            "custom_attributes": null,
            "id": "resgroup-v522179",
            "memory_expandable": true,
            "memory_limit": -1,
            "memory_reservation": 0,
            "memory_share_level": "normal",
            "memory_shares": 163840,
            "name": "test-vapp-issue",
            "parent_folder_id": "group-v3",
            "parent_resource_pool_id": "resgroup-8",
            "tags": []
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "data.vsphere_compute_cluster.cluster",
            "data.vsphere_datacenter.dc",
            "data.vsphere_folder.folder"
          ]
        }
      ]
    },
    {
      "mode": "managed",
      "type": "vsphere_vapp_container",
      "name": "vapp_container_controlplane",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "cpu_expandable": true,
            "cpu_limit": -1,
            "cpu_reservation": 0,
            "cpu_share_level": "normal",
            "cpu_shares": 4000,
            "custom_attributes": null,
            "id": "resgroup-v522180",
            "memory_expandable": true,
            "memory_limit": -1,
            "memory_reservation": 0,
            "memory_share_level": "normal",
            "memory_shares": 163840,
            "name": "test-controlplane",
            "parent_folder_id": "resgroup-v522179",
            "parent_resource_pool_id": "resgroup-v522179",
            "tags": []
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "data.vsphere_compute_cluster.cluster",
            "data.vsphere_datacenter.dc",
            "data.vsphere_folder.folder",
            "vsphere_vapp_container.vapp_container_cluster"
          ]
        }
      ]
    },
    {
      "mode": "managed",
      "type": "vsphere_vapp_container",
      "name": "vapp_container_worker",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "cpu_expandable": true,
            "cpu_limit": -1,
            "cpu_reservation": 0,
            "cpu_share_level": "normal",
            "cpu_shares": 4000,
            "custom_attributes": null,
            "id": "resgroup-v522181",
            "memory_expandable": true,
            "memory_limit": -1,
            "memory_reservation": 0,
            "memory_share_level": "normal",
            "memory_shares": 163840,
            "name": "test-worker",
            "parent_folder_id": "resgroup-v522179",
            "parent_resource_pool_id": "resgroup-v522179",
            "tags": []
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "data.vsphere_compute_cluster.cluster",
            "data.vsphere_datacenter.dc",
            "data.vsphere_folder.folder",
            "vsphere_vapp_container.vapp_container_cluster"
          ]
        }
      ]
    },
    {
      "mode": "managed",
      "type": "vsphere_vapp_entity",
      "name": "vapp_entity_vappcontrolplane",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "status": "tainted",
          "schema_version": 0,
          "attributes": {
            "container_id": "resgroup-v522179",
            "custom_attributes": null,
            "id": "resgroup-v522179:resgroup-v522180",
            "start_action": "powerOn",
            "start_delay": 120,
            "start_order": 1,
            "stop_action": "powerOff",
            "stop_delay": 120,
            "tags": null,
            "target_id": "resgroup-v522180",
            "wait_for_guest": false
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "data.vsphere_compute_cluster.cluster",
            "data.vsphere_datacenter.dc",
            "data.vsphere_folder.folder",
            "vsphere_vapp_container.vapp_container_cluster",
            "vsphere_vapp_container.vapp_container_controlplane"
          ]
        }
      ]
    },
    {
      "mode": "managed",
      "type": "vsphere_vapp_entity",
      "name": "vapp_entity_vappworker",
      "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
      "instances": [
        {
          "status": "tainted",
          "schema_version": 0,
          "attributes": {
            "container_id": "resgroup-v522179",
            "custom_attributes": null,
            "id": "resgroup-v522179:resgroup-v522181",
            "start_action": "powerOn",
            "start_delay": 120,
            "start_order": 2,
            "stop_action": "powerOff",
            "stop_delay": 120,
            "tags": null,
            "target_id": "resgroup-v522181",
            "wait_for_guest": false
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "data.vsphere_compute_cluster.cluster",
            "data.vsphere_datacenter.dc",
            "data.vsphere_folder.folder",
            "vsphere_vapp_container.vapp_container_cluster",
            "vsphere_vapp_container.vapp_container_worker"
          ]
        }
      ]
    }
  ]
}