VM always changed in eptRviMode,hvMode

[cultpony]

What happened?

I attempted to manage several VMs in a vSphere 8 installation via Pulumi, the VMs are created with cloud-init configuration and from a template, but eptRviMode and hvMode are not explicitly set at all.

Example

new vsphere.VirtualMachine("vm1", {
  name: "vm1",
  clone: { templateUuid: template.uuid },
  networkInterfaces: [ <omitted for brevity> ],
  disks: [ <omitted for brevity> ],
  <remainder omitted for brevity>,
})

Output of "pulumi preview" after completing a "pulumi up"

Previewing update (staging):
     Type                                     Name                   Plan       Info
     pulumi:pulumi:Stack                      <omitted>                   7 messages
     └─ vsphere:index:Folder                 <omitted>                       
 ~      └─ vsphere:index:VirtualMachine       <omitted>       update     [diff: ~eptRviMode,hvMode]
 ~         ├─ vsphere:index:VirtualMachine    <omitted>       update     [diff: ~eptRviMode,hvMode]
 ~         ├─ vsphere:index:VirtualMachine    <omitted>       update     [diff: ~eptRviMode,hvMode]
 +         └─ pulumi-nodejs:dynamic:Resource  <omitted>-kubeconfig  create     

Running "pulumi up" does not change this output but does cause the VMs to be shutdown, reconfigured and then started again.

Output of pulumi about

CLI          
Version      3.112.0
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME       VERSION
cloudinit  1.4.1
command    0.9.2
nodejs     unknown
random     4.16.0
vsphere    4.9.2

Host     
OS       darwin
Version  14.4.1
Arch     arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v21.7.1'

Current Stack: <omitted>

TYPE                                         URN
pulumi:pulumi:Stack                          urn:pulumi:staging::vsphere::pulumi:pulumi:Stack::<omitted>
pulumi:providers:vsphere                     urn:pulumi:staging::vsphere::pulumi:providers:vsphere::default_4_9_2
pulumi:providers:random                      urn:pulumi:staging::vsphere::pulumi:providers:random::default_4_16_0
random:index/randomPassword:RandomPassword   urn:pulumi:staging::vsphere::random:index/randomPassword:RandomPassword::<omitted>-k8sSecretToken
vsphere:index/folder:Folder                  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder::<omitted>
vsphere:index/virtualMachine:VirtualMachine  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine::<omitted>
pulumi:providers:command                     urn:pulumi:staging::vsphere::pulumi:providers:command::default_0_9_2
command:local:Command                        urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine$command:local:Command::cat-k3s-kubeconf
pulumi:providers:pulumi-nodejs               urn:pulumi:staging::vsphere::pulumi:providers:pulumi-nodejs::default
vsphere:index/virtualMachine:VirtualMachine  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine$vsphere:index/virtualMachine:VirtualMachine::<omitted>
vsphere:index/virtualMachine:VirtualMachine  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine$vsphere:index/virtualMachine:VirtualMachine::<omitted>


OPP TYPE  URN
updating  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine$vsphere:index/virtualMachine:VirtualMachine::<omitted>
updating  urn:pulumi:staging::vsphere::vsphere:index/folder:Folder$vsphere:index/virtualMachine:VirtualMachine$vsphere:index/virtualMachine:VirtualMachine::<omitted>


Backend        
Name           <omitted>
URL            file://~
User           <omitted>
Organizations  
Token type     personal

Dependencies:
NAME               VERSION
@infisical/sdk     2.1.8
@pulumi/command    0.9.2
@pulumi/random     4.16.0
@pulumi/vsphere    4.9.2
@types/node        18.19.21
crypto-js          4.2.0
js-yaml            4.1.0
@pulumi/cloudinit  1.4.1
@pulumi/pulumi     3.108.1
@types/crypto-js   4.2.2
@types/js-yaml     4.0.9

Pulumi locates its logs in /var/folders/_s/7t01k6l903q67w46sjhf5b3w0000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[guineveresaenger]

Hi @cultpony - thank you for letting us know about this.

This may be an instance of pulumi/pulumi-terraform-bridge#1785, and may have a generic fix for that in the future.

[cultpony]

I've also encountered a problem when trying to work around the issue; trying to ignore the properties causes an exception in the code somewhere, breaking the workflow entirely.

[nithom]

Same problem here. But at least with the Python SDK and version 4.10.0 of the vSphere provider, setting pulumi.ResourceOptions(ignore_changes=["eptRviMode", "hvMode"]) for the VM seems to work fine without exceptions so far.

Edit: After doing a pulumi refresh, we run into exceptions too unfortunately, happens during preview:

error: vsphere:index/virtualMachine:VirtualMachine resource 'test-vm' has a problem: expected hv_mode to be one of ["hvAuto" "hvOn" "hvOff"], got . Examine values at 'test-vm.hvMode'.
error: vsphere:index/virtualMachine:VirtualMachine resource 'test-vm' has a problem: expected ept_rvi_mode to be one of ["automatic" "on" "off"], got . Examine values at 'test-vm.eptRviMode'.

Before the refresh, running up created the resource in the first run and displayed no changes for the second run. After refreshing the state, the property seems to go missing. Explicitly setting the two options on the VM resource doesn't help either.

[mikhailshilkov]

@guineveresaenger @t0yv0 Can we rollout PlanResourceChange to this resource to see if that helps?

[t0yv0]

Perhaps we could add an env var to quickly try this without rebuilding. That could help to quickly figure it out if there's a repro at hand.

[mikhailshilkov]

@cultpony Could you please provide a larger self-sufficient repro without omitting most properties? That would help us troubleshoot the issue. Of course, replace any sensitive information.

[cultpony]

A new property has begun also exhibiting this problem: syncTimeWithHost

I took the code I'm using and removed any outside references as much as possible, not interfering with any of the actual properties being set, you'll have to obviously spin up your own vsphere cluster and substitute the IDs involved. Removing "ignoreChanges: [ "syncTimeWithHost" ]" causes that to also always be changed. Ignoring changes on eptRviMode or hvMode causes the entire thing to break entirely (throws an exception).

let metadata = Buffer.from(`# cloud-init\n`, "binary")
    .toString("base64");

let userdata = Buffer.from(`#cloud-config\n`, "binary")
    .toString("base64");

let template = await vsphere.getVirtualMachine({
    name: "TEMPLATE_NAME",
    datacenterId: "DC_ID",
});

new vsphere.VirtualMachine(`test-vm`, {
    name: "test-vm",
    clone: {
        templateUuid: template.uuid,
    },
    extraConfig: {
        "guestinfo.userdata": userdata,
        "guestinfo.userdata.encoding": Encoding.Base64,
        "guestinfo.metadata": metadata,
        "guestinfo.metadata.encoding": Encoding.Base64,
    },
    networkInterfaces: [{
        networkId: "NETWORK_ID",
        adapterType: template.networkInterfaces[0].adapterType,
    }],
    disks: [{
        label: template.disks[0].label,
        size: template.disks[0].size,
        unitNumber: 0,
        datastoreId: "DATASTORE_ID",
    },{
        label: template.disks[1].label,
        size: template.disks[1].size,
        unitNumber: 1,
        datastoreId: "DATASTORE_ID",
    }],
    numCpus: 4,
    memory: 4096,
    cpuReservation: 1000,
    memoryReservation: 1024,
    datastoreId: "DATASTORE_ID",
    firmware: template.firmware,
    resourcePoolId: "RESOURCE_POOL_ID",
    folder: "/"
    guestId: template.guestId,
    waitForGuestIpTimeout: 10,
}, {
    ignoreChanges: [ "syncTimeWithHost", ],
    parent: parent,
});

[nithom]

Regarding the exceptions for eptRviMode and hvMode, I found that the outputs in the stack state are just empty strings, no matter what you specify as input. When doing a pulumi refresh, the inputs are then set to empty strings as well. With that, the provider throws exceptions on later calls. Manually changing state (with pulumi stack export/import) such that inputs and outputs both contain the default values makes everything work normally again. But of course, that's not an ideal workaround.

[guineveresaenger]

hi @nithom and @cultpony -

per @mikhailshilkov 's suggestion, I've rolled an alpha release of the vsphere provider that includes the ability to add an environment variable that will allow the provider to use a new diffing strategy.

Could you run your local program using v4.11.0-alpha.1 and with

$ export PULUMI_ENABLE_PLAN_RESOURCE_CHANGE=true

set in the environment, and let us know if this issue persists? Thank you so much! 🙏

[nithom]

Hi @guineveresaenger , thanks for providing an alpha release. Unfortunately, the behavior stays the same with the new version and the environment variable set. After doing a pulumi refresh, the inputs and outputs of the two parameters still become empty strings in the stack state, and later pulumi up calls output a diff for the parameters.