vnc - unexpected new value

[karamelichkin]

vm description:

resource "libvirt_domain" "example" {
  provider = libvirt.kvm3
  
  name        = "example-vm"
  memory      = 1024
  memory_unit = "MiB"
  vcpu        = 1
  
  type = "kvm"

  os = {
    type         = "hvm"
    type_arch    = "x86_64"
    type_machine = "q35"
    boot = {
      devices = ["hd"]
    }
  }

  features = {
    acpi = true
  }

  devices = {
      graphics = [
    {
      type = "vnc"
      vnc = {
        autoport = true
        listen = "0.0.0.0"
        passwd = "password"
      }
    }
    ]

    disks = [
      {
        type = "volume"
        source = {
          volume = {
            pool   = libvirt_pool.st0_pool_kvm3.name
            volume = libvirt_volume.st0_volume_kvm3.name
          }
        }
        target = {
          dev = "vda"
          bus = "virtio"
        }
        driver = {
          name = "qemu"
          type = "raw"
        }
      }
    ]
    
    interfaces = [
      {
        type = "bridge"
        model = {
          type = "virtio"
        }
        source = {
          bridge = {
            bridge = "br0" 
          }
        }
        mac = {
          address = "52:54:00:12:34:56"
        }
      }
    ]

    consoles = [
      {
        type        = "pty"
        target_port = "0"
        target_type = "serial"
      }
    ]
  }
  running = true
}

vm runs successfully and vnc works fine with password i set

but terraform output

libvirt_domain.example: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to libvirt_domain.example, provider "provider[\"registry.terraform.io/dmacvicar/libvirt\"].kvm3" produced an unexpected new value:
│ .devices.graphics[0].vnc.passwd: was cty.StringVal("password"), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[sampathP]

Hi @karamelichkin ,
Thanks for opening this issue.
I'm hitting the exact same error pattern right now with libvirt_domain network interfaces in provider version 0.9.1:
I get the following errors after terrafrom apply. Regardless of the error,
VM works fine and both networks are connected properly.

╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to libvirt_domain.test_vm, provider
│ "provider[\"registry.terraform.io/dmacvicar/libvirt\"].samfw01" produced an
│ unexpected new value: .devices.interfaces[0].source.network: was
│ cty.ObjectVal(map[string]cty.Value{"bridge":cty.NullVal(cty.String),
│ "network":cty.StringVal("net_br11"), "port_group":cty.NullVal(cty.String),
│ "port_id":cty.NullVal(cty.String)}), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to libvirt_domain.test_vm, provider
│ "provider[\"registry.terraform.io/dmacvicar/libvirt\"].samfw01" produced an
│ unexpected new value: .devices.interfaces[1].source.network: was
│ cty.ObjectVal(map[string]cty.Value{"bridge":cty.NullVal(cty.String),
│ "network":cty.StringVal("net_br12"), "port_group":cty.NullVal(cty.String),
│ "port_id":cty.NullVal(cty.String)}), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵

Here is my VM description:

resource "libvirt_domain" "test_vm" {
  name   = var.test_vm.hostname
  memory = 1048576
  vcpu   = 1
  provider = libvirt.samfw01
  type   = "kvm"
  os = {
    type         = "hvm"
    type_arch    = "x86_64"
    type_machine = "q35"
    boot = {
      dev = "hd"
    }
  }
  cpu = {
    mode = "host-passthrough"
  }
  # Attached devices
  devices = {
    disks = [
      {
        type = "file"
        device = "disk"
        source = {
            volume = {
                pool = libvirt_volume.test_vm_disk.pool
                volume = libvirt_volume.test_vm_disk.name
            }
        }               
        target = {
            dev = "vda"
            bus = "virtio"
        }
        boot = {
            order = 1
        }
        driver = {
            type = "qcow2"
        }
      },
      {
        type = "block"
        device = "cdrom"
        source = {
            volume = {
                pool = libvirt_volume.test_vm_cloudinit.pool
                volume = libvirt_volume.test_vm_cloudinit.name
            }
        }
        target = {
            dev = "sda"
            bus = "sata"
            }
        }
    ]
    interfaces = [
        {
            type = "network"
            model   = {
                type = "virtio"
            }
            source  = {
                network = {
                    network = "net_br11"
                } 
            }
            # wait_for_ip = {
            #     timeout = 300 # seconds
            #     source  = "any"
            # }
      },
        {
            type = "network"
            model   = {
                type = "virtio"
            }
            source  = {
                network = {
                    network = "net_br12"
                }
            }
        }
        ]
    graphics = [
        {
            vnc = {
                auto_port = true
                listen = "0.0.0.0"
            }
        }
        ]
    serials = [
        {
            type = "pty"
            target = {
                type = "isa-serial"
                # port = "0"
            }
        }
        ]
    consoles = [
        {
            type = "pty"
            target = {
                type = "serial"
                # port = "0"
            }
        }
        ]
    channels = [
        {
          type = "unix"
          source = {
            unix = {
              mode = "bind"
            }
          }
          target = {
            virt_io = {
              name = "org.qemu.guest_agent.0"
              state = "disconnected"
            }
          }
        }
        ]
    }
  running   = true
  autostart = true
}

Here is the state of the domain looks like after apply.

❯ terraform state show 'libvirt_domain.test_vm'
# libvirt_domain.test_vm: (tainted)
resource "libvirt_domain" "test_vm" {
    autostart = true
    cpu       = {
        mode = "host-passthrough"
    }
    devices   = {
        channels   = [
            {
                source = {
                    unix = {
                        mode = "bind"
                    }
                }
                target = {
                    virt_io = {
                        name  = "org.qemu.guest_agent.0"
                        state = "disconnected"
                    }
                }
            },
        ]
        consoles   = [
            {
                target = {
                    type = "serial"
                }
            },
        ]
        disks      = [
            {
                boot   = {
                    order = 1
                }
                device = "disk"
                driver = {
                    type = "qcow2"
                }
                source = {
                    volume = {
                        pool   = "default"
                        volume = "test_vm_disk.qcow2"
                    }
                }
                target = {
                    bus = "virtio"
                    dev = "vda"
                }
            },
            {
                device = "cdrom"
                source = {
                    volume = {
                        pool   = "default"
                        volume = "test_vm_seed.iso"
                    }
                }
                target = {
                    bus = "sata"
                    dev = "sda"
                }
            },
        ]
        graphics   = [
            {
                vnc = {
                    auto_port = true
                    listen    = "0.0.0.0"
                }
            },
        ]
        interfaces = [
            {
                model  = {
                    type = "virtio"
                }
                source = {}     
            },
            {
                model  = {
                    type = "virtio"
                }
                source = {}
            },
        ]
        serials    = [
            {
                target = {
                    type = "isa-serial"
                }
            },
        ]
    }
    id        = 1
    memory    = 1048576
    name      = "test-vm"
    os        = {
        type         = "hvm"
        type_arch    = "x86_64"
        type_machine = "q35"
    }
    running   = true
    type      = "kvm"
    uuid      = "feb43460-4ad5-4fbc-9244-4876ce256695"
    vcpu      = 1
}

And here is the actual domain looks on host:

sampath@samnetfw01:~$ sudo virsh dumpxml test-vm 
<domain type='kvm' id='1'>
  <name>test-vm</name>
  <uuid>feb43460-4ad5-4fbc-9244-4876ce256695</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
  </os>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source pool='default' volume='test_vm_disk.qcow2' index='2'/>
      <backingStore type='file' index='3'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/ubuntu-base.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='volume' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source pool='default' volume='test_vm_seed.iso' index='1'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:98:f0:5a'/>
      <source network='net_br11' portid='14e8b052-4181-47dd-ae44-086a3e5d8d2a' bridge='br11'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:f7:09:d5'/>
      <source network='net_br12' portid='a22b6564-8fd1-44c9-a47a-47a4ce1a1a10' bridge='br12'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/run/libvirt/qemu/channel/1-test-vm/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

IMHO, the issue is caused by the provider failing to correctly parse the domain XML during the refresh/read phase after apply. Even though libvirt returns perfectly valid XML containing the expected network interfaces:

    <interface type='bridge'>
      <mac address='52:54:00:98:f0:5a'/>
      <source network='net_br11' portid='14e8b052-4181-47dd-ae44-086a3e5d8d2a' bridge='br11'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:f7:09:d5'/>
      <source network='net_br12' portid='a22b6564-8fd1-44c9-a47a-47a4ce1a1a10' bridge='br12'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>

However, provider fail to parse it correctly.

        interfaces = [
            {
                model  = {
                    type = "virtio"
                }
                source = {}     
            },
            {
                model  = {
                    type = "virtio"
                }
                source = {}
            },
        ]