Lost ALL VM from services

[Amdusis]

Hello,

In our MIQ infrastructure VMs was lost from services but allive on vCenter. We have no idea why it happened.

In logs we see "MiqAeEvent.build_evm_event" for all orphand VMs in the same time.

WARN -- evm: MIQ(MiqAlert.evaluate_alerts) [vm_create] Target: ManageIQ::Providers::Vmware::InfraManager::Vm Name: [*****], Id: [1000000036541] Result: No enabled alerts are assigned to target! Nothing to do.

INFO -- evm: MIQ(MiqAlert.evaluate_alerts) [vm_create] Target: ManageIQ::Providers::Vmware::InfraManager::Vm Name: [*****], Id: [1000000036541]

INFO -- policy: MIQ(policy-enforce_policy): Event: [vm_create], To: [*******]

INFO -- automation: MiqAeEvent.build_evm_event >> event=<"vm_create"> inputs=<{:vm=>#<ManageIQ::Providers::Vmware::InfraManager::Vm id: 1000000036541, vendor: "vmware", format: nil, version: nil, name: "*******", description: "Owner: nEmail: ...", location: "bf5c1469-8632-994e-e6e8-88e9a4147000/", config_xml: nil, autostart: nil, host_id: 1000000000269, last_sync_on: nil, created_on: "2025-11-18 08:24:11.245767000 +0000", updated_on: "2025-11-18 08:24:11.245767000 +0000", storage_id: 1000000000001, guid: "82189a1a-53d4-451c-afe5-5a796d4f4880", ems_id: 1000000000003, last_scan_on: nil, last_scan_attempt_on: nil, uid_ems: "4205a578-5133-dfd9-a844-8aa672cbf927", retires_on: nil, retired: nil, boot_time: "2025-11-16 02:46:53.998340000 +0000", tools_status: "toolsOk", standby_action: "checkpoint", power_state: "on", state_changed_on: "2025-11-18 08:24:11.240019000 +0000", previous_state: nil, connection_state: "connected", last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: 0, memory_reserve_expand: false, memory_limit: -1, memory_shares: 40960, memory_shares_level: "normal", cpu_reserve: 0, cpu_reserve_expand: false, cpu_limit: -1, cpu_shares: 4000, cpu_shares_level: "normal", cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, miq_group_id: 1000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Vmware::InfraManager::Vm", ems_ref: "vm-2657135", ems_cluster_id: 1000000000003, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: nil, availability_zone_id: nil, cloud: false, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "poweredOn", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 1000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: false, cpu_hot_remove_enabled: false, memory_hot_add_enabled: false, memory_hot_add_limit: 4096, memory_hot_add_increment: 0, hostname: nil, ems_ref_type: "VirtualMachine", restart_needed: nil, ancestry: nil, placement_group_id: nil>, :host=>#<ManageIQ::Providers::Vmware::InfraManager::HostEsx id: 1000000000269, name: "*******", hostname: "********", ipaddress: "******", vmm_vendor: "vmware", vmm_version: "8.0.3", vmm_product: "ESXi", vmm_buildnumber: "24859861", created_on: "2024-10-18 09:25:07.371837000 +0000", updated_on: "2025-09-13 11:17:50.823450000 +0000", guid: "34d8b08b-1935-47d3-ba9c-9931123069d6", ems_id: 1000000000003, user_assigned_os: nil, power_state: "on", smart: 1, settings: {}, last_perf_capture_on: nil, uid_ems: "********", connection_state: "connected", ssh_permit_root_login: nil, admin_disabled: true, service_tag: "*****", asset_tag: nil, ipmi_address: nil, mac_address: nil, type: "ManageIQ::Providers::Vmware::InfraManager::HostEsx", failover: nil, ems_ref: "host-1803363", hyperthreading: true, ems_cluster_id: 1000000000003, next_available_vnc_port: nil, hypervisor_hostname: nil, availability_zone_id: nil, maintenance: false, maintenance_reason: nil, physical_server_id: nil, ems_ref_type: "HostSystem">, "MiqEvent::miq_event"=>1000102667621, :miq_event_id=>1000102667621, "EventStream::event_stream"=>1000102667621, :event_stream_id=>1000102667621}>

Seems like manageiq recreated VMs.

Can you help with this problem? Very critical for us.

Thnaks!

[Fryguy]

That's creating the new VMs, but do you see it removing the old VMs?

[Amdusis]

@agrare
No VMs in archive.

from rails:
No objects in vmdb

irb(main):005:0> vm = $evm.vmdb('vm').find('1000000036554')
(irb):5:in <main>': undefined methodvmdb' for nil (NoMethodError)
        from <internal:kernel>:187:in loop'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:74:instart'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:19:in start'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:106:inperform'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/command.rb:28:in run'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/invocation.rb:127:ininvoke_command'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor.rb:538:in dispatch'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command/base.rb:87:inperform'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command.rb:48:in invoke'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands.rb:18:in<main>'
        from /usr/share/ruby/bundled_gems.rb:69:in require'
        from /usr/share/ruby/bundled_gems.rb:69:inblock (2 levels) in replace_require'
        from /opt/manageiq/manageiq-gemset/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in require'
        from bin/rails:4:in<main>'

In miq:

irb(main):003:0> vm = Vm.find_by(name: "****redis08")
=>

#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007fc22d04a000
...
irb(main):004:0> vm
=>

#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007fc22d04a000
 id: 1000000036554,
 vendor: "vmware",
 format: nil,
 version: nil,
 name: "redis08",
 description: "\n\nMIQ GUID=06ddb69d-fa3b-4a12-99eb-a348cc711485",
 location: "db851f69-aa44-bdd5-2cfe-1402ece3bf38/redis08.vmx",
 config_xml: nil,
 autostart: nil,
 host_id: 1000000000038,
 last_sync_on: nil,
 created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00,
 updated_on: Sun, 23 Nov 2025 00:19:04.363323000 UTC +00:00,
 storage_id: 1000000000752,
 guid: "97953454-6ac7-4db7-854b-2d294f88ae0f",
 ems_id: 1000000000003,
 last_scan_on: nil,
 last_scan_attempt_on: nil,
 uid_ems: "4205928a-5b51-255b-1d87-4d4cd6813d99",
 retires_on: nil,
 retired: nil,
 boot_time: Sun, 23 Nov 2025 00:18:43.000000000 UTC +00:00,
 tools_status: "toolsOk",
 standby_action: "checkpoint",
 power_state: "on",
 state_changed_on: Tue, 18 Nov 2025 08:33:49.566513000 UTC +00:00,
 previous_state: nil,
 connection_state: "connected",
 last_perf_capture_on: nil,
 registered: nil,
 busy: nil,
 smart: nil,
 memory_reserve: 0,
 memory_reserve_expand: false,
 memory_limit: -1,
 memory_shares: 40960,
 memory_shares_level: "normal",
 cpu_reserve: 0,
 cpu_reserve_expand: false,
 cpu_limit: -1,
 cpu_shares: 4000,
 cpu_shares_level: "normal",
 cpu_affinity: nil,
 ems_created_on: nil,
 template: false,
 evm_owner_id: nil,
 miq_group_id: 1000000000001,
 linked_clone: nil,
 fault_tolerance: nil,
 type: "ManageIQ::Providers::Vmware::InfraManager::Vm",
 ems_ref: "vm-2657128",
 ems_cluster_id: 1000000000226,
 retirement_warn: nil,
 retirement_last_warn: nil,
 vnc_port: nil,
 flavor_id: nil,
 availability_zone_id: nil,
 cloud: false,
 retirement_state: nil,
 cloud_network_id: nil,
 cloud_subnet_id: nil,
 cloud_tenant_id: nil,
 raw_power_state: "poweredOn",
 publicly_available: nil,
 orchestration_stack_id: nil,
 retirement_requester: nil,
 tenant_id: 1000000000001,
 resource_group_id: nil,
 deprecated: nil,
 storage_profile_id: nil,
 cpu_hot_add_enabled: false,
 cpu_hot_remove_enabled: false,
 memory_hot_add_enabled: false,
 memory_hot_add_limit: 4096,
 memory_hot_add_increment: 0,
 hostname: nil,
 ems_ref_type: "VirtualMachine",
 restart_needed: nil,
 ancestry: nil,
 placement_group_id: nil>

created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00, - time when all vms lost

[agrare]

Don't use find_by it will return only a single record, try using Vm.where(:name => "....")

[Amdusis]

@agrare
Just one record

irb(main):003:0> Vm.where(:name => "redis08")
=>
[#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007f55f4dc9438
  id: 1000000036554,
  vendor: "vmware",
  format: nil,
  version: nil,
  name: "redis08",
  description: "\nMIQ GUID=06ddb69d-fa3b-4a12-99eb-a348cc711485",
  location: "db851f69-aa44-bdd5-2cfe-1402ece3bf38/redis08.vmx",
  config_xml: nil,
  autostart: nil,
  host_id: 1000000000038,
  last_sync_on: nil,
  created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00,
  updated_on: Sun, 23 Nov 2025 00:19:04.363323000 UTC +00:00,
  storage_id: 1000000000752,
  guid: "97953454-6ac7-4db7-854b-2d294f88ae0f",
  ems_id: 1000000000003,
  last_scan_on: nil,
  last_scan_attempt_on: nil,
  uid_ems: "4205928a-5b51-255b-1d87-4d4cd6813d99",
  retires_on: nil,
  retired: nil,
  boot_time: Sun, 23 Nov 2025 00:18:43.000000000 UTC +00:00,
  tools_status: "toolsOk",
  standby_action: "checkpoint",
  power_state: "on",
  state_changed_on: Tue, 18 Nov 2025 08:33:49.566513000 UTC +00:00,
  previous_state: nil,
  connection_state: "connected",
  last_perf_capture_on: nil,
  registered: nil,
  busy: nil,
  smart: nil,
  memory_reserve: 0,
  memory_reserve_expand: false,
  memory_limit: -1,
  memory_shares: 40960,
  memory_shares_level: "normal",
  cpu_reserve: 0,
  cpu_reserve_expand: false,
  cpu_limit: -1,
  cpu_shares: 4000,
  cpu_shares_level: "normal",
  cpu_affinity: nil,
  ems_created_on: nil,
  template: false,
  evm_owner_id: nil,
  miq_group_id: 1000000000001,
  linked_clone: nil,
  fault_tolerance: nil,
  type: "ManageIQ::Providers::Vmware::InfraManager::Vm",
  ems_ref: "vm-2657128",
  ems_cluster_id: 1000000000226,
  retirement_warn: nil,
  retirement_last_warn: nil,
  vnc_port: nil,
  flavor_id: nil,
  availability_zone_id: nil,
  cloud: false,
  retirement_state: nil,
  cloud_network_id: nil,
  cloud_subnet_id: nil,
  cloud_tenant_id: nil,
  raw_power_state: "poweredOn",
  publicly_available: nil,
  orchestration_stack_id: nil,
  retirement_requester: nil,
  tenant_id: 1000000000001,
  resource_group_id: nil,
  deprecated: nil,
  storage_profile_id: nil,
  cpu_hot_add_enabled: false,
  cpu_hot_remove_enabled: false,
  memory_hot_add_enabled: false,
  memory_hot_add_limit: 4096,
  memory_hot_add_increment: 0,
  hostname: nil,
  ems_ref_type: "VirtualMachine",
  restart_needed: nil,
  ancestry: nil,
  placement_group_id: nil>]

[agrare]

Okay thanks, I'd be surprised if an error with the Refresher caused this because we don't actually delete VMs from the database we only archive them. Not ruling anything fully out though, I'll have to see the logs from the error.

[Amdusis]

Thanks!
If you need any additional logs or information we are redy to help)

[nikobo]

Same issue since November 14. vCenter sync crash and vcenter inventory on manageiq is reseted.
The strange think is that we have a another manageiq instance targeting the same vcenter with the same issue..

[nikobo]

Hello @agrare, have you find the patch to debug more ?

[agrare]

Hey sorry I was off for the holidays I'll get on this patch this week

[Amdusis]

Hello!

Any updates?)

[agrare]

Hey @Amdusis sorry for the delay.

I've created a script to record the XML interactions with your vCenter using the VCR gem.

Please download [vmware_refresh.rb] and place it in the root of a manageiq development environment. It cannot be a production appliance because we have to have the vcr and webmock gems installed and you don't want to go breaking your production gemset.

$ rails r vmware_refresh_vcr.rb -- --ems-id=EMS_ID

Where EMS_ID is the id of your VMware system, you can find the ID in the URL:

This will start up a VMware refresh worker. You'll want to tail the logs to see when the refresh finishes and then continue to receive updates. Once the updates stop coming in ctrl-c the script. This will instruct the worker to exit and write out the XML interactions to a file ./spec/vcr_cassettes/vmware-refresh.yml.

The username/password will be in this file, so before sending it make sure you change at least the <password> attribute:

- request:
    method: post
    uri: https://vc.example.com/sdk
    body:
      encoding: UTF-8
      string: <env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><env:Body><Login xmlns="urn:vim25"><_this
        type="SessionManager">SessionManager</_this><userName>USERNAME</userName><password>PASSWORD</password></Login></env:Body></env:Envelope>

You can also change the hostname/uri to something else if you'd like but there will be many more instances of that (sed is your friend here).

Once you've done this you can email the results to myself mailto:adam@grare.com or create a secret gist and send me that link.

Once I have that I'll be able to run the exact refresh that you are seeing on my system and investigate the results.

Look forward to hearing from you

[Amdusis]

@agrare
Big thanks!

@nikobo
FYI