LXD upgrade removes instance socket from system container instances

[jocado]

Please confirm

• I have searched existing issues to check if an issue already exists for the bug I encountered.

Distribution

Ubuntu

Distribution version

Noble

Output of snap list --all lxd core20 core22 core24 snapd

Name    Version      Rev    Tracking       Publisher   Notes
core24  20260317     1587   latest/stable  canonical✓  base
lxd     6.5-ccdfb39  36020  latest/stable  canonical✓  disabled
lxd     6.8-da9ddd7  39231  latest/stable  canonical✓  -
snapd   2.75.2       26865  latest/stable  canonical✓  snapd

System info

api_extensions:

• storage_zfs_remove_snapshots
• container_host_shutdown_timeout
• container_stop_priority
• container_syscall_filtering
• auth_pki
• container_last_used_at
• etag
• patch
• usb_devices
• https_allowed_credentials
• image_compression_algorithm
• directory_manipulation
• container_cpu_time
• storage_zfs_use_refquota
• storage_lvm_mount_options
• network
• profile_usedby
• container_push
• container_exec_recording
• certificate_update
• container_exec_signal_handling
• gpu_devices
• container_image_properties
• migration_progress
• id_map
• network_firewall_filtering
• network_routes
• storage
• file_delete
• file_append
• network_dhcp_expiry
• storage_lvm_vg_rename
• storage_lvm_thinpool_rename
• network_vlan
• image_create_aliases
• container_stateless_copy
• container_only_migration
• storage_zfs_clone_copy
• unix_device_rename
• storage_lvm_use_thinpool
• storage_rsync_bwlimit
• network_vxlan_interface
• storage_btrfs_mount_options
• entity_description
• image_force_refresh
• storage_lvm_lv_resizing
• id_map_base
• file_symlinks
• container_push_target
• network_vlan_physical
• storage_images_delete
• container_edit_metadata
• container_snapshot_stateful_migration
• storage_driver_ceph
• storage_ceph_user_name
• resource_limits
• storage_volatile_initial_source
• storage_ceph_force_osd_reuse
• storage_block_filesystem_btrfs
• resources
• kernel_limits
• storage_api_volume_rename
• network_sriov
• console
• restrict_devlxd
• migration_pre_copy
• infiniband
• devlxd_events
• proxy
• network_dhcp_gateway
• file_get_symlink
• network_leases
• unix_device_hotplug
• storage_api_local_volume_handling
• operation_description
• clustering
• event_lifecycle
• storage_api_remote_volume_handling
• nvidia_runtime
• container_mount_propagation
• container_backup
• devlxd_images
• container_local_cross_pool_handling
• proxy_unix
• proxy_udp
• clustering_join
• proxy_tcp_udp_multi_port_handling
• network_state
• proxy_unix_dac_properties
• container_protection_delete
• unix_priv_drop
• pprof_http
• proxy_haproxy_protocol
• network_hwaddr
• proxy_nat
• network_nat_order
• container_full
• backup_compression
• nvidia_runtime_config
• storage_api_volume_snapshots
• storage_unmapped
• projects
• network_vxlan_ttl
• container_incremental_copy
• usb_optional_vendorid
• snapshot_scheduling
• snapshot_schedule_aliases
• container_copy_project
• clustering_server_address
• clustering_image_replication
• container_protection_shift
• snapshot_expiry
• container_backup_override_pool
• snapshot_expiry_creation
• network_leases_location
• resources_cpu_socket
• resources_gpu
• resources_numa
• kernel_features
• id_map_current
• event_location
• storage_api_remote_volume_snapshots
• network_nat_address
• container_nic_routes
• cluster_internal_copy
• seccomp_notify
• lxc_features
• container_nic_ipvlan
• network_vlan_sriov
• storage_cephfs
• container_nic_ipfilter
• resources_v2
• container_exec_user_group_cwd
• container_syscall_intercept
• container_disk_shift
• storage_shifted
• resources_infiniband
• daemon_storage
• instances
• image_types
• resources_disk_sata
• clustering_roles
• images_expiry
• resources_network_firmware
• backup_compression_algorithm
• ceph_data_pool_name
• container_syscall_intercept_mount
• compression_squashfs
• container_raw_mount
• container_nic_routed
• container_syscall_intercept_mount_fuse
• container_disk_ceph
• virtual-machines
• image_profiles
• clustering_architecture
• resources_disk_id
• storage_lvm_stripes
• vm_boot_priority
• unix_hotplug_devices
• api_filtering
• instance_nic_network
• clustering_sizing
• firewall_driver
• projects_limits
• container_syscall_intercept_hugetlbfs
• limits_hugepages
• container_nic_routed_gateway
• projects_restrictions
• custom_volume_snapshot_expiry
• volume_snapshot_scheduling
• trust_ca_certificates
• snapshot_disk_usage
• clustering_edit_roles
• container_nic_routed_host_address
• container_nic_ipvlan_gateway
• resources_usb_pci
• resources_cpu_threads_numa
• resources_cpu_core_die
• api_os
• container_nic_routed_host_table
• container_nic_ipvlan_host_table
• container_nic_ipvlan_mode
• resources_system
• images_push_relay
• network_dns_search
• container_nic_routed_limits
• instance_nic_bridged_vlan
• network_state_bond_bridge
• usedby_consistency
• custom_block_volumes
• clustering_failure_domains
• resources_gpu_mdev
• console_vga_type
• projects_limits_disk
• network_type_macvlan
• network_type_sriov
• container_syscall_intercept_bpf_devices
• network_type_ovn
• projects_networks
• projects_networks_restricted_uplinks
• custom_volume_backup
• backup_override_name
• storage_rsync_compression
• network_type_physical
• network_ovn_external_subnets
• network_ovn_nat
• network_ovn_external_routes_remove
• tpm_device_type
• storage_zfs_clone_copy_rebase
• gpu_mdev
• resources_pci_iommu
• resources_network_usb
• resources_disk_address
• network_physical_ovn_ingress_mode
• network_ovn_dhcp
• network_physical_routes_anycast
• projects_limits_instances
• network_state_vlan
• instance_nic_bridged_port_isolation
• instance_bulk_state_change
• network_gvrp
• instance_pool_move
• gpu_sriov
• pci_device_type
• storage_volume_state
• network_acl
• migration_stateful
• disk_state_quota
• storage_ceph_features
• projects_compression
• projects_images_remote_cache_expiry
• certificate_project
• network_ovn_acl
• projects_images_auto_update
• projects_restricted_cluster_target
• images_default_architecture
• network_ovn_acl_defaults
• gpu_mig
• project_usage
• network_bridge_acl
• warnings
• projects_restricted_backups_and_snapshots
• clustering_join_token
• clustering_description
• server_trusted_proxy
• clustering_update_cert
• storage_api_project
• server_instance_driver_operational
• server_supported_storage_drivers
• event_lifecycle_requestor_address
• resources_gpu_usb
• clustering_evacuation
• network_ovn_nat_address
• network_bgp
• network_forward
• custom_volume_refresh
• network_counters_errors_dropped
• metrics
• image_source_project
• clustering_config
• network_peer
• linux_sysctl
• network_dns
• ovn_nic_acceleration
• certificate_self_renewal
• instance_project_move
• storage_volume_project_move
• cloud_init
• network_dns_nat
• database_leader
• instance_all_projects
• clustering_groups
• ceph_rbd_du
• instance_get_full
• qemu_metrics
• gpu_mig_uuid
• event_project
• clustering_evacuation_live
• instance_allow_inconsistent_copy
• network_state_ovn
• storage_volume_api_filtering
• image_restrictions
• storage_zfs_export
• network_dns_records
• storage_zfs_reserve_space
• network_acl_log
• storage_zfs_blocksize
• metrics_cpu_seconds
• instance_snapshot_never
• certificate_token
• instance_nic_routed_neighbor_probe
• event_hub
• agent_nic_config
• projects_restricted_intercept
• metrics_authentication
• images_target_project
• cluster_migration_inconsistent_copy
• cluster_ovn_chassis
• container_syscall_intercept_sched_setscheduler
• storage_lvm_thinpool_metadata_size
• storage_volume_state_total
• instance_file_head
• instances_nic_host_name
• image_copy_profile
• container_syscall_intercept_sysinfo
• clustering_evacuation_mode
• resources_pci_vpd
• qemu_raw_conf
• storage_cephfs_fscache
• network_load_balancer
• vsock_api
• instance_ready_state
• network_bgp_holdtime
• storage_volumes_all_projects
• metrics_memory_oom_total
• storage_buckets
• storage_buckets_create_credentials
• metrics_cpu_effective_total
• projects_networks_restricted_access
• loki
• acme
• internal_metrics
• cluster_join_token_expiry
• remote_token_expiry
• init_preseed
• storage_volumes_created_at
• cpu_hotplug
• projects_networks_zones
• network_txqueuelen
• cluster_member_state
• storage_pool_source_wipe
• zfs_block_mode
• instance_generation_id
• disk_io_cache
• amd_sev
• storage_pool_loop_resize
• migration_vm_live
• ovn_nic_nesting
• oidc
• network_ovn_l3only
• ovn_nic_acceleration_vdpa
• cluster_healing
• instances_state_total
• auth_user
• security_csm
• instances_rebuild
• numa_cpu_placement
• custom_volume_iso
• network_allocations
• storage_api_remote_volume_snapshot_copy
• zfs_delegate
• operations_get_query_all_projects
• metadata_configuration
• syslog_socket
• event_lifecycle_name_and_project
• instances_nic_limits_priority
• disk_initial_volume_configuration
• operation_wait
• cluster_internal_custom_volume_copy
• disk_io_bus
• storage_cephfs_create_missing
• instance_move_config
• ovn_ssl_config
• init_preseed_storage_volumes
• metrics_instances_count
• server_instance_type_info
• resources_disk_mounted
• server_version_lts
• oidc_groups_claim
• loki_config_instance
• storage_volatile_uuid
• import_instance_devices
• instances_uefi_vars
• instances_migration_stateful
• container_syscall_filtering_allow_deny_syntax
• access_management
• vm_disk_io_limits
• storage_volumes_all
• instances_files_modify_permissions
• image_restriction_nesting
• container_syscall_intercept_finit_module
• device_usb_serial
• network_allocate_external_ips
• explicit_trust_token
• shared_custom_block_volumes
• instance_import_conversion
• instance_create_start
• instance_protection_start
• devlxd_images_vm
• disk_io_bus_virtio_blk
• metrics_api_requests
• projects_limits_disk_pool
• ubuntu_pro_guest_attach
• metadata_configuration_entity_types
• access_management_tls
• network_allocations_ovn_uplink
• network_ovn_uplink_vlan
• state_logical_cpus
• vm_limits_cpu_pin_strategy
• gpu_cdi
• images_all_projects
• metadata_configuration_scope
• unix_device_hotplug_ownership_inherit
• unix_device_hotplug_subsystem_device_option
• storage_ceph_osd_pool_size
• network_get_target
• network_zones_all_projects
• vm_root_volume_attachment
• projects_limits_uplink_ips
• entities_with_entitlements
• profiles_all_projects
• storage_driver_powerflex
• storage_driver_pure
• cloud_init_ssh_keys
• oidc_scopes
• project_default_network_and_storage
• client_cert_presence
• clustering_groups_used_by
• container_bpf_delegation
• override_snapshot_profiles_on_copy
• resources_device_fs_uuid
• backup_metadata_version
• storage_buckets_all_projects
• network_acls_all_projects
• networks_all_projects
• clustering_restore_skip_mode
• disk_io_threads_virtiofsd
• oidc_client_secret
• pci_hotplug
• device_patch_removal
• daemon_storage_per_project
• ovn_internal_load_balancer
• auth_bearer_devlxd
• devlxd_volume_management
• storage_driver_alletra
• resources_disk_used_by
• ovn_dhcp_ranges
• operation_requestor
• import_custom_volume_tar
• projects_force_delete
• auth_oidc_sessions
• instance_snapshots_multi_volume
• vm_persistent_bus
• instance_placement_groups
• ovn_nic_acceleration_parent
• storage_and_profile_operations
• storage_source_recover
• instance_force_delete
• operation_metadata_entity_url
• instance_boot_mode
• auth_bearer
• vm_limits_max_bus_ports
• instances_state_selective_recursion
• project_delete_operation
• gpu_cdi_amd
• instance_refresh_config
• clustering_control_plane
• storage_remote_drop_source
• storage_ceph_use_rbd_defaults
• bulk_operations
• ovn_dynamic_northbound_connection
• storage_zfs_promote
• storage_and_network_operations
• gpu_cdi_hotplug
• image_extended_metadata
• cluster_links
• replicators
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
• tls
• bearer
  client_certificate: false
  config:
  images.auto_update_interval: "0"
  volatile.uuid: 019dd085-d20a-7203-8558-4e7bf7b0548a
  auth_user_name: root
  auth_user_method: unix
  environment:
  addresses: []
  architectures:
  • x86_64
  • i686
    backup_metadata_version_range:
  • 1
  • 2
    certificate: |
    -----BEGIN CERTIFICATE-----
    MIICDjCCAZSgAwIBAgIRAPzLTHtqXRRsycFHy4U4SCEwCgYIKoZIzj0EAwMwMjEM
    MAoGA1UEChMDTFhEMSIwIAYDVQQDDBlyb290QGluc3RhbmNlLXNvY2tldC10ZXN0
    MB4XDTI2MDQyNzE5NTU1MloXDTM2MDQyNDE5NTU1MlowMjEMMAoGA1UEChMDTFhE
    MSIwIAYDVQQDDBlyb290QGluc3RhbmNlLXNvY2tldC10ZXN0MHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAE9z2GjgPzrZvihf+FnbvV868j4pZNZJjvSIkDwqywpPNjd1i9
    +J9Dl8wO365YlNqZvyoE6oB3IeNhv5h5VrWVTBQKFW8V4FKblQpqVkPK/8dVyVsv
    7102q+FCs81oJCmAo24wbDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB
    BQUHAwEwDAYDVR0TAQH/BAIwADA3BgNVHREEMDAughRpbnN0YW5jZS1zb2NrZXQt
    dGVzdIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNoADBlAjB7
    DLE3TENB0jAM01tfxOH/qsAgtd14X35Y6KYTnI3OGYHEhbppR4i44PKFBxNAGYEC
    MQCRA53FsMVYgGBQNtLmxBdcTC2Dc1YFCc35pzY0kOEPzw1ugUYQkmppdhaKV7gp
    PoQ=
    -----END CERTIFICATE-----
    certificate_fingerprint: 65c9c7f287d7de9ef27b2db4583a969fed5b3974d9e066d7bd1a379f252ad136
    driver: lxc | qemu
    driver_version: 6.0.6 | 10.2.1
    instance_types:
  • container
  • virtual-machine
    firewall: nftables
    kernel: Linux
    kernel_architecture: x86_64
    kernel_features:
    bpf_token: "false"
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
    kernel_version: 6.8.0-106-generic
    lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
    os_name: Ubuntu
    os_version: "24.04"
    project: default
    server: lxd
    server_clustered: false
    server_event_mode: full-mesh
    server_name: instance-socket-test
    server_pid: 8745
    server_version: "6.8"
    server_lts: false
    storage: dir
    storage_version: "1"
    storage_supported_drivers:
  • name: ceph
    version: 19.2.3
    remote: true
  • name: cephfs
    version: 19.2.3
    remote: true
  • name: cephobject
    version: 19.2.3
    remote: true
  • name: zfs
    version: 2.2.2-0ubuntu9.4
    remote: false
  • name: dir
    version: "1"
    remote: false
  • name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: false
  • name: powerflex
    version: 2.8 (nvme-cli)
    remote: true
  • name: pure
    version: 2.1.9 (iscsiadm) / 2.8 (nvme-cli)
    remote: true
  • name: alletra
    version: 2.8 (nvme-cli) / 2.1.9 (iscsiadm)
    remote: true
  • name: btrfs
    version: 6.6.3
    remote: false

Instance log

Name: test1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 6953
Created: 2026/04/27 20:57 BST
Last Used: 2026/04/27 20:57 BST

Resources:
Processes: 24
Disk usage:
root: 0B
Disk total:
root: 0B
CPU usage:
CPU usage (in seconds): 4
Memory usage:
Memory (current): 46.86MiB
Network usage:
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 1.22kB
Bytes sent: 1.22kB
Packets received: 12
Packets sent: 12
IP addresses:
inet: 127.0.0.1/8 (local)
inet6: ::1/128 (local)
eth0:
Type: broadcast
State: UP
Host interface: veth2718f785
MAC address: 00:16:3e:62:5f:fb
MTU: 1500
Bytes received: 11.19kB
Bytes sent: 4.72kB
Packets received: 26
Packets sent: 40
IP addresses:
inet: 10.0.255.189/24 (global)
inet6: fe80::216:3eff:fe62:5ffb/64 (link)

Log:

Expected behavior

root@instance-socket-test:~# lxc exec test1 -- ls -al /dev/lxd/sock
srw-rw-rw- 1 nobody nogroup 0 Apr 27 19:42 /dev/lxd/sock

Actual behavior

root@instance-socket-test:~# lxc exec test1 -- ls -al /dev/lxd/sock
ls: cannot access '/dev/lxd/sock': No such file or directory

Steps to reproduce

Create an systom container instance, which wil have access to the instance socket [ /dev/lxd/sock ] by default
Upgrade LXD from 6.5 to either 6.7 or 6.8
The instance socket will be missing inside the isntance

Information to attach

• Any relevant kernel output (dmesg)
• Instance configuration (lxc config show <instance> --expanded)
• Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• If a lxc command fails, output of the command with --debug
• Output of the daemon with --debug (or use lxc monitor while reproducing the issue)

[jocado]

This behaviour is present upgrading from 6.5 to 6.7 or 6.8, but other upgrade paths are no tested.

Specifying the config security.devlxd=true before upgrade makes no difference.

Work around to restore the isntance socket are:

• Restart the instances immediately after the lxd upgrade
• Do set security.devlxd=true then unset security.devlxd, which also restore the instance socket

[jocado]

Reproduction Steps

Launch a VM [ I'm using multipass ]

multipass launch -c 2 -m 4G -d 40G --name instance-socket-test noble

Then exec in and sudo

multipass exec instance-socket-test -- sudo -i

Install older version of LXD, a revision of 6.5 in this case, rev 36020:

snap install --revision 36020 lxd

Setup LXD, creating a profile, then create an instance.

cat << END | lxd init --preseed -
config:
  images.auto_update_interval: '0'
networks:
  - name: lxdbr0
    type: bridge
    config:
      ipv4.address: '10.0.255.1/24'
      ipv4.nat: true
      ipv6.address: none
storage_pools:
  - name: default
    driver: dir
profiles:
  - name: default
    devices:
      root:
        path: /
        pool: default
        type: disk
      eth0:
        name: eth0
        nictype: bridged
        parent: lxdbr0
        type: nic
    config:
      limits.kernel.nice: "-20"
      limits.kernel.rtprio: "99"
      security.idmap.isolated: "true"
      security.nesting: "true"
      security.syscalls.intercept.mknod: "true"
      security.syscalls.intercept.setxattr: "true"
END

cat << END |  lxc launch ubuntu:noble test1
profiles:
- default
description: test
END

Check that the instance socket exists

lxc exec test1 -- ls -al /dev/lxd/
total 0
drwxr-xr-x 2 nobody nogroup  60 Apr 27 19:42 .
drwxr-xr-x 9 root   root    540 Apr 27 19:45 ..
srw-rw-rw- 1 nobody nogroup   0 Apr 27 19:42 sock

Now, upgrade LXD. Choosing 6.7 here:

snap refresh --revision 38768 lxd

Check for the instance socket:

lxc exec test1 -- ls -al /dev/lxd/
total 0
drwxr-xr-x 2 nobody nogroup  40 Apr 27 19:47 .
drwxr-xr-x 9 root   root    540 Apr 27 19:45 ..

You can restore it by setting the security config to default value or just restarting the instance.

lxc config set test1 security.devlxd=true
lxc config unset test1 security.devlxd # it's already fixed at this point - removing this makes no actual difference

lxc exec test1 -- ls -al /dev/lxd/
total 0
drwxr-xr-x 2 nobody nogroup  60 Apr 27 19:47 .
drwxr-xr-x 9 root   root    540 Apr 27 19:45 ..
srw-rw-rw- 1 nobody nogroup   0 Apr 27 19:47 sock

Cheers,
Just

[jocado]

Hi @tomponline - I wanted to bring this one to your attention.

We found that upgrading to the the latest or next latest version from 6.5 breaks the instance socket, and requires manual intervention.

Unfortunately, we found this at 10% of a release of 6.7 to our fleet.and cannot now continue with that release without a lot of manual recovery work. If you can find the issue, and fix in the pending 6.8 release, we would perhaps go that instead.

[tomponline]

Hi @jocado ill take a look at this tomorrow. Thanks

[tomponline]

@jocado is this a new regression since 6.5 or just a behaviour that has been now experienced and observed? Thanks

[jocado]

@tomponline Last time we upgraded, it did not create a problem. There is potential we didn't notice it, but I can't say for sure.

That upgrade was from 6.4 [ 6.4-e858240 rev 34099 ] to 6.5 [ 6.5-ccdfb39 rev 36020 ].

It's easy enough to test with the reproduction steps, let me try quickly.

[jocado]

I was able to reproduce going from 6.4 to 6.5

[tomponline]

OK so this may have been a long-time behaviour then.