OVN Network ACLs: @internal and ACL-name subject selectors fail to allow intra-subnet traffic (while explicit CIDRs work)

[nsakkos]

Please confirm

• I have searched existing issues to check if an issue already exists for the bug I encountered.

It is worth mentioning this relevant bug from incus, in case this is an OVN issue lxc/incus#1423

Distribution

Ubuntu

Distribution version

24.04

Output of snap list --all lxd core20 core22 core24 snapd

-

System info

driver_version: 6.0.6 | 8.2.2
kernel_version: 6.8.0-62-generic
server_version: 5.21.4
storage: ceph | cephobject

Instance log

n/a

Expected behavior

When applying OVN Network ACLs to allow all intra-project communication using

• the @internal network subject selector (applied either to the network or instance NICs)
• an ACL-name group selector (applied to instance NICs)
  should allow the instances to communicate with one another

Actual behavior

Instances are unable to communicate with each other, resulting in Connection refused.
Even when lxc network acl show <acl> confirms the rule is state: enabled and the instances correctly appear in the used_by list, the OVN firewall seem to be blocking the cross-instance traffic.

However, when the @internal or ACL-name selector is replaced with the explicit subnet CIDR (e.g., 10.171.2.128/26), the exact same traffic succeeds instantly.

Steps to reproduce

1. Create an OVN network (acl-test) with a defined IPv4 subnet (e.g., 10.171.2.128/26).
2. Create two instances attached to this network.

@internal issue

1. Create a network ACL allowing ingress/egress for the @internal subject. lxc network acl rule add test-acl ingress action=allow source='@internal' destination_port=8201 protocol=tcp and lxc network acl rule add test-acl egress action=allow destination='@internal'

2. Apply the ACL to the network

3. Start a listener on Instance A

4. Attempt to connect from Instance B (fails)

5. Modify the ACL to use its own name as the source/destination (e.g., source=my-service-acls).

6. Remove the ACL from the network, and apply it directly to the instances' NICs via profile (lxc profile device set default eth0 security.acls=my-service-acls).

7. Verify instances show up in the ACL's used_by list.

8. Attempt the nc test from Instance B to Instance A again (fails)

Workaround

1. Modify the ACL, replacing the logical selectors with the explicit subnet: source=10.171.2.128/26.
2. Attempt the nc test from Instance B to Instance A (succeeds)

**

instance_config.txt

**

An example of @internal used to allow traffic on 8201 (failing) and CIDR for 8200 (succeeding) https://pastebin.canonical.com/p/xBszSqXYYJ/

Information to attach

• Any relevant kernel output (dmesg)
• Instance configuration (lxc config show <instance> --expanded)
• Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• If a lxc command fails, output of the command with --debug
• Output of the daemon with --debug (or use lxc monitor while reproducing the issue)

[nsakkos]

I've done some more testing, the @internal and ACL-name subject selectors only work when both source and destination are on the same host. I hope this helps narrowing it down a bit.

For the @internal selector:

 lxc config show juju-6d5e60-2 --project vault-drs-test-2 | tail -20
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: f34bcd0c-2fb3-433c-8b17-dad25386b368
  volatile.uuid.generation: f34bcd0c-2fb3-433c-8b17-dad25386b368
devices:
  eth0:
    network: vault-drs-test-2
    security.acls: vault-role-internal
    type: nic
  root:
    path: /
    pool: remote
    size: 51200MiB
    type: disk
ephemeral: false
profiles:
- default
- juju-vault-drs-test-2-772a26
stateful: false
description: ""

~ 
❯ lxc network acl show vault-role-internal
name: vault-role-internal
description: ""
egress:
- action: allow
  destination: '@internal'
  protocol: tcp
  destination_port: "8201"
  state: enabled
ingress:
- action: allow
  source: '@internal'
  protocol: tcp
  destination_port: "8201"
  state: enabled
config: {}
used_by:
- /1.0/instances/juju-6d5e60-0?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-1?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-2?project=vault-drs-test-2
project: vault-drs-test-2

~ 
❯ lxc ls
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
|     NAME      |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS | LOCATION |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| juju-6d5e60-0 | RUNNING | 10.171.2.132 (eth0) | fd42:de23:7a8e:1b03:216:3eff:fe96:c592 (eth0) | CONTAINER | 0         | drs-is3  |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| juju-6d5e60-1 | RUNNING | 10.171.2.130 (eth0) | fd42:de23:7a8e:1b03:216:3eff:fe65:cd74 (eth0) | CONTAINER | 0         | drs-is2  |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| juju-6d5e60-2 | RUNNING | 10.171.2.131 (eth0) | fd42:de23:7a8e:1b03:216:3eff:fecf:35db (eth0) | CONTAINER | 0         | drs-is1  |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| juju-6d5e60-4 | RUNNING | 10.171.2.133 (eth0) | fd42:de23:7a8e:1b03:216:3eff:fe3e:4188 (eth0) | CONTAINER | 0         | drs-is1  |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| juju-6d5e60-5 | RUNNING | 10.171.2.134 (eth0) | fd42:de23:7a8e:1b03:216:3eff:fe14:250 (eth0)  | CONTAINER | 0         | drs-is2  |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+

~ took 2s 
❯ lxc exec juju-6d5e60-5 --project vault-drs-test-2 -- bash -c 'for ip in 10.171.2.130 10.171.2.131 10.171.2.132; do   echo "=== Testing from $(hostname -f) to $ip ===";   nc -vz -w3 "$ip" 8200;   nc -vz -w3 "$ip" 8201;   echo; done'
=== Testing from juju-6d5e60-5.lxd to 10.171.2.130 ===
Connection to 10.171.2.130 8200 port [tcp/*] succeeded!
Connection to 10.171.2.130 8201 port [tcp/*] succeeded!

=== Testing from juju-6d5e60-5.lxd to 10.171.2.131 ===
Connection to 10.171.2.131 8200 port [tcp/*] succeeded!
nc: connect to 10.171.2.131 port 8201 (tcp) failed: Connection refused

=== Testing from juju-6d5e60-5.lxd to 10.171.2.132 ===
Connection to 10.171.2.132 8200 port [tcp/*] succeeded!
nc: connect to 10.171.2.132 port 8201 (tcp) failed: Connection refused

For the ACL-name selector:

❯ lxc network acl create vault-members --project vault-drs-test-2
Network ACL vault-members created

~ 
❯ lxc network acl show vault-members --project vault-drs-test-2
name: vault-members
description: ""
egress: []
ingress: []
config: {}
used_by: []
project: vault-drs-test-2

~ 
❯ lxc network acl create vault-policy-group --project vault-drs-test-2
Network ACL vault-policy-group created

~ 
❯ lxc network acl rule add vault-policy-group ingress \
  action=allow \
  source=vault-members \
  protocol=tcp \
  destination_port=8201 \
  --project vault-drs-test-2

lxc network acl rule add vault-policy-group egress \
  action=allow \
  destination=vault-members \
  protocol=tcp \
  destination_port=8201 \
  --project vault-drs-test-2

~ took 2s 
❯ lxc network acl show vault-policy-group --project vault-drs-test-2
name: vault-policy-group
description: ""
egress:
- action: allow
  destination: vault-members
  protocol: tcp
  destination_port: "8201"
  state: enabled
ingress:
- action: allow
  source: vault-members
  protocol: tcp
  destination_port: "8201"
  state: enabled
config: {}
used_by: []
project: vault-drs-test-2

~ 
❯ lxc config device override juju-6d5e60-0 eth0 \
  security.acls=vault-members,vault-policy-group \
  --project vault-drs-test-2

lxc config device override juju-6d5e60-1 eth0 \
  security.acls=vault-members,vault-policy-group \
  --project vault-drs-test-2

lxc config device override juju-6d5e60-2 eth0 \
  security.acls=vault-members,vault-policy-group \
  --project vault-drs-test-2
Device eth0 overridden for juju-6d5e60-0
Device eth0 overridden for juju-6d5e60-1
Device eth0 overridden for juju-6d5e60-2

❯ lxc network acl show vault-members --project vault-drs-test-2
lxc network acl show vault-policy-group --project vault-drs-test-2
name: vault-members
description: ""
egress: []
ingress: []
config: {}
used_by:
- /1.0/instances/juju-6d5e60-0?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-1?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-2?project=vault-drs-test-2
- /1.0/network-acls/vault-policy-group?project=vault-drs-test-2
project: vault-drs-test-2
name: vault-policy-group
description: ""
egress:
- action: allow
  destination: vault-members
  protocol: tcp
  destination_port: "8201"
  state: enabled
ingress:
- action: allow
  source: vault-members
  protocol: tcp
  destination_port: "8201"
  state: enabled
config: {}
used_by:
- /1.0/instances/juju-6d5e60-0?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-1?project=vault-drs-test-2
- /1.0/instances/juju-6d5e60-2?project=vault-drs-test-2
project: vault-drs-test-2

❯ lxc exec juju-6d5e60-1 --project vault-drs-test-2 -- bash -c '
for ip in 10.171.2.131 10.171.2.132; do
  echo "=== Testing from $(hostname -f) to $ip ==="
  nc -vz -w3 "$ip" 8201
  echo
done'
=== Testing from juju-6d5e60-1.lxd to 10.171.2.131 ===
nc: connect to 10.171.2.131 port 8201 (tcp) failed: Connection refused

=== Testing from juju-6d5e60-1.lxd to 10.171.2.132 ===
nc: connect to 10.171.2.132 port 8201 (tcp) failed: Connection refused


~ took 3s 
❯ lxc exec juju-6d5e60-2 --project vault-drs-test-2 -- bash -c '
for ip in 10.171.2.130 10.171.2.132; do
  echo "=== Testing from $(hostname -f) to $ip ==="
  nc -vz -w3 "$ip" 8201
  echo
done'
=== Testing from juju-6d5e60-2.lxd to 10.171.2.130 ===
nc: connect to 10.171.2.130 port 8201 (tcp) failed: Connection refused

=== Testing from juju-6d5e60-2.lxd to 10.171.2.132 ===
nc: connect to 10.171.2.132 port 8201 (tcp) failed: Connection refused


❯ lxc network acl create testers-members --project vault-drs-test-2
Network ACL testers-members created

~ 
❯ lxc config device override juju-6d5e60-4 eth0 \
  security.acls=testers-members \
  --project vault-drs-test-2

lxc config device override juju-6d5e60-5 eth0 \
  security.acls=testers-members \
  --project vault-drs-test-2
Device eth0 overridden for juju-6d5e60-4
Device eth0 overridden for juju-6d5e60-5

~ took 3s 
❯ lxc network acl rule add vault-policy-group ingress   action=allow   source=testers-members   protocol=tcp   destination_port=8201   --project vault-drs-test-2

~ took 2s 
❯ lxc network acl rule add vault-policy-group egress   action=allow   destination=testers-members   protocol=tcp   destination_port=8201   --project vault-drs-test-2

~ took 2s 


❯ lxc exec juju-6d5e60-4 --project vault-drs-test-2 -- bash -c '
for ip in 10.171.2.130 10.171.2.131 10.171.2.132; do
  echo "=== Testing from $(hostname -f) to $ip ==="
  nc -vz -w3 "$ip" 8201
  echo
done'
=== Testing from juju-6d5e60-4.lxd to 10.171.2.130 ===
nc: connect to 10.171.2.130 port 8201 (tcp) failed: Connection refused

=== Testing from juju-6d5e60-4.lxd to 10.171.2.131 ===
Connection to 10.171.2.131 8201 port [tcp/*] succeeded!

=== Testing from juju-6d5e60-4.lxd to 10.171.2.132 ===
nc: connect to 10.171.2.132 port 8201 (tcp) failed: Connection refused

❯ lxc exec juju-6d5e60-5 --project vault-drs-test-2 -- bash -c '
for ip in 10.171.2.130 10.171.2.131 10.171.2.132; do
  echo "=== Testing from $(hostname -f) to $ip ==="
  nc -vz -w3 "$ip" 8201
  echo
done'
=== Testing from juju-6d5e60-5.lxd to 10.171.2.130 ===
Connection to 10.171.2.130 8201 port [tcp/*] succeeded!

=== Testing from juju-6d5e60-5.lxd to 10.171.2.131 ===
nc: connect to 10.171.2.131 port 8201 (tcp) failed: Connection refused

=== Testing from juju-6d5e60-5.lxd to 10.171.2.132 ===
nc: connect to 10.171.2.132 port 8201 (tcp) failed: Connection refused