[Bug]: Deltaproxy sub-proxies share one module object, so module-level state in proxy/execution modules leaks across all controlled minions #70760

What happened?

Description

This is a follow-up to #68248. That issue was closed by #69426, which re-packs __grains__ into each sub-proxy's loaders. The underlying problem is broader and that change does not fix it: with deltaproxy, every sub-proxy's LazyLoader resolves a given module to the same Python module object. Any module-level state (connection handles, DETAILS dicts, caches) is therefore shared by all controlled minions, and the last sub-proxy to run init() wins. Every execution module that reaches the device through __proxy__ then returns the data of that one device, which is why grains, and in fact any function, return identical data for all controlled minions.

I applied #69426 to a 3006.9 install and the behaviour remains, as noted in #68248.

Root cause

In salt/loader/lazy.py, LazyLoader._load_module() loads modules with the deprecated spec.loader.load_module(), under a namespace built from loaded_base_name and the tag, e.g. salt.loaded.ext.proxy.fakedev. Every sub-proxy loader created in subproxy_post_master_init() uses the same loaded_base_name (salt.loaded), so they all compute the same namespace. When that name is already in sys.modules, load_module() re-executes the code into the existing module object instead of creating a new one.

The packed dunders (__context__, __grains__, __pillar__, __proxy__, ...) are injected as NamedLoaderContext objects that resolve through the active loader, so those stay per sub-proxy. Plain module globals do not.

Setup

Deltaproxy config (/etc/salt/proxy):

master: 127.0.0.1
metaproxy: deltaproxy
enable_fqdns_grains: False
multiprocessing: False   # also reproduces with True
skip_connect_on_init: True

Pillar top.sls:

base:
  dp_control:
    - control
  minion1:
    - p_minion1
  minion2:
    - p_minion2
  minion3:
    - p_minion3

control.sls:

proxy:
  proxytype: deltaproxy
  ids: [minion1, minion2, minion3]

p_minionN.sls (N = 1, 2, 3):

proxy:
  proxytype: fakedev
  host: 10.0.0.1N

salt://_proxy/fakedev.py. It follows the common pattern of keeping connection state in a module global, and also stores the host in __context__ for comparison:

__proxyenabled__ = ["fakedev"]
DETAILS = {}

def __virtual__():
    return "fakedev"

def init(opts):
    DETAILS["host"] = opts["proxy"]["host"]
    DETAILS["initialized"] = True
    __context__["fakedev"] = {"host": opts["proxy"]["host"]}

def initialized():
    return DETAILS.get("initialized", False)

def shutdown(opts):
    return True

def grains():
    return {"device_host": DETAILS.get("host")}

def host():
    return DETAILS.get("host")

def context_host():
    return __context__["fakedev"]["host"]

salt://_modules/fakedev.py:

__proxyenabled__ = ["fakedev"]

def __virtual__():
    return "fakedev"

def host():
    return __proxy__["fakedev.host"]()

def context_host():
    return __proxy__["fakedev.context_host"]()

Steps to Reproduce the behavior

salt-proxy --proxyid=dp_control -d
salt-key -A
salt 'minion*' fakedev.host --out=txt
salt 'minion*' fakedev.context_host --out=txt
salt 'minion*' grains.item device_host --out=txt
salt 'minion*' pillar.get proxy:host --out=txt

Output:

# module global: every minion reports the last-initialised device
minion1: 10.0.0.13
minion2: 10.0.0.13
minion3: 10.0.0.13

# __context__: correct per minion
minion1: 10.0.0.11
minion2: 10.0.0.12
minion3: 10.0.0.13

# grains derived from the proxy module: same wrong value everywhere
minion1: {'device_host': '10.0.0.13'}
minion2: {'device_host': '10.0.0.13'}
minion3: {'device_host': '10.0.0.13'}

# pillar: correct per minion
minion1: 10.0.0.11
minion2: 10.0.0.12
minion3: 10.0.0.13

The same result occurs with multiprocessing: True, because forked job processes inherit the already-shared module.

Expected behavior

Each sub-proxy should have isolated module state, so each controlled minion talks to and reports on its own device, as it would under a regular per-device salt-proxy.

Additional context

Related: #68248 (closed by #69426, which only addresses __grains__ re-packing).

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04, ubuntu-22.04

salt --versions-report output

Salt Version:
          Salt: 3006.9
 
Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.4 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-185-generic
        system: Linux
       version: Ubuntu 22.04.4 jammy