[Bug]: DeltaProxy children miss targeted jobs with TCP or filtered ZeroMQ #70711

What happened?

With transport: tcp, a fully initialized DeltaProxy child does not receive jobs
targeted using its ordinary minion ID. The master logs the child as not connected,
even though its DeltaProxy controller is connected and the child is initialized.
Using a compound list expression for the same ID delivers the job to the child.

From the operator's perspective, a valid command such as
salt 'repro-child' test.ping times out and makes a running child look unresponsive.
The target syntax changes whether the job reaches the child at all.

Minimal setup

One loopback Salt master, one DeltaProxy controller (repro-controller), and one
built-in dummy child (repro-child). Both keys are preaccepted before startup.
No Salt extensions, external devices, or existing host Salt configuration are used.
The reproduction uses unmodified Salt 3008.5 installed with pip in a Debian 12
container with Python 3.11.18.

Master configuration, with temporary paths and dynamically allocated ports omitted:

interface: 127.0.0.1
transport: tcp
auto_accept: false
open_mode: false
pillar_roots:
  base:
    - /temporary/pillar

Proxy configuration, with temporary paths and matching ports omitted:

master: 127.0.0.1
transport: tcp
metaproxy: deltaproxy
multiprocessing: false
mine_enabled: false
append_minionid_config_dirs:
  - pki_dir
  - cachedir
  - extension_modules

pillar/top.sls:

base:
  repro-controller:
    - controller
  repro-child:
    - child

pillar/controller.sls:

proxy:
  proxytype: deltaproxy
  ids:
    - repro-child

pillar/child.sls:

proxy:
  proxytype: dummy

Steps to reproduce

Standalone reproduction script and full results: reproduction gist.

python3.11 -m venv /tmp/deltaproxy-tcp-venv
/tmp/deltaproxy-tcp-venv/bin/python -m pip install 'salt==3008.5'
/tmp/deltaproxy-tcp-venv/bin/python reproduce.py --artifacts ./tcp-targeting-results

The script creates isolated configurations and keys, starts both daemons, waits
for controller and child initialization, and runs the following commands against
the temporary master. Here $MASTER_CONFIG is its master configuration directory.

salt -c "$MASTER_CONFIG" -t 5 'repro-controller' test.ping
salt -c "$MASTER_CONFIG" -t 5 'repro-child' test.ping
salt -c "$MASTER_CONFIG" -t 5 -L 'repro-child' test.ping
salt -c "$MASTER_CONFIG" -t 5 -E '^repro-child$' test.ping
salt -c "$MASTER_CONFIG" -t 5 -C 'L@repro-child' test.ping
salt -c "$MASTER_CONFIG" -t 5 'repro-child' test.ping

Actual behavior

The compound job is dispatched to repro-child; the proxy log also records
dummy proxy returning ping. This clean run does not receive the compound result
at the CLI. This report concerns job delivery: ordinary child jobs
never reach the DeltaProxy, whereas the compound job reaches and executes on the
same child. The script records dispatch and CLI returns separately and does not
use a timeout alone as proof of this bug.

For ordinary/list/regex child targets, the master logs:

Publish target repro-child not connected {<salt.transport.tcp.Subscriber object at ...>}

For the compound child target, the proxy logs:

Command details {'fun': 'test.ping', 'arg': [], 'tgt': 'L@repro-child', ... 'tgt_type': 'compound', ...}
Deltaproxy minion_instance ... (ID: repro-child). Target: repro-child
dummy proxy returning ping

The same ordinary child target fails again after the compound test. Neither daemon
is restarted between commands. Full per-command output and matching log evidence
are retained in result.json.

Expected behavior

A connected DeltaProxy controller should receive publications addressed to its
initialized children, regardless of whether those children are selected with
ordinary ID/glob, list, regex, or compound targeting. Valid ordinary child targets
should reach the child without requiring a compound expression.

Suspected cause

In salt/channel/server.py, PublishServerChannel._package_payload creates
topic_lst for glob, list, and PCRE targets. The selected IDs are child IDs.
In salt/transport/tcp.py, PubServer.publish_payload matches those topics against
subscriber id_ values. The connected subscriber belongs to the controller,
so the child topic has no matching connection. Compound targets take the broadcast
path, allowing the DeltaProxy to match and dispatch the child locally.

This interpretation is supported by the master delivery log and the proxy's
compound dispatch log. It is offered as a diagnosis, not a proposed patch.

Additional reproduction: ZeroMQ with transport filtering

The same child-targeting failure also reproduces on 3008.x with the same isolated controller and built-in dummy child when both master and proxy use:

transport: zeromq
zmq_filtering: true

The proxy still uses metaproxy: deltaproxy and multiprocessing: false; both controller and child keys are preaccepted. The standalone reproducer linked above was adapted to select ZeroMQ and enable filtering on both daemons.

Before a ZeroMQ fix, the results were identical on the tests-only baseline (7e2262b1fa, based on upstream 3008.x) and the initial TCP-only fix (9410631be5):

With the default zmq_filtering: false, all six commands dispatch and return true on the initial PR branch. The filtered failure therefore concerns the shared DeltaProxy subscription: the controller subscribes to its own ID topic, while targeted child publications use child ID topics. Compound publications use the broadcast topic.

Separate startup issue: the initial ZeroMQ daemon comparisons used an external, diagnostic-only workaround for SyncWrapper cleanup cancelling a queue task during initial module synchronization. That independently reproduced startup bug is now tracked in #70740 and fixed by the standalone prerequisite PR #70741. Transport PR #70739 is stacked on that fix and includes its exact commit (0d5e723ddc), allowing final validation with normal remote file synchronization and no diagnostic startup hook. The publication-delivery failure documented above remains distinct from initialization.

Related historical report: #61367 explicitly describes DeltaProxy failures with TCP and filtered ZeroMQ and was closed during backlog cleanup. PR #70739 now covers both transports and explicitly documents its startup-fix dependency on #70741. This issue continues to distinguish publication delivery from CLI return handling.

Stacked fix verification

PR #70739 now combines the TCP and filtered ZeroMQ publication fixes with the independently reviewable startup prerequisite #70741. With the diagnostic hook removed and normal remote file synchronization enabled, all six standalone commands dispatch and return true over both TCP and filtered ZeroMQ. The automated integration matrices pass 24 cases for each transport, covering both filtering settings and both startup modes. The expanded unit and live ZeroMQ functional suites pass 332 and 24 cases respectively.

Type of salt install

pip (pypi)

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output

Salt Version:
          Salt: 3008.5
 
Python Version:
        Python: 3.11.18 (main, Nov  5 2026, 08:12:05) [GCC 12.2.0]
 
Dependency Versions:
          cffi: 2.1.1
      cherrypy: 18.10.0
  cryptography: 50.0.2
      dateutil: 4.0.0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.2.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 26.3
     pycparser: 3.12
      pycrypto: Not Installed
  pycryptodome: 3.24.2
        pygit2: Not Installed
  python-gnupg: 0.5.7
        PyYAML: 6.0.3
         PyZMQ: 27.3.0
        relenv: Not Installed
         smmap: 5.0.3
       timelib: 0.3.0
       Tornado: 6.6
           ZMQ: 4.3.5
 
Salt Package Information:
  Package Type: Not Installed
 
System Versions:
          dist: debian 12.15 bookworm
        locale: utf-8
       machine: aarch64
       release: 7.0.12-linuxkit
        system: Linux
       version: Debian GNU/Linux 12.15 bookworm