[dhcp_server] Wait for caclmgrd-owned docker0 syslog rule on start/stop

[Xichen96]

Why I did it

dhcp_server is the only container that forwards its rsyslog to the host over docker0 (other containers, including bridge-mode redfish, log to 127.0.0.1); that forwarded traffic uses RELP (tcp/2514). caclmgrd flushes and rebuilds the INPUT chain on every control-plane ACL change and appends a catch-all DROP. Ownership of the docker0/2514 ACCEPT exception has moved from the container start/stop script to caclmgrd, which re-installs it on every rebuild keyed on FEATURE|dhcp_server (fixes #27584).

This is step 3 of 3:

1. [dhcp_server] Revert container-side docker0 syslog iptables add/del (#26637); caclmgrd to own the rule #28328 (merged) removed the old container-side add/del of the rule.
2. [caclmgrd] Own the dhcp_server docker0 syslog INPUT exception sonic-host-services#412 (merged) made caclmgrd own the rule.
3. This PR replaces the removed container-side add/del with feature-gated bounded waits that synchronize with caclmgrd.

Work item tracking

• Microsoft ADO (number only): 38745012

How I did it

Re-add the dhcp_server branches in files/build_templates/docker_image_ctl.j2 at the same spots #28328 reverted, but as bounded waits instead of add/del:

• preStartAction: when FEATURE|dhcp_server is enabled, wait up to 10s for the docker0 tcp/2514 ACCEPT rule to be present before the container starts, so the RELP client's first (state NEW) connection is not dropped by the control-plane catch-all DROP.
• stop(): when FEATURE|dhcp_server is disabled, wait up to 10s for the rule to be removed. Plain container restarts, shutdown, and CONFIG_DB-unreachable all skip the wait (positive-state gate, so a restart with the feature still enabled does not block).

The canonical iptables -C command is defined once and reused by both waits. Each wait checks at seconds 0 through 10 without a duplicate final command: status 0 means present, status 1 means absent, and status 4 (xtables resource/lock contention) is retried. Other statuses produce a warning. The rule specification matches the stored form caclmgrd installs and the live iptables -S INPUT output.

How to verify it

The wait behavior was hardware-validated as part of the 3-PR set by hand-patching the rendered dhcp_server ctl script and caclmgrd change onto a running DUT (no image rebuild).

• DUT / topology: bjw-can-720dt-2 (720dt, mx), dhcp_server enabled; RELP tcp/2514 to host rsyslogd on 240.127.1.1:2514 (the docker0 IP); a control-plane ACL rule present so caclmgrd programs its catch-all -A INPUT -j DROP.
• preStartAction wait (present): with the feature enabled, the container start proceeds once caclmgrd has the docker0/2514 ACCEPT rule installed (INPUT pos 3, above the catch-all DROP); end-to-end docker exec dhcp_server logger ... reached host /var/log/syslog.
• stop() wait (absent): config feature state dhcp_server disabled -> caclmgrd removes the rule and the container reaches Exited(0); re-enabled re-adds the rule and the container starts.
• Restart does not block (by design; not separately hardware-exercised): the stop wait is gated on FEATURE|dhcp_server being disabled, so a plain container restart with the feature still enabled skips it.
• Final polling implementation: rendered the dhcp_server specialization and passed bash -n; a controlled shell harness passed present, absent, lock-busy, unexpected-error, and timeout cases; the exact iptables -C command returned status 0 against the live rule on the 202605 DUT.

Note: this hardware validation was performed on a 202605 image (identical container ctl / caclmgrd path); the change under test is the same logic.

Which release branch to backport (provide reason below if selected)

• 202305
• 202311
• 202405
• 202411
• 202505
• 202511
• 202512
• 202605
• 202608

Coordinated with the caclmgrd change (step 2); the buildimage backport is handled together with #28328 as a set so the branch is never left with neither party owning the rule.
Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO): Microsoft ADO 38745012
Failure type: day-one issue (the container-owned docker0 syslog rule never survived a caclmgrd control-plane ACL rebuild)

Tested branch

• master
• 202305
• 202311
• 202405
• 202411
• 202505
• 202511
• 202512
• 202605
• 202608

Test result

• 202605: 20260510.07 — hand-patched HW validation on bjw-can-720dt-2 (720dt, mx); see "How to verify it" above. The docker0 tcp/2514 ACCEPT rule is programmed above the catch-all DROP, end-to-end RELP syslog is delivered, the feature toggle adds/removes the rule cleanly, and the rule survives an unrelated control-plane ACL re-walk (bug Bug: CACL updates remove dhcp_server syslog iptables rule #27584 fix).
• master: validation pending — the rendered dhcp_server ctl script and caclmgrd code path are identical to 202605; a master-image run is the only remaining validation step.

Description for the changelog

[dhcp_server] Replace the container-side docker0 syslog iptables add/del with feature-gated bounded waits on the caclmgrd-owned tcp/2514 rule.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR updates the rendered dhcp_server container control script template to synchronize start/stop with the iptables rule that caclmgrd now owns (docker0 → host rsyslog RELP tcp/2514), using bounded waits instead of container-side rule add/delete. This helps prevent syslog connectivity drops during control-plane ACL rebuilds by ensuring the exception rule is present before start, and (on explicit feature disable) removed before stop returns.

Changes:

• Add a preStartAction wait (up to 10s) for the docker0 tcp/2514 ACCEPT rule when FEATURE|dhcp_server is enabled.
• Add a stop() wait (up to 10s) for the rule to be removed when FEATURE|dhcp_server is disabled.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

files/build_templates/docker_image_ctl.j2:930

• Same bounding issue in stop(): the loop intends to wait ~10s, but iptables -w 5 -C can block up to 5s per probe under xtables lock contention, so this path may take far longer than 10s (and the warning text becomes misleading). Tightening -w and retry count keeps the stop path’s worst-case delay aligned with the intended bound.

    if [ "$($SONIC_DB_CLI CONFIG_DB HGET 'FEATURE|dhcp_server' state 2>/dev/null)" = "disabled" ]; then
        for i in $(seq 1 10); do
            iptables -w 5 -C INPUT -i docker0 -p tcp -m tcp --dport 2514 -m comment --comment "dhcp_server_syslog" -j ACCEPT 2>/dev/null || break
            sleep 1
        done
        iptables -w 5 -C INPUT -i docker0 -p tcp -m tcp --dport 2514 -m comment --comment "dhcp_server_syslog" -j ACCEPT 2>/dev/null && \
            echo "Warning: caclmgrd-owned dhcp_server docker0 syslog ACCEPT rule still present after 10s despite dhcp_server feature disabled."
    fi

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

files/build_templates/docker_image_ctl.j2:45

• Consider adding a short xtables-lock wait to the iptables existence probe. Without -w, iptables -C returns immediately with status 4 when caclmgrd is holding the lock, which forces the loop to sleep 1s even for sub-second contention (delaying start/stop and possibly producing spurious warnings).

DHCP_SERVER_SYSLOG_CHECK=(iptables -C INPUT -i docker0 -p tcp -m tcp --dport 2514 -m comment --comment dhcp_server_syslog -j ACCEPT)

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Xichen96]

/azpw run

[mssonicbld]

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

[mssonicbld]

No failed(or canceled) jobs found in completed stages. Only failed(or canceled) jobs in completed stages can be retried.

Stages in progress: BuildVS, Build. Please wait for the stages to complete and then retry again.