[vpp] SAIVPP unit-test harness (docker-sai-test-vpp) + devdocs by nicholasching · Pull Request #1950 · sonic-net/sonic-sairedis

nicholasching · 2026-06-23T16:04:13Z

Description of PR

Summary:
Adds the self-contained Docker harness (docker-sai-test-vpp) that runs the upstream OCP sai_test PTF suite against the real VPP SAI backend (libsaivs) in one container — VPP + saiserver + PTF + veth/AF_PACKET topology — and produces a per-test compatibility matrix. This is test scaffolding only (everything under .azure-pipelines/docker-sai-test-vpp/); it does not touch any production/runtime code, so it cannot affect existing sonic-sairedis functionality. Reviewers can start at .azure-pipelines/docker-sai-test-vpp/README.md and run_test.sh.

Fixes # (N/A — no upstream issue)

Type of change

Approach

What is the motivation for this PR?

There was no in-tree way to run the OCP sai_test suite against the VPP SAI backend and track per-test coverage. This harness provides that, plus the development docs and compatibility matrices used to drive the backend fixes.

How did you do it?

Added under .azure-pipelines/docker-sai-test-vpp/:

Dockerfile and run_test.sh (orchestrates Redis → veth/PortChannel topology → VPP → saiserver → PTF), plus T0 config (sai.profile, lanemap.ini, port-map.ini, ptf-port-map.ini) and gen_compatibility_matrix.py.
Config-signature grouping: because the VPP SAI backend builds the switch once per saiserver process, run_test.sh groups tests by their setUp config signature and restarts the backend per group, so one container can run any mix of tests.
ISOLATE_EACH_TEST (default on): run each test in its own group with a fresh backend + own config to eliminate cross-test config-reuse contamination; set 0 for faster grouped runs.
L3-over-LAG forwarding (harness portion): run_test.sh assigns the DUT-side connected IPs to each LAG (be<N>) and SVI (bvi<vlan>) so routed-to-LAG/SVI traffic forwards in the standalone PTF environment (the backend portion is in the backend-fixes PR).
Debug hold to keep VPP/saiserver/veths alive after a --debug run for vppctl inspection.
devdocs/, demodocs/, and results/ compatibility matrices; .gitignore for generated artifacts.

How did you verify/test it?

Running the harness against sai_route_test/sai_rif_test/sai_neighbor_test/sai_ecmp_test produces a compatibility matrix (current state: 34 PASS, captured in devdocs//results/). The harness is self-contained and does not build or run any production code.

Any platform specific information?

VPP platform test infrastructure only; isolated under .azure-pipelines/docker-sai-test-vpp/. No impact on the sonic-sairedis build or runtime behavior.

Documentation

This PR adds the harness README.md plus rolling/dated progress notes and deep-dive debug logs under .azure-pipelines/docker-sai-test-vpp/devdocs/ and demo docs under demodocs/.

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…test Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…dog, --relax Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…ature and restarting backend per group Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…ugging Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

UT-harness portion of the L3-over-LAG forwarding fix: run_test.sh assigns the DUT-side connected IPs to each LAG (be<N>) and SVI (bvi<vlan>) so routed-to-LAG /SVI traffic forwards in the standalone PTF environment. The backend portion (SwitchVppFdb.cpp) is in the VPP SAI backend PR. Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

linux-foundation-easycla · 2026-06-23T16:04:20Z

The committers listed above are authorized under a signed CLA.

✅ login: nicholasching / name: Nicholas Ching (01a0762, 02b11fa, 120e57f, 145aec2, 15776bb, 2661943, 4cd92d8, 4e1091b, 5780b35, 5cda18e, 77a77e9, 82ca412, b2ee2a3, b34a713, b882a30, c36a5f1, c78809e, dcc9a8f)

mssonicbld · 2026-06-23T16:04:21Z

/azp run

azure-pipelines · 2026-06-23T16:04:35Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-23T17:59:21Z

/azp run

azure-pipelines · 2026-06-23T17:59:33Z

Azure Pipelines successfully started running 1 pipeline(s).

…SER) - gen_compatibility_matrix.py: prefer defusedxml for parsing PTF JUnit XML (XXE-hardened), with a stdlib fallback; input is our own local test output. - Dockerfile: annotate the root entrypoint with a nosemgrep justification — the --privileged harness must run as root to start VPP/Redis/saiserver and manage veth/AF_PACKET; it is a disposable test container, never a deployed service. Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-06-23T18:04:03Z

/azp run

azure-pipelines · 2026-06-23T18:04:16Z

Azure Pipelines successfully started running 1 pipeline(s).

Satisfy tests/checkwhitespace.sh (Azure Build amd64 gate). Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

azure-pipelines · 2026-07-02T15:23:29Z

Azure Pipelines successfully started running 1 pipeline(s).

nicholasching · 2026-07-03T01:23:53Z

Hi Fred (@yue-fred-gao), the t1-lag-vpp sanity test has passed on the associated sonic-buildimage PR: sonic-net/sonic-buildimage#28145

I have attached the results below:
Github Run: https://github.com/sonic-net/sonic-buildimage/runs/84906281352
Azure Build: https://dev.azure.com/mssonic/build/_build/results?buildId=1154935&view=logs&s=859b8d9a-8fd6-5a5c-6f5e-f84f1990894e&j=314bc92a-d36a-5eac-5c45-43e8f1288876

PortChannel netdev creation and LAG/SVI connected-IP assignment now run from sai_test config helpers (vpp_ut_support.py) when each LAG or RIF is created, instead of global run_test.sh startup/watchdog loops. Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-07-03T19:35:37Z

/azp run

azure-pipelines · 2026-07-03T19:35:49Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-03T21:15:16Z

/azp run

azure-pipelines · 2026-07-03T21:15:28Z

Azure Pipelines successfully started running 1 pipeline(s).

yue-fred-gao · 2026-07-03T21:25:20Z

+COPY SAI/test/ptf /opt/ptf
+COPY SAI/test/sai_test /sai_test
+COPY .azure-pipelines/docker-sai-test-vpp/vpp_ut_support.py /sai_test/config/vpp_ut_support.py
+COPY .azure-pipelines/docker-sai-test-vpp/harness_sai_test/overrides/lag_configer.py /sai_test/config/lag_configer.py


This creates maintenance debt. If the one in sai_test is modified and creates a new dependency from certain test case, the "overrides" may become incompatible. One option is to modify sai_test lag_configer.py. It can be based on some env or config option like SIMULATE_SONIC. It is not necessary to be vpp specific.

This reply consolidates this thread with #1950 (comment)

I agree, this creates two sources of truth. I've moved the PortChannel and LAG/SVI-IP setup out of the harness overrides and into sai_test, gated by a SIMULATE_SONIC option, disabled by default. As a result, the harness no longer needs to overlay sai_test, and run_test.sh just configures SIMULATE_SONIC=1 instead.

The sai_test change is a separate SAI PR (opencomputeproject/SAI#2316). The changes in SAI were made to be platform agnostic to respect the shared framework, therefore, VPP operations are passed in from the harness as a command template rather than hardcoding vppctl in sai_test.

I've re-validated the full integration of the changes made across (#1950, #1951, #1952, and the 4 SAI PRs) and verified there are no regressions against the 34-PASS baseline.

mssonicbld · 2026-07-06T13:56:07Z

/azp run

azure-pipelines · 2026-07-06T13:56:20Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-06T19:11:56Z

/azp run

azure-pipelines · 2026-07-06T19:12:08Z

Azure Pipelines successfully started running 1 pipeline(s).

…f using SAI teardown Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…MULATE_SONIC Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-07-06T19:55:55Z

/azp run

azure-pipelines · 2026-07-06T19:57:28Z

Azure Pipelines successfully started running 1 pipeline(s).

nicholasching · 2026-07-06T20:11:01Z

/azp run

azure-pipelines · 2026-07-06T20:11:12Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-07T22:09:36Z

/azp run

azure-pipelines · 2026-07-07T22:09:48Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-07-08T20:51:06Z

/azp run

azure-pipelines · 2026-07-08T20:51:20Z

Azure Pipelines successfully started running 1 pipeline(s).

nicholasching added 17 commits June 23, 2026 07:49

feat: configuration for 32 ports; phase 1, task 2

77a77e9

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: configuration and run script; phase 1, task 3-4; fails to pass …

b2ee2a3

…test Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

fix: ifmap prefix, IPv6 disable, PortChannel emulation, veth-up watch…

145aec2

…dog, --relax Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

fix: run multiple tests sequentially by grouping tests by config sign…

4cd92d8

…ature and restarting backend per group Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: script to generate compatabilty matrix from test results

2661943

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add feature development progress context

82ca412

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add build and usage instructions for UT harness

15776bb

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

chore: add hold when debug enabled to leave container running for deb…

dcc9a8f

…ugging Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add demo documentation

c36a5f1

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add coverage by functionality

b882a30

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: minor updates to demo doc

01a0762

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: current dev state, compatibility matrix

c78809e

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: LAG-RIF neighbor/route + egress-disable progress (6-18)

4e1091b

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: harness per-test isolation mode, restart backend between tests

5780b35

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: update to span full width

b34a713

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: neighbor host-route local0 fix progress (6-19)

5cda18e

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

This was referenced Jun 23, 2026

[vpp] Dedicated binary-API socket and thread for asynchronous VPP events #1951

Open

[vpp] VPP SAI backend fixes: CPU queue, host-interface admin-up, L3-over-LAG/host-route forwarding, LAG egress-disable #1952

Open

nicholasching force-pushed the vpp_ut_harness branch from 73630af to 02b11fa Compare June 23, 2026 17:59

nicholasching force-pushed the vpp_ut_harness branch from 02b11fa to 4618cda Compare June 23, 2026 18:03

chore: strip trailing whitespace in progress-6-10 devdoc

7805b63

Satisfy tests/checkwhitespace.sh (Azure Build amd64 gate). Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

nicholasching closed this Jul 3, 2026

nicholasching reopened this Jul 3, 2026

yue-fred-gao reviewed Jul 3, 2026

View reviewed changes

nicholasching closed this Jul 6, 2026

nicholasching reopened this Jul 6, 2026

nicholasching added 2 commits July 6, 2026 12:54

docs: explain why the harness restarts the backend per test instead o…

4da0942

…f using SAI teardown Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

chore: drop sai_test overrides and drive PortChannel/RIF setup via SI…

e57cb6e

…MULATE_SONIC Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

nicholasching force-pushed the vpp_ut_harness branch from 641e9de to e57cb6e Compare July 6, 2026 19:55

nicholasching closed this Jul 7, 2026

nicholasching reopened this Jul 7, 2026

fix: satisfy aspell and SWSS_LOG_ENTER checks in UT harness preload shim

a01722c

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Uh oh!

Conversation

nicholasching commented Jun 23, 2026

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

linux-foundation-easycla Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jul 2, 2026

Uh oh!

nicholasching commented Jul 3, 2026

Uh oh!

mssonicbld commented Jul 3, 2026

Uh oh!

azure-pipelines Bot commented Jul 3, 2026

Uh oh!

mssonicbld commented Jul 3, 2026

Uh oh!

azure-pipelines Bot commented Jul 3, 2026

Uh oh!

yue-fred-gao Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

nicholasching Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

mssonicbld commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

mssonicbld commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

nicholasching commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

mssonicbld commented Jul 7, 2026

Uh oh!

azure-pipelines Bot commented Jul 7, 2026

Uh oh!

mssonicbld commented Jul 8, 2026

Uh oh!

azure-pipelines Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

linux-foundation-easycla Bot commented Jun 23, 2026 •

edited

Loading