Documentation for mocked BLE-WiFi commissioning (project-chip#41359)

* Add missing codeblock language specifiers in docs/testing

* Documentation for mocked BLE-WiFi commissioning

* Add more verbose doc for WPA supplicant mock class

* Restyled by prettier-markdown

* Fix doc build and add marmaid support

---------

Co-authored-by: Restyled.io <[email protected]>

Author: arkq

.github/workflows/tests.yaml

@@ -368,7 +368,7 @@ jobs:
                   ./scripts/run_in_build_env.sh \
                   "./scripts/tests/run_test_suite.py \
                      --runner chip_tool_python \
-                     --target TestCommissionerNodeId \
+                     --target TestOperationalState \
                      --chip-tool ./out/linux-x64-chip-tool${CHIP_TOOL_VARIANT}-${BUILD_VARIANT}/chip-tool \
                      run \
                      --iterations 1 \

docs/conf.py

@@ -21,6 +21,7 @@
 extensions = [
     "myst_parser",
     "external_content",
+    "sphinxcontrib.mermaid",
 ]
 exclude_patterns = [
     "_build",
@@ -61,7 +62,7 @@
     "myst.xref_missing",
 ]
 myst_enable_extensions = ["html_image"]
-
+myst_fence_as_directive = ["mermaid"]
 
 # -- Options for external_content --------------------------------------------
 

docs/requirements.txt

@@ -1,6 +1,7 @@
 docutils==0.21.2
 Sphinx==8.2.3
 sphinx-book-theme
+sphinxcontrib-mermaid
 myst-parser>=2.0.0
 breathe>=4.34
 pydata-sphinx-theme==0.16.1

docs/testing/fuzz_testing.md

@@ -22,7 +22,7 @@ test. Each fuzzer function is defined using
 
 The Fuzzer must be located in a Test Folder : `src/some_directory/tests/`
 
-```
+```cpp
 #include <cstddef>
 #include <cstdint>
 
@@ -39,7 +39,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t * data, size_t len)
 
     return 0;
 }
-
 ```
 
 See
@@ -104,11 +103,11 @@ for an example of a simple fuzz test.
         ```
 
 -   Build all fuzzers
-    ```
+    ```shell
     ./scripts/build/build_examples.py --target <host>-<compiler>-tests-asan-libfuzzer-clang build
     ```
     e.g.
-    ```
+    ```shell
     ./scripts/build/build_examples.py --target darwin-arm64-tests-asan-libfuzzer-clang build
     ```
     \*\* Make sure to put the right host and compiler
@@ -263,14 +262,14 @@ There are several ways to run the tests:
 
 1.  Unit-test mode (where the inputs are only fuzzed for a second):
 
-```bash
+```shell
 ./fuzz-chip-cert-pw
 ```
 
 2.  Continuous fuzzing mode; we need to first list the tests, then specify the
     FuzzTestCase to run:
 
-```bash
+```console
 $ ./fuzz-chip-cert-pw --list_fuzz_tests
 [.] Sanitizer coverage enabled. Counter map size: 11134, Cmp map size: 262144
 [*] Fuzz test: ChipCert.ChipCertFuzzer
@@ -281,20 +280,19 @@ $ ./fuzz-chip-cert-pw --fuzz=ChipCert.DecodeChipCertFuzzer
 
 3. Running all Tests in a TestSuite for a specific time, e.g for 10 minutes
 
-```bash
-#both Fuzz Tests will be run for 10 minutes each
+```shell
+# both Fuzz Tests will be run for 10 minutes each
 ./fuzz-chip-cert-pw --fuzz_for=10m
 ```
 
 4. For Help
 
-```bash
+```shell
 # FuzzTest related help
 ./fuzz-chip-cert-pw --helpfull
 
 # gtest related help
 ./fuzz-chip-cert-pw --help
-
 ```
 
 ### Coverage Report Generation

docs/testing/integration_test_utilities.md

@@ -99,7 +99,7 @@ To add new fault injection code paths:
 
 ### Fault Injection example
 
-```
+```cpp
 CHIP_ERROR CASEServer::OnMessageReceived(Messaging::ExchangeContext * ec,
    const PayloadHeader & payloadHeader,
                                         System::PacketBufferHandle && payload)
@@ -110,7 +110,6 @@ CHIP_ERROR CASEServer::OnMessageReceived(Messaging::ExchangeContext * ec,
    CHIP_FAULT_INJECT(FaultInjection::kFault_CASEServerBusy, busy = true);
    if (busy)
    {
-…
 ```
 
 ### Fault Injection cluster

docs/testing/integration_tests.md

@@ -44,3 +44,104 @@ way that is understandable, readable, and has the features you need
         (certs, commissioning, etc), less plumbing if you need to add features,
         can use python libraries
     -   cons: more complex, can be harder to read
+
+## Running integration tests locally
+
+When integration tests are run locally, the test runner (YAML or python) needs
+to mock network connectivity between the controller and the device under test,
+so that all tests can be run without actual hardware. In case of a simple test
+case when a single device is tested with on-network commissioning, nothing
+special is needed - the controller can connect to the device directly over the
+local (loopback) network interface. However, for more complex test cases that
+involve multiple devices or other than on-network commissioning (e.g. ble-wifi
+or ble-thread), some additional setup is needed.
+
+### Running tests in Linux network namespaces
+
+The simplest way to mock more complex network topologies is to use Linux network
+namespaces. Each device (controller or DUT) is run in its own network namespace,
+which allows them to have their own network interfaces and corresponding IP
+addresses.
+
+For convenience, there is a script that can set up network namespaces and run a
+test case in them:
+
+```shell
+# Build the chip-tool and chip-all-clusters-app if not done already
+scripts/build/build_examples.py --target linux-x64-chip-tool --target linux-x64-all-clusters build
+# Run the TestOperationalState test case in the Linux network namespaces
+scripts/tests/run_test_suite.py --runner chip_tool_python \
+    --chip-tool out/linux-x64-chip-tool/chip-tool \
+    --target TestOperationalState \
+    --log-level=debug \
+    run
+```
+
+### Running tests with mocked BLE and Wi-Fi connectivity
+
+For more complex commissioning flows that involve BLE and Wi-Fi, the test runner
+needs to mock BLE and Wi-Fi connectivity as well. On Linux, BLE and Wi-Fi are
+provided by the BlueZ stack and WPA supplicant, respectively. The SDK uses D-Bus
+to interact with these services. This allows the test runner to mock BLE and
+Wi-Fi connectivity by simply mocking used D-Bus APIs of these services.
+
+Additionally, the Matter specification forbids device to advertise on more than
+one network type at a time, so in case of ble-wifi commissioning, the DUT shall
+not be accessible over Wi-Fi until it is commissioned. This can be achieved by
+using Linux network namespaces as described above, but instead of setting up
+network interfaces before commissioning, the test runner assigns IP address to
+the DUT's network interface only after mock Wi-Fi connection is associated.
+
+See the diagram below for an overview of the setup:
+
+```mermaid
+flowchart TD
+
+    subgraph Mocked BlueZ
+        BA1[adapter1<br>00:00:00:11:11:11]
+        BA2[adapter2<br>00:00:00:22:22:22]
+    end
+
+    subgraph Mocked WPA supplicant
+        WL0[wlan0]
+    end
+
+    subgraph Test D-Bus system bus
+        direction TB
+        org.bluez.hci0[org.bluez.Adapter<br>hci0]
+        org.bluez.hci1[org.bluez.Adapter<br>hci1]
+        fi.w1[fi.w1.wpa_supplicant1.Interface<br>wlan0]
+    end
+
+    BA1 --- org.bluez.hci0
+    BA2 --- org.bluez.hci1
+    WL0 --- fi.w1
+
+    subgraph ETH["TOOL network namespace"]
+        CONTROLLER[chip-tool]
+        ETH0[eth0<br>fd00:0:1:1::2]
+    end
+
+    subgraph WLAN["DUT network namespace"]
+        DUT[chip-all-clusters-app]
+        WLAN0[wlan0<br>not assigned]
+    end
+
+    fi.w1 --- DUT
+    org.bluez.hci0 --- DUT
+    org.bluez.hci1 --- CONTROLLER
+
+```
+
+In order to run tests with mocked BLE and Wi-Fi connectivity and Linux network
+namespaces use the `--ble-wifi` option to the `run` command of the
+`scripts/tests/run_test_suite.py` script:
+
+```shell
+# Run the TestOperationalState test case with ble-wifi commissioning
+scripts/tests/run_test_suite.py --runner chip_tool_python \
+    --chip-tool out/linux-x64-chip-tool/chip-tool \
+    --target TestOperationalState \
+    --log-level=debug \
+    run --ble-wifi
+```

docs/testing/python.md

@@ -45,7 +45,7 @@ Python tests located in src/python_testing
 
 ### A simple test
 
-```
+```python
 # See https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/python.md#defining-the-ci-test-arguments
 # for details about the block below.
 #
@@ -71,18 +71,16 @@ class TC_MYTEST_1_1(MatterBaseTest):
     async def test_TC_MYTEST_1_1(self):
 
         vendor_name = await self.read_single_attribute_check_success(
-            dev_ctrl=self.default_controller, <span style="color:#38761D"># defaults to
-self.default_controlller</span>
-            node_id = self.dut_node_id, <span style="color:#38761D"># defaults to
-self.dut_node_id</span>
+            dev_ctrl=self.default_controller,  # defaults to self.default_controller
+            node_id = self.dut_node_id,  # defaults to self.dut_node_id
             cluster=Clusters.BasicInformation,
             attribute=Clusters.BasicInformation.Attributes.VendorName,
-            endpoint = 0, <span style="color:#38761D">#defaults to 0</span>
+            endpoint = 0,  # defaults to 0
         )
-        asserts.assert_equal(vendor_name, “Test vendor name”, “Unexpected vendor name”)
+        asserts.assert_equal(vendor_name, "Test vendor name", "Unexpected vendor name")
 
 if __name__ == "__main__":
-default_matter_test_main()
+    default_matter_test_main()
 ```
 
 ---
@@ -321,7 +319,7 @@ callbacks are called on update.
 
 Example for setting callbacks:
 
-```
+```python
 cb = EventSubscriptionHandler(cluster, cluster_id, event_id)
 
 urgent = 1
@@ -345,9 +343,9 @@ PyChipError
 
 Example:
 
-```
+```python
 res = await devCtrl.WriteAttribute(nodeid=0, attributes=[(0,Clusters.BasicInformation.Attributes.NodeLabel("Test"))])
-asserts.assert_equal(ret[0].status, Status.Success, “write failed”)
+asserts.assert_equal(ret[0].status, Status.Success, "write failed")
 ```
 
 ### [SendCommand](./ChipDeviceCtrlAPI.md#sendcommand)
@@ -532,11 +530,11 @@ Note: The name of the pipe can be anything while is a valid file path.
 
 Example of usage:
 
-```bash
-First run the app with the desired app-pipe path:
+```shell
+# First run the app with the desired app-pipe path:
 ./out/darwin-arm64-all-clusters/chip-all-clusters-app --app-pipe  /tmp/ref_alm_2_2
 
-Then execute the test with the app-pipe argument with the value defined while running the app.
+# Then execute the test with the app-pipe argument with the value defined while running the app.
 python3 src/python_testing/TC_REFALM_2_2.py --commissioning-method on-network --qr-code MT:-24J0AFN00KA0648G00  --PICS src/app/tests/suites/certification/ci-pics-values --app-pipe /tmp/ref_alm_2_2  --int-arg PIXIT.REFALM.AlarmThreshold:1
 ```
 
@@ -682,15 +680,15 @@ First ensure the device is in commissioning mode. Then run the test against the
 device by supplying either the QR code, the manual pairing code or the
 discriminator / password pair.
 
-```
+```shell
 python3 TC_DeviceConformance.py --qr-code MT:-24J0AFN00KA0648G00
 ```
 
-```
+```shell
 python3 TC_DeviceConformance.py --manual-code 34970112332
 ```
 
-```
+```shell
 python3 TC_DeviceConformance.py --discriminator 3840 --passcode 20202021
 ```
 
@@ -699,7 +697,7 @@ python3 TC_DeviceConformance.py --discriminator 3840 --passcode 20202021
 If the device has already been commissioned into the python testing fabric, you
 can run the test directly
 
-```
+```shell
 python3 TC_DeviceConformance.py
 ```
 
@@ -712,7 +710,7 @@ using the `--commissioning-method` flag and the `--qr-code`, `--manual-code` or
 
 For example:
 
-```
+```shell
 python3 TC_DeviceConformance.py --discriminator 3840 --passcode 20202021 --commissioning-method on-network
 ```
 
@@ -736,15 +734,15 @@ the device, they can also be run against a MatterTlvJson machine readable file
 without requiring the device to be physically present. To run against a
 previously generated MatterTlvJson device dump file:
 
-```
+```shell
 python3 TC_DeviceConformance.py --string-arg test_from_file:device_dump_0xFFF1_0x8001_1.json
 ```
 
 You can generate a MatterTlvJson file for a device by leveraging the
 certification test that generates these (TC-IDM-12.1, implemented in
 TC_DeviceBasicComposition.py).
 
-```
+```shell
 python3 TC_DeviceBasicComposition.py --qr-code MT:-24J0AFN00KA0648G00 --tests test_TC_IDM_12_1
 ```