[Bug] [SVE 1.6] Tests commissions but does not continue to actual test

[BryanULSilva]

Describe the bug

Running test cases TC-IDM-10.X via CLI, the harness completes commissioning but does not continue to running the test and exiting with error with no error message.

th-cli run-tests --tests-list TC_IDM_10_1 --project-id 1 -c /home/ubuntu/sample_config_files/TC-IDM-10.1.json --title TC-IDM-10_1

Steps to reproduce the behavior

1. Install harness based on SVE user manual steps on RP4
2. Side loaded the TC_DeviceConformance.py
3. Ran TC-IDM-10.X test cases in similar fashion as described above

Expected behavior

No response

Log files

TC-IDM-10_1.txt

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[rquidute]

Hi @BryanULSilva It seems the test results from the SDK container never reach the backend, which is why the CLI shows "failed but has no error messages". This is likely due to an RPC connection failure on port 50000 between the SDK container and the backend, possibly caused by the slower startup time on the Raspberry Pi 4.

You should still be able to see the full test output in this file:
backend/test_collections/matter/sdk_tests/sdk_checkout/python_testing/test_output.txt

Could you confirm if that file contains the execution logs? Also, would it be possible to test this on a Raspberry Pi 5, just to check if the results are the same?

[BryanULSilva]

Hi @rquidute yes I am able to see the logs in backend/test_collections/matter/sdk_tests/sdk_checkout/python_testing/test_output.txt

I have not been able to reproduce the issue in RP5

[rquidute]

Thanks for confirming @BryanULSilva! Good to know the logs are accessible there and that the issue doesn't reproduce on the Raspberry Pi 5. We'll look into adding a better error message or fallback for Raspberry Pi 4 users so the failure reason is clearer.

[rquidute]

Maybe we should wait for issue #944 to be fixed before testing again to see if it works.

[KishokG]

@rquidute do you think we need this issue to be fixed for 1.6 final release? Seems this issue noticed only in RPI4. QA also executed in RPI4, but we couldn't able to reproduce it.

[rquidute]

Hi @KishokG, I agree. I think we can target this for the next version.