Waiver does not apply correctly with prebuilt v25.10_3.1.0

[RadxaYuntian]

I'm currently testing our upcoming product against ACS. Due to a known issue in FWTS, I want to apply waiver on this failed case.

This is the acs_waiver.json I came up with according to the documentation:

{
  "Suites": [
    {
      "Suite": "FWTS",
      "TestSuites": [
        {
          "TestCase": {
            "SubTests": [
              {
                "sub_Test_Description": "Test UEFI RT service set wakeup time interface.",
                "Reason": "Same reason as 1344-5: https://dl.radxa.com/orion/o6/images/bios/SystemReady/latest/Arm_SystemReady_SR_Waivers_Radxa_Computer_Shenzhen_Co_Ltd_Radxa_Orion_O6_1344.pdf"
              },
              {
                "sub_Test_Description": "MADT subtable tests.",
                "Reason": "A bug in fwts V25.01.00 cause false positive on disabled cores.\nThis has been fixed in later versions of fwts: https://github.com/fwts/fwts/commit/4098744507f832ee1417476f2c5ee118b54828c2"
              }
            ]
          }
        }
      ]
    }
  ]
}

For some reason I just keep getting the case marked as Failed instead of Failed with Waiver. I finally checked the console after I tried all possible combination I can come up with, and saw this:

Waivers will be applied using:
  Waiver File        : /mnt/acs_tests/config/acs_waiver.json

ERROR: Log file /mnt/acs_results/uefi/BsaResults.log is missing.
ERROR: Log file /mnt/acs_results/linux/BsaResultsKernel.log is missing.
WARNING: Log file /mnt/acs_results/uefi/SbsaResults.log is missing.
WARNING: Log file /mnt/acs_results/linux/SbsaResultsKernel.log is missing.
Processing /mnt/acs_results/fwts/FWTSResults.log file.
Traceback (most recent call last):
  File "/usr/bin/log_parser/apply_waivers.py", line 782, in <module>
    main()
  File "/usr/bin/log_parser/apply_waivers.py", line 779, in main
    apply_waivers(args.suite_name, args.json_file, args.waiver_file, args.output_json_file)
  File "/usr/bin/log_parser/apply_waivers.py", line 565, in apply_waivers
    if row.get("Suite", "").lower() == suite_name.lower() and row.get("Test Suite", "").lower() == test_suite_name.lower():
AttributeError: 'str' object has no attribute 'get'
ERROR: Log file /mnt/acs_results/sct_results/Overall/Summary.log is missing.
WARNING: Log file /mnt/acs_results/bbsr/fwts/FWTSResults.log is missing.
WARNING: Log file /mnt/acs_results/bbsr/sct_results/Overall/Summary.log is missing.
WARNING: Log file /mnt/acs_results/bbsr/tpm2/verify_tpm_measurements.log is missing.
ERROR: Log file /mnt/acs_results/sbmr/sbmr_in_band_logs/console.log is missing.
ERROR: Log file /mnt/acs_results/sbmr/sbmr_out_of_band_logs/console.log is missing.

Checking the code in apply_waivers.py seems to indicate there is issue with the test_category.json. However, I cannot find this file in the acs_results.

[chetan-rathore]

Hi @RadxaYuntian,

Can you please mention which band image you have used for testing. Share the link of pre-built image used.

[RadxaYuntian]

https://github.com/ARM-software/arm-systemready/releases/download/v25.10_SR_3.1.0/systemready_acs_live_image.img.xz.zip

[RadxaYuntian]

Here is a previous acs_summary result, and you can see acs_waiver.json was copied to config, but fwts_summary.html still have them as failure.

acs_summary.zip

[chetan-rathore]

Thanks @RadxaYuntian for all the details.

Waivers processing for FWTS is not enabled as its a mandatory test suite for compliance and expectation is it should pass.
I will discuss internally on the two FWTS failures you highlighted in your waiver.json, especially the MADT sub table failures which is limitation of fwts test suite.

Thanks,
Chetan

[RadxaYuntian]

The other timer failure is also related to the kernel 6.10 used in the ACS. It was fixed in a later kernel. That was mentioned in the included waiver, not the one in the issue body.