The certification_status_overrides in the sub test plan do not apply to the top level test plan

[clairlin53]

Bug Description

When a test plan (Parent) includes another test plan (Child) via nested_part, the certification_status_overrides defined in the Child plan are ignored during the execution of the Parent plan.
To make a job a "blocker," the override must be duplicated in the Parent plan, which leads to maintenance overhead and potential configuration errors across large certification suites.

Cert-blocker Test Case

• cert-blocker

To Reproduce

1. Create a Sub Test Plan (sub-plan) with a certification_status_overrides setting a specific job to blocker.
   For example:

id: audio-base-26-04-automated
unit: test plan
_name: Audio for 26.04 (Automated)
_description: Automated tests for audio
include:
nested_part:
    audio-cert-automated
    after-suspend-audio-cert-automated
certification_status_overrides:
    apply blocker to com.canonical.certification::audio/detect_sinks
    apply blocker to com.canonical.certification::audio/detect_sources
    apply blocker to com.canonical.certification::audio/alsa_record_playback_automated
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sinks
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sources
    apply blocker to com.canonical.certification::after-suspend-audio/alsa_record_playback_automated

2. Execute checkbox-cli run the client-cert-desktop-26-04-automated test plan
3. Observe the status of the job defined in the Sub Plan's override.

Expected Result

The job should inherit the blocker status defined in the nested Sub Plan.

Actual Result

The job retains its original status unless the override is explicitly redefined in the Top Plan.

Environment

• checkbox version: 7.1.0.dev100
• CID: 202512-38201

Relevant log output

Additional context

sub_audio_submission.html

26_Auto_testplan_submission.html

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-2234.

This message was autogenerated

[pieqq]

I'm not sure I understand the issue here very well.

While developing a fix for #2480 (see #2488 ), I've tested the following:

Create two test plans:

id: pieq-base-26-04
_name: PIEQ test plan 26.04
unit: test plan
_description: Test cert blocker override
include:
nested_part:
    pieq-base
certification_status_overrides:
    apply non-blocker to com.canonical.certification::acpi/oem_osi

id: pieq-base
_name: PIEQ test plan
unit: test plan
_description: Test cert blocker override
include:
    com.canonical.certification::acpi/oem_osi certification_status=blocker

Before applying my patch:

$ checkbox-cli  expand com.canonical.certification::pieq-base -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

$ checkbox-cli  expand com.canonical.certification::pieq-base-26-04 -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

After:

$ checkbox-cli  expand com.canonical.certification::pieq-base -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

$ checkbox-cli  expand com.canonical.certification::pieq-base-26-04 -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "non-blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

However, even on the current version of Checkbox:

• when running pieq-base-26-04, acpi/oem_osi is set to blocker in the JSON report and the HTML (as expected)
• when running pieq-base, acpi/oem_osi is set to blocker in the JSON report and the HTML (as expected)

[clairlin53]

I'm not sure I understand the issue here very well.

While developing a fix for #2480 (see #2488 ), I've tested the following:

Create two test plans:

id: pieq-base-26-04
_name: PIEQ test plan 26.04
unit: test plan
_description: Test cert blocker override
include:
nested_part:
    pieq-base
certification_status_overrides:
    apply non-blocker to com.canonical.certification::acpi/oem_osi

id: pieq-base
_name: PIEQ test plan
unit: test plan
_description: Test cert blocker override
include:
    com.canonical.certification::acpi/oem_osi certification_status=blocker

Before applying my patch:

$ checkbox-cli  expand com.canonical.certification::pieq-base -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

$ checkbox-cli  expand com.canonical.certification::pieq-base-26-04 -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

After:

$ checkbox-cli  expand com.canonical.certification::pieq-base -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

$ checkbox-cli  expand com.canonical.certification::pieq-base-26-04 -f json | jq
[
  {
    "_purpose": " This checks if the deprecated OEM _OSI strings are still used by checking the ACPI DSDT and SSDT tables",
    "_summary": "Test ACPI OEM _OSI strings",
    "category_id": "acpi",
    "certification-status": "non-blocker",
    "command": "acpi_tests.sh",
    "estimated_duration": "1",
    "id": "com.canonical.certification::acpi/oem_osi",
    "plugin": "shell",
    "requires": "cpuinfo.platform in (\"i386\", \"x86_64\")",
    "unit": "job",
    "user": "root"
  }
]

However, even on the current version of Checkbox:

• when running pieq-base-26-04, acpi/oem_osi is set to blocker in the JSON report and the HTML (as expected)
• when running pieq-base, acpi/oem_osi is set to blocker in the JSON report and the HTML (as expected)

Hi @pieqq ,
Thanks for the reply. I think the confusion might be in the three-layer hierarchy I am using. In my testing, the overrides fail when they are nested two levels deep.
The Problem:
When I run the layer 2, the overrides apply correctly.
When I run the layer 3, the overrides defined in the layer 2 are ignored, even though the layer 2 is included as a nested_part.

Layer 1: Job List

id: audio-cert-automated
unit: test plan
_name: Audio tests (automated)
_description:
 Audio tests (automated)
include:
    audio/detect_sinks
    audio/alsa_record_playback_automated
    audio/alsa_info_collect
    audio/alsa_info_attachment
    audio/list_devices
    audio/valid-sof-firmware-sig

Layer 2: Feature Plan

id: audio-base-26-04-automated
unit: test plan
_name: Audio for 26.04 (Automated)
_description: Automated tests for audio
include:
nested_part:
    audio-cert-automated
    after-suspend-audio-cert-automated
certification_status_overrides:
    apply blocker to com.canonical.certification::audio/detect_sinks
    apply blocker to com.canonical.certification::audio/detect_sources
    apply blocker to com.canonical.certification::audio/alsa_record_playback_automated
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sinks
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sources
    apply blocker to com.canonical.certification::after-suspend-audio/alsa_record_playback_automated

Layer 3: Top Level Plan >> execute the test plan

id: client-cert-desktop-26-04-automated
_name: Desktop Preload Certification Tests for 26.04 (Automated Tests)
unit: test plan
_description: 
 Includes tests that do not require operator input.
include:
    recovery_info_attachment
nested_part:
    submission-cert-automated
    info-attachment-cert-automated
    audio-base-26-04-automated

[pieqq]

I cannot reproduce this issue with the latest Checkbox version in main.

In providers/base/units/audio/test-plan.pxu, I make the following changes:

id: audio-base-26-04-automated
unit: test plan
_name: Audio for 26.04 (Automated)
_description: Automated tests for audio
include:
nested_part:
    audio-cert-automated
    after-suspend-audio-cert-automated
certification_status_overrides: # <-------------------- add this section, which is currently not present
    apply blocker to com.canonical.certification::audio/detect_sinks
    apply blocker to com.canonical.certification::audio/detect_sources
    apply blocker to com.canonical.certification::audio/alsa_record_playback_automated
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sinks
    apply blocker to com.canonical.certification::after-suspend-audio/detect_sources
    apply blocker to com.canonical.certification::after-suspend-audio/alsa_record_playback_automated


id: pieq-audio-26-04 # <--------------------- Create this test plan, which nests audio-base-26-04-automated
unit: test plan
_name: PIEQ audio 26.04
include:
nested_part:
    audio-base-26-04-automated

Then, either with expand or list-bootstrapped, I export both test plans:

$ checkbox-cli expand --format json audio-base-26-04-automated | jq > /var/tmp/audio-base-26-04-automated-override.json

$ checkbox-cli expand --format json pieq-audio-26-04 | jq > /var/tmp/audio-base-26-04-automated-pieq.json

$ checkbox-cli list-bootstrapped --format json audio-base-26-04-automated | jq > /var/tmp/list_bootstrap_audio-base-26-04-automated.json

$ checkbox-cli list-bootstrapped --format json pieq-audio-26-04 | jq > /var/tmp/list_bootstrap_pieq-audio-26-04.json

I then compare each, and I found no differences:

With expand:

#   "certification-status": "blocker",
$ jq '.[] | select(.id == "com.canonical.certification::audio/detect_sinks")' /var/tmp/audio-base-26-04-automated-override.json
{
  "_purpose": "Test to detect if there are available sinks",
  "_summary": "Ensure audio sinks are available for detection.",
  "category_id": "com.canonical.plainbox::audio",
  "certification-status": "blocker",
  "command": " if check_audio_daemon.sh ; then\n   checkbox-support-pipewire-utils detect -t audio -c sinks\n else\n   pactl_list.sh sinks\n fi",
  "estimated_duration": "1.0",
  "flags": "also-after-suspend",
  "id": "com.canonical.certification::audio/detect_sinks",
  "imports": "from com.canonical.plainbox import manifest",
  "plugin": "shell",
  "requires": " package.name in ['pulseaudio-utils', 'pipewire']\n manifest.has_audio_playback == 'True'",
  "unit": "job"
}

#   "certification-status": "blocker",
$ jq '.[] | select(.id == "com.canonical.certification::audio/detect_sinks")' /var/tmp/audio-base-26-04-automated-pieq.json 
{
  "_purpose": "Test to detect if there are available sinks",
  "_summary": "Ensure audio sinks are available for detection.",
  "category_id": "com.canonical.plainbox::audio",
  "certification-status": "blocker",
  "command": " if check_audio_daemon.sh ; then\n   checkbox-support-pipewire-utils detect -t audio -c sinks\n else\n   pactl_list.sh sinks\n fi",
  "estimated_duration": "1.0",
  "flags": "also-after-suspend",
  "id": "com.canonical.certification::audio/detect_sinks",
  "imports": "from com.canonical.plainbox import manifest",
  "plugin": "shell",
  "requires": " package.name in ['pulseaudio-utils', 'pipewire']\n manifest.has_audio_playback == 'True'",
  "unit": "job"
}

With list-bootstrapped:

#   "certification_status": "blocker"
$ jq '.[] | select(.id == "audio/detect_sinks")' /var/tmp/list_bootstrap_audio-base-26-04-automated.json
{
  "plugin": "shell",
  "category_id": "com.canonical.plainbox::audio",
  "id": "audio/detect_sinks",
  "flags": "also-after-suspend",
  "estimated_duration": "1.0",
  "imports": "from com.canonical.plainbox import manifest",
  "requires": "package.name in ['pulseaudio-utils', 'pipewire']\nmanifest.has_audio_playback == 'True'",
  "command": "if check_audio_daemon.sh ; then\n  checkbox-support-pipewire-utils detect -t audio -c sinks\nelse\n  pactl_list.sh sinks\nfi",
  "_purpose": "Test to detect if there are available sinks",
  "_summary": "Ensure audio sinks are available for detection.",
  "full_id": "com.canonical.certification::audio/detect_sinks",
  "certification_status": "blocker"
}

#   "certification_status": "blocker"
$ jq '.[] | select(.id == "audio/detect_sinks")' /var/tmp/list_bootstrap_pieq-audio-26-04.json 
{
  "plugin": "shell",
  "category_id": "com.canonical.plainbox::audio",
  "id": "audio/detect_sinks",
  "flags": "also-after-suspend",
  "estimated_duration": "1.0",
  "imports": "from com.canonical.plainbox import manifest",
  "requires": "package.name in ['pulseaudio-utils', 'pipewire']\nmanifest.has_audio_playback == 'True'",
  "command": "if check_audio_daemon.sh ; then\n  checkbox-support-pipewire-utils detect -t audio -c sinks\nelse\n  pactl_list.sh sinks\nfi",
  "_purpose": "Test to detect if there are available sinks",
  "_summary": "Ensure audio sinks are available for detection.",
  "full_id": "com.canonical.certification::audio/detect_sinks",
  "certification_status": "blocker"
}

Can you try on your side, or provide step-by-step instructions on how to reproduce this issue?