Change certification level status (BugFix)

[clairlin53]

Description

WARNING: This modifies com.canonical.certification::sru-server

Promoted to Focused Stability (Non-Blocker to Blocker-Ready):

These features have reached a stable state with broader DUT (Device Under Test) support. We are transitioning our primary focus toward these areas to ensure full compliance and reliability.

• audio/channels
• audio/external-linein
• audio/external-lineout
• power-management/light_sensor
• monitor/1_thunderbolt3_.*
• audio/1_playback_thunderbolt3_.*
• thunderbolt3/daisy-chain
• graphics/1_cycle_resolution_.*
• fingerprint/detect
• fingerprint/enroll
• fingerprint/verify-no-match
• fingerprint/verify-match
• fingerprint/unlock
• fingerprint/delete
• keys/fn-lock
• led/fn
• touchpad/palm-rejection
• touchscreen/multitouch-rotate
• usb-c/c-to-ethernet-adapter-insert
• audio/detect_sinks
• audio/alsa_record_playback_automated
• audio/detect_sources
• graphics/1_vkcube_.*
• graphics/2_vkcube_.*
• graphics/vulkaninfo.*
• wwan/check-sim-present-manufacturer-model-hw_id-auto
• wwan/detect
• wwan/check-sim-present-manufacturer-model-hw_id-auto
• usb/storage-detect
• usb/storage-preinserted-.*
• usb3/storage-preinserted-.*
• wireless/detect
• suspend/validate_suspend_status
• ethernet/iperf3_.*
• warm-boot-loop-.*
• cold-boot-loop-.*

Downgraded Priority:

Based on current SIG certification scopes, Eddystone is no longer a required certification item. Its priority has been lowered to Non-Blocker to optimize our testing bandwidth

• bluetooth4/beacon_eddystone_url_.*

Resolved issues

Documentation

Tests

26_Auto_submission.html
26_Manual_submission.html
26_Stress_submission.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.89%. Comparing base (06722a3) to head (730fe3e).
⚠️ Report is 22 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2424      +/-   ##
==========================================
+ Coverage   58.86%   58.89%   +0.03%     
==========================================
  Files         476      476              
  Lines       48007    48007              
  Branches     8570     8570              
==========================================
+ Hits        28260    28276      +16     
+ Misses      18855    18839      -16     
  Partials      892      892              

Flag	Coverage Δ
provider-base	34.10% <ø> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pieqq]

The modifications made here affect nested parts that are used for different versions of Ubuntu.

In particular, it modifies the certification status of 24.04 test plans as well, which means it will modify the Coverage Guides.

I ran checkbox-cli expand com.canonical.certification::client-cert-desktop-24-04 -f json | jq to get a view of the 24.04 test plan before and after applying the patch:

• client-cert-desktop-24-04_before.json
• client-cert-desktop-24-04_after.json

and indeed, all the jobs mentioned here get their certification-status changed.

@zongminl are we OK with this? It may have an impact on 24.04 projects and contractual requirements...

[clairlin53]

The modifications made here affect nested parts that are used for different versions of Ubuntu.

In particular, it modifies the certification status of 24.04 test plans as well, which means it will modify the Coverage Guides.

I ran checkbox-cli expand com.canonical.certification::client-cert-desktop-24-04 -f json | jq to get a view of the 24.04 test plan before and after applying the patch:

• client-cert-desktop-24-04_before.json
• client-cert-desktop-24-04_after.json

and indeed, all the jobs mentioned here get their certification-status changed.

@zongminl are we OK with this? It may have an impact on 24.04 projects and contractual requirements...

Hi @pieqq Thanks for the feedback.

1. If we want to define certification levels strictly by OS version, we would need to create specific plans like <feature>-cert-26-04-auto and after-suspend-<feature>-cert-26-04-auto or <feature>-cert-26-04-manual and after-suspend-<feature>-cert-26-04-manual to handle the blocker/non-blocker logic for 26.04. However, this would mean manually updating multiple files every time a job changes.

   For example, to handle a status change for just one OS version, we would have to maintain all of these:

id: touchpad-cert-26.04-manual
unit: test plan
_name: Touchpad tests (Manual)
_description: Touchpad tests (Manual)
include:
touchpad/basic                                 certification-status=blocker
touchpad/palm-rejection                        certification-status=blocker
touchpad/continuous-move                       certification-status=blocker
touchpad/singletouch-selection                 certification-status=blocker
touchpad/drag-and-drop                         certification-status=blocker
touchpad/multitouch-rightclick                 certification-status=blocker
touchpad/multitouch                            certification-status=blocker

id: after-suspend-touchpad-cert-26.04-manual
unit: test plan
_name: Touchpad tests after suspend (Manual)
_description: Touchpad tests after suspend (Manual)
include:
after-suspend-touchpad/basic                                 certification-status=blocker
after-suspend-touchpad/palm-rejection                        certification-status=blocker
after-suspend-touchpad/continuous-move                       certification-status=blocker
after-suspend-touchpad/singletouch-selection                 certification-status=blocker
after-suspend-touchpad/drag-and-drop                         certification-status=blocker
after-suspend-touchpad/multitouch-rightclick                 certification-status=blocker
after-suspend-touchpad/multitouch                            certification-status=blocker

id: touchpad-base-26-04-manual
unit: test plan
_name: Touchpad for 26.04 (Manual)
include:
nested_part:
submission-cert-full
Touchpad-cert-26.04-manual
after-suspend-touchpad-cert-26.04-manual
info-attachment-cert-full

id: touchpad-base-26-04-automated
unit: test plan
_name: Touchpad for 26.04 (Automated)
_description: Automated tests for touchpad
include:
nested_part:
touchpad-cert-26.04-automated
After-suspend-touchpad-cert-26.04-automated

2. Alternatively, could you provide or design a mechanism within the framework that allows us a way to fill blocker/non-blocker settings dynamically? This would avoid the need to duplicate and maintain separate test plans for every OS release.

[tomli380576]

Just to add to Clair's comment, the "fill blocker/non-blocker settings dynamically" part could look something like this:

Instead of

id: cpu-cert-automated
unit: test plan
_name: CPU tests (automated)
_description: CPU tests (automated)
include:
    cpu/cstates     certification-status=blocker

We could have certification-blocker-versions=focal,jammy,noble

id: cpu-cert-automated
unit: test plan
_name: CPU tests (automated)
_description: CPU tests (automated)
include:
    cpu/cstates     certification-blocker-versions=focal,jammy,noble

or flip it:

id: cpu-cert-automated
unit: test plan
_name: CPU tests (automated)
_description: CPU tests (automated)
include:
    cpu/cstates     certification-non-blocker-versions=focal,jammy,noble

if the current code name is not included in certification-blocker-versions, assume non-blocker. Some additional syntax is probably necessary to describe cases that are blocker/non-blocker by default, but this is the basic idea.

[Hook25]

The feature would be quite complicated to implement and have a consistent semantic about, so my vote is on no.
Why don't we use an alias here:

id: <feature>-cert-26.04-manual
unit: test plan
_name: Feature tests (Manual)
_description: Feature tests (Manual)
include:
nested_part:
    <feature>-cert-24.04-manual
certification-status-overrides:
    apply blocker to .*::.*

[clairlin53]

The feature would be quite complicated to implement and have a consistent semantic about, so my vote is on no. Why don't we use an alias here:

id: <feature>-cert-26.04-manual
unit: test plan
_name: Feature tests (Manual)
_description: Feature tests (Manual)
include:
nested_part:
    <feature>-cert-24.04-manual
certification-status-overrides:
    apply blocker to .*::.*

@Hook25
I've updated with certification_status_overrides, please help to review.
Besides, I found the two issues regarding how test plan overrides are handled in the current structure:

1. Overrides defined in a sub-test plan (e.g., audio-base-26-04-automated) do not affect the results when run via a top-level plan (e.g., client-cert-desktop-26-04-automated). I have filed a bug The certification_status_overrides in the sub test plan do not apply to the top level test plan #2479.

2. Even when I manually add the certification_status_overrides directly to the top-level plan as a workaround for 2479, non-blocker statuses are still not being honored for the top test plan. Interestingly, if I execute the sub-test plan individually, the non-blocker status applies perfectly. Here is the bug The certification_status_overrides fails to set non-blocker status in the top test plan #2480

[pieqq]

For the sake of testing, I ran client-cert-desktop-26-04-automated after removing the certification_status_overrides section, selected a bunch of jobs that have had their certification status overridden to "blocker", and I got this:

submission_2026-04-24T13.12.10.561875.html

So the blocker override works for me even when it's not set at the top-level.

Therefore, I think you can remove them at top-level sections, but keep them in the smaller nested parts.

See my comments below.

[pieqq]

As I've explained in #2479 (comment), I've ran checkbox-cli expand --format json on this test plan with and without this section, and I get the same result.

When using list-bootstrapped, it's a bit more complicated to parse, but I basically get the same thing (at least, the few jobs I checked all had blocker for the certification status).

So I don't think this is needed.

[pieqq]

Same as above comment. Probably not needed, unless you can provide some clear steps to reproduce in #2479

[pieqq]

See my comment below, but I don't think this is needed, unless you can provide a way to reproduce in #2479.

[clairlin53]

See my comment below, but I don't think this is needed, unless you can provide a way to reproduce in #2479.

"Hi @pieqq,

I've updated the PR to remove the overrides from the top-level 26.04.pxu.

Interestingly, I encountered a blocking behavior in my previous test where the feature test plan overrides didn't seem to be applied by the top plan.
I re-verified the test in a clean environment that the overrides in the feature test plan are working as expected without adding override in the top level test plan.

I've removed the redundant code. Please help to review again. Thanks!