Skip to content

process: adjust testing levels and traceability #823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

process: adjust testing levels and traceability #823

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
907 changes: 903 additions & 4 deletions ...rocess/general_concepts/_assets/score_traceability_model_wp_overview.drawio.svg
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what is still unclear: are the component integration tests 6-9 or 6-10? In the SW verification plan it seems to say that "level 1" is 6-9. Is this what you want to say? And I also think we need to explain better what a "Unit Tester" has to do in the guideline: 1. read the Detailed Desig, create a test for every interface of the unit showing at least every flow in dynamic diagrams, 2. follow the DD to the component requirements - test these 3. fill attributes including test description 4. link test to ??? (not component requirements)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From WG working meeting 6-10 goes down to green "Component".
6-9 is for grey part.
Unclear is the handling of requirements.
Start with the simple use case (without Component architecture due to unit simplicity)
Detailed design is directly related to component requirements and implicitly covered.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking if we not just also add a link from Unit Test to the Component Requirements. Reasoning is that this would be also easier to demonstrate a full requirement coverage by testing. But also we should keep the DD link possibility for unit test because we do want also the possibility to test DD instead have a Component requirement for everything - maybe best to say you have to have one or the other.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/process/general_concepts/score_traceability_concept.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ High level traceability overview

The following figure shows the traceability between the major **S-CORE** work products on each
requirements level. Starting from top, the platform level, going down to feature, component
to the bottom the unit level. The concept is based on the classical V-Cylce approach.
to the bottom the unit level. The concept is based on the classical V-model approach.

Change request are traced to all affected work products.

Expand Down
2 changes: 1 addition & 1 deletion ...process_areas/safety_management/guidance/guideline_component_classification.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ This is namely (for ASIL B) to provide the following work products according to

* :need:`wp__requirements__comp` including their inspection
* :need:`wp__requirements__comp_aou` derived from the OSS components user manual and interface description, includes specification of the component's configuration
* :need:`wp__verification__component_test` to test requirement and AoU implementation
* :need:`wp__verification__comp_int_test` to test requirement and AoU implementation

Integration of the OSS component is performed via the modules's SW build config and checked by feature integration tests (component integration if the OSS element is considered as a sub-component).

Expand Down
6 changes: 1 addition & 5 deletions ...rocess/process_areas/safety_management/guidance/template_module_safety_plan.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ This document implements <add "need" link>
- <Link to WP>
- <automated>

* - :need:`wp__verification__component_test`
* - :need:`wp__verification__comp_int_test`
- <Link to process>
- <automated>
- <Link to issue>
Expand Down Expand Up @@ -291,10 +291,6 @@ This document implements <add "need" link>
- <Reasoning for tailoring, can improve deficits in process Id 6 and complexity Id 2>

* - :need:`wp__verification__comp_int_test`
- <Link to issue>
- <Reasoning for tailoring, can improve deficits in process Id 6 and complexity Id 3&5>

* - :need:`wp__verification__component_test`
- <Link to issue>
- Always needed (for Q and QR classification)

Expand Down
27 changes: 5 additions & 22 deletions docs/process/process_areas/verification/workflows.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,44 +38,27 @@ Workflow Verification
The actual :need:`rl__committer` of the implementation can also be the creator of the unit tests.
Independence is achieved by different approver at PRs and by the :need:`wp__verification__module_ver_report`.

.. workflow:: Create/Maintain Component Test
:id: wf__verification__comp_test
:status: valid
:tags: verification
:responsible: rl__contributor
:approved_by: rl__committer, rl__testing_community
:supported_by: rl__safety_manager
:input: wp__requirements__comp, wp__requirements__comp_aou, wp__verification__plan
:output: wp__verification__component_test
:contains: gd_req__link_tests, gd_guidl__verification_specification
:has: doc_concept__verification__process, doc_getstrt__verification__process

Component test cases are based on component requirements.
Any contributor can create a component test and create a PR for it.
During the review process the test cases will be approved by a committer.
Committer and contributor need to differ.
The tests are automatically executed as part of the CI after PR merge.
In case of changes at inputs, the workflow need to be executed again as part of maintenance.

.. workflow:: Create/Maintain Component Integration Test
:id: wf__verification__comp_int_test
:status: valid
:tags: verification
:responsible: rl__contributor
:approved_by: rl__committer, rl__testing_community
:supported_by: rl__safety_manager
:input: wp__component_arch, wp__sw_implementation, wp__verification__plan
:input: wp__component_arch, wp__sw_implementation, wp__requirements__comp, wp__requirements__comp_aou, wp__verification__plan
:output: wp__verification__comp_int_test
:contains: gd_req__link_tests, gd_guidl__verification_specification
:has: doc_concept__verification__process, doc_getstrt__verification__process

Component Integration test cases are based on component architecture and detailed design of implementation.
Component Integration test cases are based on component architecture and component requirements.
They may optionally cover the detailed design of implementation as an optional element where units are seen complex.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not say so. The detailed design describes the the units forming a component and their interactions - see https://eclipse-score.github.io/score/main/process/process_areas/implementation/_assets/detailed_design_example.html - so the interfaces and interactions between units can be used as a basis for testing. Maybe rather make the integration testing of component architecture optional (in case a component has no (sub-)components.

Any contributor can create a component integration test and create a PR for it.
During the review process the test cases will be approved by a committer.
Committer and contributor need to differ.
The tests are automatically executed as part of the CI after PR merge.
In case of changes at inputs, the workflow need to be executed again as part of maintenance.


.. workflow:: Create/Maintain Feature Integration Test
:id: wf__verification__feat_int_test
:status: valid
Expand Down Expand Up @@ -165,7 +148,7 @@ Workflow Verification
wp__component_arch, wp__module_sw_release_note, wp__platform_mgmt,
wp__sw_component_safety_analysis, wp__sw_component_dfa,
wp__sw_arch_verification, wp__sw_implementation_inspection, wp__requirements__inspect,
wp__verification__component_test, wp__verification__comp_int_test, wp__verification__sw_unit_test
wp__verification__comp_int_test, wp__verification__sw_unit_test
:output: wp__verification__module_ver_report
:contains: gd_temp__mod_ver_report
:has: doc_concept__verification__process, doc_getstrt__verification__process
Expand Down
14 changes: 6 additions & 8 deletions docs/process/process_areas/verification/workproducts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,26 +91,24 @@ Module
Component
*********

.. workproduct:: Component test
:id: wp__verification__component_test
:status: valid
:complies: std_wp__iso26262__software_1051, std_wp__iso26262__support_952, std_wp__isopas8926__4525

Component Testing verifies Component Requirements

.. workproduct:: Component Integration test
:id: wp__verification__comp_int_test
:status: valid
:complies: std_wp__iso26262__software_1051, std_wp__iso26262__support_952, std_wp__isopas8926__4525

Component Integration Testing verifies the detailed design and component architecture:
Component Integration Testing verifies the component architecture and component requirements:

- all interfaces from Static view and
- all flows from Dynamic View
- integration of units into components based on detailed design
- detailed design of complex units

Performance (i.e. RAM and processor usage) is only tested on reference HW.

As an optional part component integration tests can cover additional testing for complex units
touching specifically the detailed design. This is needed where :need:`wp__verification__sw_unit_test`
is not sufficient to cover the measures defined in the implementation of the :need:`wp__verification__plan`.

.. workproduct:: Unit test
:id: wp__verification__sw_unit_test
:status: valid
Expand Down