tests: add feature tagging workflow #15148

maykathm · 2025-02-27T15:32:55Z

Dependent on feature tag logging in #15091

This adds the ability to generate feature tags for a spread run in two ways:

Using the workflow dispatch in the new Feature Tagging workflow
Pushing on master (will only tag spread tests that have been changed)
Using run-spread with SPREAD_TAG_FEATURES set

Both methods permit automatically rerunning spread and consolidating the ending results into system-by-system json files of features. In the workflow, the results are uploaded as an artifact, though this is a temporary measure since in the future the tags will instead be loaded into a DB elsewhere.

Both the run-spread additions and the workflow contain a rerunning mechanism to automatically rerun spread with the failed tests a maximum number of times. Once the workflow/run-spread script has finished all spread runs, including reruns, it will consolidate the feature tagging results into a single final json file for each system. It consolidates results by creating a final version with all tests run where more recent individual test results substitute the less recent.

github-actions · 2025-02-27T15:42:31Z

Fri Mar 21 11:00:02 UTC 2025

Spread tests skipped

codecov · 2025-02-28T16:09:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.10%. Comparing base (a272aac) to head (49a5046).
Report is 206 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15148      +/-   ##
==========================================
+ Coverage   78.07%   78.10%   +0.02%     
==========================================
  Files        1182     1191       +9     
  Lines      157743   159148    +1405     
==========================================
+ Hits       123154   124297    +1143     
- Misses      26943    27118     +175     
- Partials     7646     7733      +87

Flag	Coverage Δ
unittests	`78.10% <ø> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

.github/workflows/feature-tagging.yaml

.github/workflows/rerun.yaml

run-spread

tests/lib/compose-features.py

…sing '\' to using '--' to reflect changes in debug tagging PR

zyga

First pass. I'm kind of -1 on the python that looks more like perl in the sense that everything is a dict. Can we avoid that and actually use objects of some type (typedict is a hack but passable). If loading json is a problem we can sync to find a solution.

zyga · 2025-03-19T14:03:54Z

tests/lib/spread/rules/feature-tagging.yaml

@@ -0,0 +1,18 @@
+rules:


Can you please start with a comment that links to an explanation of what this file is for and what reads it?

zyga · 2025-03-19T14:04:58Z

tests/lib/compose-features.py

+    for system in systems:
+        composed = compose_system(dir=args.dir, system=system,
+                                  failed_tests=args.failed_tests, env_variables=args.env_variables)
+        system = "_".join(system.split(':'))


You can just replace:

Suggested change

system = "_".join(system.split(':'))

system = system.replace(":", "_")

zyga · 2025-03-19T14:05:33Z

tests/lib/compose-features.py

+                                  failed_tests=args.failed_tests, env_variables=args.env_variables)
+        system = "_".join(system.split(':'))
+        with open(os.path.join(args.output, system + attempt + '.json'), 'w') as f:
+            f.write(json.dumps(composed))


Or you can avoid the intermediate:

Suggested change

f.write(json.dumps(composed))

json.dump(composed, f)

zyga · 2025-03-19T14:06:32Z

tests/lib/compose-features.py

+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description="""


Maybe move the description variable above main to make this format with less indent?

zyga · 2025-03-19T14:06:50Z

tests/lib/compose-features.py

+def run_attempt_type(value):
+    if value is not int or int(value) <= 0:
+        raise argparse.ArgumentTypeError(
+            "%s is invalid. Run attempts are integers and start at 1" % value)


% is somewhat discouraged, perhaps, f-string or just plain .format?

zyga · 2025-03-19T14:08:23Z

tests/lib/compose-features.py

+        '_1') and any(rerun for rerun in reruns_no_ext if rerun.startswith(file[:-2]))]
+    reruns_no_ext.sort(key=lambda x: int(x.split('_')[-1]))
+    for rerun in reruns_no_ext:
+        beginning = '_'.join(rerun.split('_')[:-1])


This wants to be a helper but I'm not sure how to call it

zyga · 2025-03-19T14:09:41Z

tests/lib/compose-features.py

+    '''
+    files = [f for f in os.listdir(
+        dir) if os.path.isfile(os.path.join(dir, f))]
+    systems = [":".join(file.split(':')[:2])


Could this be a helper that returns a namedtuple and the exepression here grabs the system attribute?

Can you make the whole thing a set comprehension please?

zyga · 2025-03-19T14:10:11Z

tests/lib/compose-features.py

+    return original_name, suite_name, test_name, variant_name
+
+
+def _compose_test(dir: str, file: str, failed_tests: str) -> dict:


You know you want to use typed dict?

zyga · 2025-03-19T14:11:13Z

run-spread

+if [ -z "$SPREAD_TAG_FEATURES" ]; then
+    SPREAD_USE_PREBUILT_SNAPD_SNAP=true exec spread "$@"
+else
+    WRITE_DIR="/tmp/features"


Hmm, can we use a temp dir?

Since this would be for personal use, I thought tmp would be the best option.

zyga · 2025-03-19T14:11:29Z

run-spread

+    done
+
+    ./tests/lib/compose-features.py \
+        --dir ${WRITE_DIR}/composed-feature-tags \


Can we shellcheck the file and shfmt the text later?

maykathm added 2 commits February 27, 2025 16:05

tests: add feature-tagging workflow

1471393

tests: add workflow rerun in case of failure

45ccafc

maykathm added the Skip spread Indicate that spread job should not run label Feb 27, 2025

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch 2 times, most recently from 3d46364 to 34ed733 Compare February 28, 2025 15:49

maykathm added 2 commits February 28, 2025 17:12

tests: add feature-tag consolidation if spread task has been rerun

feef96e

tests: added feature tagging on master pushes of only changed tests

44ba42a

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch from 34ed733 to 44ba42a Compare February 28, 2025 16:13

maykathm marked this pull request as ready for review March 11, 2025 12:47

maykathm requested review from bboozzoo, sergiocazzolato and zyga March 11, 2025 12:48

bboozzoo reviewed Mar 13, 2025

View reviewed changes

tests: addressed review comments and changed tests name format from u…

0fe129b

…sing '\' to using '--' to reflect changes in debug tagging PR

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch 5 times, most recently from c4ff4f0 to ba0efa9 Compare March 14, 2025 16:10

github: added option to force the use of the rules file

75369e9

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch from ba0efa9 to 75369e9 Compare March 14, 2025 16:18

zyga reviewed Mar 19, 2025

View reviewed changes

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch from d4ad170 to 56daeb9 Compare March 21, 2025 10:34

tests: create structures for features

49a5046

maykathm force-pushed the SNAPDENG-34441-add-feature-tagging-workflow branch from 56daeb9 to 49a5046 Compare March 21, 2025 10:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: add feature tagging workflow #15148

tests: add feature tagging workflow #15148

maykathm commented Feb 27, 2025 •

edited

Loading

github-actions bot commented Feb 27, 2025 •

edited

Loading

codecov bot commented Feb 28, 2025 •

edited

Loading

zyga left a comment

zyga Mar 19, 2025

zyga Mar 19, 2025

zyga Mar 19, 2025

zyga Mar 19, 2025

zyga Mar 19, 2025

zyga Mar 19, 2025

maykathm Mar 21, 2025

zyga Mar 19, 2025

zyga Mar 19, 2025

maykathm Mar 21, 2025

zyga Mar 19, 2025

maykathm Mar 20, 2025

zyga Mar 19, 2025

	system = "_".join(system.split(':'))
	system = system.replace(":", "_")



		if __name__ == '__main__':
		parser = argparse.ArgumentParser(description="""

		return original_name, suite_name, test_name, variant_name


		def _compose_test(dir: str, file: str, failed_tests: str) -> dict:

tests: add feature tagging workflow #15148

Are you sure you want to change the base?

tests: add feature tagging workflow #15148

Conversation

maykathm commented Feb 27, 2025 • edited Loading

github-actions bot commented Feb 27, 2025 • edited Loading

Spread tests skipped

codecov bot commented Feb 28, 2025 • edited Loading

Codecov Report

zyga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maykathm commented Feb 27, 2025 •

edited

Loading

github-actions bot commented Feb 27, 2025 •

edited

Loading

codecov bot commented Feb 28, 2025 •

edited

Loading