Skip to content

Commit bd478bd

Browse files
committed
feat(projectconfig): accept [tests] and [test-groups] schema in config files
1 parent 6978b63 commit bd478bd

12 files changed

Lines changed: 700 additions & 1 deletion

File tree

docs/user/reference/config/components.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ A component definition tells azldev where to find the spec file, how to customiz
1515
| Render config | `render` | [RenderConfig](#render-configuration) | No | Options controlling spec rendering behavior |
1616
| Source files | `source-files` | array of [SourceFileReference](#source-file-references) | No | Additional source files to download for this component |
1717
| Package overrides | `packages` | map of string → [PackageConfig](package-groups.md#package-config) | No | Exact per-package configuration overrides; highest priority in the resolution order |
18+
| Tests | `tests` | [ComponentTests](#component-tests) | No | Test references that apply to this component (see [Tests and Test Groups](tests.md)) |
1819

1920
### Bare Components
2021

@@ -299,6 +300,24 @@ rpm-channel = "rpm-devel"
299300
rpm-channel = "none"
300301
```
301302

303+
## Component Tests
304+
305+
The `[components.<name>.tests]` subtable lists test or test-group
306+
references that apply to the component. Each entry is a [TestRef](tests.md#test-reference)
307+
with exactly one of `name` or `group`.
308+
309+
| Field | TOML Key | Type | Required | Description |
310+
|-------|----------|------|----------|-------------|
311+
| Tests | `tests` | array of [TestRef](tests.md#test-reference) | No | References to `[tests.<name>]` entries or `[test-groups.<name>]` entries |
312+
313+
```toml
314+
[components.kernel.tests]
315+
tests = [
316+
{ group = "kernel-bvt" },
317+
{ name = "kdump-smoke" },
318+
]
319+
```
320+
302321
## Source File References
303322

304323
The `[[components.<name>.source-files]]` array defines additional source files that azldev should download before building. These are files not available in the dist-git repository or lookaside cache — typically binaries, pre-built artifacts, or files from custom hosting.
@@ -460,5 +479,6 @@ lines = ["cp -vf %{shimdirx64}/$(basename %{shimefix64}) %{shimefix64} ||:"]
460479
- [Distros](distros.md) — distro definitions and `default-component-config` inheritance
461480
- [Component Groups](component-groups.md) — grouping components with shared defaults
462481
- [Package Groups](package-groups.md) — project-level package groups and full resolution order
482+
- [Tests and Test Groups](tests.md) — definitions referenced by `[components.<name>.tests]`
463483
- [Configuration System](../../explanation/config-system.md) — inheritance and merge behavior
464484
- [JSON Schema](../../../../schemas/azldev.schema.json) — machine-readable schema

docs/user/reference/config/config-file.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ All config files share the same schema — there is no distinction between a "ro
1515
| `component-groups` | map of objects | Named groups of components with shared defaults | [Component Groups](component-groups.md) |
1616
| `images` | map of objects | Image definitions (VMs, containers) | [Images](images.md) |
1717
| `test-suites` | map of objects | Named test suite definitions referenced by images | [Test Suites](test-suites.md) |
18+
| `tests` | map of objects | Named test definitions (new-shape, parse-only) | [Tests and Test Groups](tests.md) |
19+
| `test-groups` | map of objects | Named bundles of test references (new-shape, parse-only) | [Tests and Test Groups](tests.md) |
1820
| `tools` | object | Configuration for external tools used by azldev | [Tools](tools.md) |
1921
| `default-package-config` | object | Project-wide default applied to all binary packages | [Package Groups — Resolution Order](package-groups.md#resolution-order) |
2022
| `package-groups` | map of objects | Named groups of binary packages with shared config | [Package Groups](package-groups.md) |

docs/user/reference/config/images.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,12 @@ The `capabilities` subtable describes what the image supports. All fields are op
3535

3636
## Image Tests
3737

38-
The `tests` subtable links an image to one or more test suites defined in the top-level [`[test-suites]`](test-suites.md) section.
38+
The `tests` subtable links an image to one or more test suites defined in the top-level [`[test-suites]`](test-suites.md) section, and/or to entries from the new-shape [`[tests]` / `[test-groups]`](tests.md) sections.
3939

4040
| Field | TOML Key | Type | Required | Description |
4141
|-------|----------|------|----------|-------------|
4242
| Test Suites | `test-suites` | array of inline tables | No | List of test suite references. Each entry must have a `name` field matching a key in `[test-suites]`. |
43+
| Tests | `tests` | array of [TestRef](tests.md#test-reference) | No | References to `[tests.<name>]` entries or `[test-groups.<name>]` entries (parse-only; see [Tests and Test Groups](tests.md)). |
4344

4445
## Image Publish
4546

@@ -118,4 +119,5 @@ channels = ["registry-prod", "registry-staging"]
118119

119120
- [Config File Structure](config-file.md) — top-level config file layout
120121
- [Test Suites](test-suites.md) — test suite definitions
122+
- [Tests and Test Groups](tests.md) — new-shape test/group definitions referenced by `[images.<name>.tests]`
121123
- [Tools](tools.md) — Image Customizer tool configuration
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Tests and Test Groups
2+
3+
The `[tests]` and `[test-groups]` sections declare framework-agnostic test
4+
metadata that components and images can target by name. Each test entry
5+
binds a single test (a pytest run, a LISA case, or a TMT plan)
6+
to a named identifier; each group entry bundles tests (and
7+
nested groups) under one name so callers can reference a curated set
8+
without enumerating every member.
9+
10+
## Test Definition
11+
12+
Each entry under `[tests.<name>]` describes one configuration of one
13+
runner. Framework-specific options live in a typed subtable
14+
(`pytest`, `lisa`, `tmt`) whose contents are passed through
15+
to the runner; their internal schemas are intentionally not validated
16+
by azldev so frameworks can evolve independently.
17+
18+
| Field | TOML Key | Type | Required | Description |
19+
|-------|----------|------|----------|-------------|
20+
| Type | `type` | string | Yes | Test framework: `pytest`, `lisa`, or `tmt` |
21+
| Description | `description` | string | No | Human-readable description |
22+
| Kind | `kind` | string array | No | Free-form hints (e.g. `functional`, `performance`, `bvt`) |
23+
| Long running | `long-running` | boolean | No | Hints that this test may run for hours |
24+
| Required capabilities | `required-capabilities` | string array | No | Capability tokens the image must declare for this test to be applicable |
25+
| Lisa | `lisa` | table | No | LISA-specific configuration (opaque to azldev) |
26+
| Tmt | `tmt` | table | No | TMT-specific configuration (opaque to azldev) |
27+
| Pytest | `pytest` | table | No | pytest-specific configuration (opaque to azldev) |
28+
29+
## Test Group
30+
31+
Each entry under `[test-groups.<name>]` names an ordered list of test or
32+
nested-group references that callers can target as a single unit.
33+
34+
| Field | TOML Key | Type | Required | Description |
35+
|-------|----------|------|----------|-------------|
36+
| Description | `description` | string | No | Human-readable description |
37+
| Tests | `tests` | array of [TestRef](#test-reference) | No | Ordered members of the group |
38+
39+
## Test Reference
40+
41+
`TestRef` is an inline table with exactly one of `name` or `group`:
42+
43+
| Field | TOML Key | Type | Description |
44+
|-------|----------|------|-------------|
45+
| Name | `name` | string | References a `[tests.<name>]` entry |
46+
| Group | `group` | string | References a `[test-groups.<name>]` entry |
47+
48+
## Referencing from Components and Images
49+
50+
Components and images both expose a `tests` subtable that holds a list
51+
of `TestRef`s:
52+
53+
```toml
54+
[components.kernel.tests]
55+
tests = [{ group = "kernel-bvt" }, { name = "kdump-smoke" }]
56+
57+
[images.vm-base.tests]
58+
tests = [{ group = "bvt" }]
59+
```
60+
61+
## Example
62+
63+
```toml
64+
[tests.bvt-ssh]
65+
type = "pytest"
66+
description = "Basic SSH boot verification"
67+
kind = ["functional", "bvt"]
68+
required-capabilities = ["ssh"]
69+
pytest = { working-dir = "tests/bvt", test-paths = ["test_ssh.py"] }
70+
71+
[tests.kdump-smoke]
72+
type = "lisa"
73+
description = "Smoke test for kdump"
74+
lisa = { case = "kdump.smoke" }
75+
76+
[test-groups.bvt]
77+
description = "Build verification tests"
78+
tests = [
79+
{ name = "bvt-ssh" },
80+
{ group = "bvt-extras" },
81+
]
82+
83+
[test-groups.bvt-extras]
84+
tests = [{ name = "kdump-smoke" }]
85+
```
86+
87+
## Related Resources
88+
89+
- [Test Suites](test-suites.md) - legacy test suite definitions
90+
- [Components](components.md#component-tests) — per-component `tests` field
91+
- [Images](images.md#image-tests) — per-image `tests` field
92+
- [Config File Structure](config-file.md) — top-level config layout

internal/projectconfig/component.go

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,14 @@ type ComponentConfig struct {
290290
// all packages produced by this component. Overridden by package-group and per-package settings
291291
// for binary and debuginfo channels.
292292
Publish ComponentPublishConfig `toml:"publish,omitempty" json:"publish,omitempty" table:"-" jsonschema:"title=Publish settings,description=Component-level publish channel settings" fingerprint:"-"`
293+
294+
// Tests holds the new-shape per-component tests block:
295+
//
296+
// tests.tests = [{ name = "..." }, { group = "..." }]
297+
//
298+
// References must resolve to entries in the project-level [tests] or
299+
// [test-groups] maps; resolution is the responsibility of the test layer.
300+
Tests ComponentTestsConfig `toml:"tests,omitempty" json:"tests,omitempty" table:"-" jsonschema:"title=Tests,description=Per-component test or test-group references" fingerprint:"-"`
293301
}
294302

295303
// AllowedSourceFilesHashTypes defines the set of hash types that are supported
@@ -383,6 +391,7 @@ func (c *ComponentConfig) WithAbsolutePaths(referenceDir string) *ComponentConfi
383391
SourceFiles: deep.MustCopy(c.SourceFiles),
384392
Packages: deep.MustCopy(c.Packages),
385393
Publish: deep.MustCopy(c.Publish),
394+
Tests: deep.MustCopy(c.Tests),
386395
}
387396

388397
// Fix up paths.

internal/projectconfig/configfile.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@ type ConfigFile struct {
6666
// Definitions of test suites.
6767
TestSuites map[string]TestSuiteConfig `toml:"test-suites,omitempty" validate:"dive" jsonschema:"title=Test Suites,description=Definitions of test suites for this project"`
6868

69+
// Definitions of individual tests (new schema, [tests.X]).
70+
Tests map[string]TestDefinition `toml:"tests,omitempty" validate:"dive" jsonschema:"title=Tests,description=Definitions of individual tests"`
71+
72+
// Definitions of test groups (new schema, [test-groups.X]).
73+
TestGroups map[string]TestGroup `toml:"test-groups,omitempty" validate:"dive" jsonschema:"title=Test Groups,description=Definitions of named bundles of tests"`
74+
6975
// Internal fields used to track the origin of the config file; `dir` is the directory
7076
// that the config file's relative paths are based from.
7177
sourcePath string `toml:"-"`

internal/projectconfig/fingerprint_test.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,9 @@ func TestAllFingerprintedFieldsHaveDecision(t *testing.T) {
6565
// ComponentConfig.Publish — post-build routing (where to publish), not a build input.
6666
"ComponentConfig.Publish": true,
6767

68+
// ComponentConfig.Tests — test selection metadata (new schema), not a build input.
69+
"ComponentConfig.Tests": true,
70+
6871
// ComponentOverlay.Description — human-readable documentation for the overlay.
6972
"ComponentOverlay.Description": true,
7073
// ComponentOverlay.Source — absolute path that varies by checkout location.

internal/projectconfig/image.go

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,11 @@ type ImageTestsConfig struct {
115115
// reference identifies a test suite defined in the top-level [test-suites] section
116116
// and may carry per-test metadata in the future (e.g., required vs optional).
117117
TestSuites []TestSuiteRef `toml:"test-suites,omitempty" json:"testSuites,omitempty" jsonschema:"title=Test Suites,description=List of test suite references that apply to this image"`
118+
119+
// Tests is the new-shape list of test or test-group references that apply to this
120+
// image. References must resolve to entries in the project-level [tests] or
121+
// [test-groups] maps; resolution is the responsibility of the test layer.
122+
Tests []TestRef `toml:"tests,omitempty" json:"tests,omitempty" jsonschema:"title=Tests,description=List of test or test-group references that apply to this image"`
118123
}
119124

120125
// TestSuiteRef is a reference to a named test suite. Using a structured type (rather than

internal/projectconfig/tests.go

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
package projectconfig
5+
6+
import (
7+
"github.com/invopop/jsonschema"
8+
)
9+
10+
// TestDefinition is the new-shape [tests.X] declaration: one configuration of one
11+
// runner/harness with framework-specific options. Framework subtables are kept as
12+
// loosely-typed maps so the resolver can evolve their schemas without requiring
13+
// matching struct changes here.
14+
type TestDefinition struct {
15+
// Type identifies the framework/runner. Required, and constrained to the
16+
// closed enum in the schema tag at the schema layer. The loader still
17+
// accepts unknown values permissively; the resolver is the source of truth.
18+
Type string `toml:"type" json:"type" jsonschema:"required,title=Type,description=Test framework type,enum=pytest,enum=lisa,enum=tmt"`
19+
20+
// Human-readable description.
21+
Description string `toml:"description,omitempty" json:"description,omitempty" jsonschema:"title=Description,description=Description of this test"`
22+
23+
// Kind hints at what the test exercises (e.g. functional, performance).
24+
Kind []string `toml:"kind,omitempty" json:"kind,omitempty" jsonschema:"title=Kind,description=Test kind hints (e.g. functional or performance)"`
25+
26+
// LongRunning hints to schedulers/policy that this test may take a long time.
27+
LongRunning bool `toml:"long-running,omitempty" json:"longRunning,omitempty" jsonschema:"title=Long running,description=Hints that this test may run for hours"`
28+
29+
// RequiredCapabilities lists capability tokens an image must declare to be a
30+
// valid target for this test. Tokens are matched against [ImageCapabilities].
31+
RequiredCapabilities []string `toml:"required-capabilities,omitempty" json:"requiredCapabilities,omitempty" jsonschema:"title=Required capabilities,description=Capability tokens the image must declare"`
32+
33+
// Framework-specific subtables. Kept untyped so framework schema can evolve
34+
// independently of the dev-tools type definitions.
35+
Lisa map[string]any `toml:"lisa,omitempty" json:"lisa,omitempty" jsonschema:"title=LISA config,description=LISA-specific configuration"`
36+
Tmt map[string]any `toml:"tmt,omitempty" json:"tmt,omitempty" jsonschema:"title=TMT config,description=TMT-specific configuration"`
37+
Pytest map[string]any `toml:"pytest,omitempty" json:"pytest,omitempty" jsonschema:"title=Pytest config,description=pytest-specific configuration"`
38+
}
39+
40+
// TestGroup is a [test-groups.X] declaration: a named bundle of test references that
41+
// images or components can target via a single name.
42+
type TestGroup struct {
43+
// Human-readable description.
44+
Description string `toml:"description,omitempty" json:"description,omitempty" jsonschema:"title=Description,description=Description of this test group"`
45+
46+
// Tests is the ordered list of test or nested-group references that make up
47+
// the group's membership.
48+
Tests []TestRef `toml:"tests,omitempty" json:"tests,omitempty" jsonschema:"title=Tests,description=Member references (each is either {name=...} or {group=...})"`
49+
}
50+
51+
// TestRef is a reference to either a test (by name) or another group (by name).
52+
// Exactly one of Name or Group should be set; semantic validation is the resolver's
53+
// responsibility.
54+
type TestRef struct {
55+
// Name references a [tests.X] entry.
56+
Name string `toml:"name,omitempty" json:"name,omitempty" jsonschema:"title=Name,description=Name of a test (mutually exclusive with group)"`
57+
58+
// Group references a [test-groups.X] entry.
59+
Group string `toml:"group,omitempty" json:"group,omitempty" jsonschema:"title=Group,description=Name of a test group (mutually exclusive with name)"`
60+
}
61+
62+
// ComponentTestsConfig holds the new-shape per-component tests block:
63+
//
64+
// tests.tests = [{ name = "..." }, { group = "..." }]
65+
type ComponentTestsConfig struct {
66+
// Tests is the list of test or test-group references that apply to the component.
67+
Tests []TestRef `toml:"tests,omitempty" json:"tests,omitempty" jsonschema:"title=Tests,description=Per-component test or test-group references"`
68+
}
69+
70+
// JSONSchemaExtend tightens the generated schema for [TestRef] so editors can
71+
// flag refs that set neither or both of name/group. The runtime resolver
72+
// ([ErrInvalidTestRef]) is the source of truth; this keeps the schema in sync.
73+
func (TestRef) JSONSchemaExtend(schema *jsonschema.Schema) {
74+
// Exactly one of name|group: encoded as oneOf with `required` on each and
75+
// `not` excluding the other, which forbids both `{}` and `{name, group}`.
76+
schema.OneOf = []*jsonschema.Schema{
77+
{Required: []string{"name"}, Not: &jsonschema.Schema{Required: []string{"group"}}},
78+
{Required: []string{"group"}, Not: &jsonschema.Schema{Required: []string{"name"}}},
79+
}
80+
}

0 commit comments

Comments
 (0)