Skip to content

Commit 402ebd3

Browse files
committed
ci: add static validation checks for sample assets
Adds an offline pytest suite (tests/) and a GitHub Actions workflow that runs it on pull requests, so problems in the samples are caught in CI instead of by a customer. The motivating case: a fleet host configuration script that exceeds the service's 15000-character scriptBody limit looks fine in the repo but fails the moment a customer applies it. Checks: - OpenJD job and environment templates pass 'openjd check'. - Host configuration scripts stay within the service scriptBody size limit and Linux shell scripts pass 'bash -n' syntax checking. - Queue environment templates stay within the EnvironmentTemplate size limit. - CloudFormation templates parse (incl. !Sub/!Ref intrinsics) and pass cfn-lint. - Conda recipes have a well-formed deadline-cloud.yaml and a parseable recipe. Service limits are pinned to the AWS Deadline Cloud API model in tests/service_limits.py. The workflow caches pip installs keyed on tests/requirements.txt. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent ff3c32e commit 402ebd3

11 files changed

Lines changed: 585 additions & 0 deletions
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Static Checks
2+
3+
# Runs fast, offline static validation of the sample assets (OpenJD templates,
4+
# host configuration scripts, CloudFormation templates, and Conda recipes) so
5+
# that problems -- for example a host configuration script that exceeds the
6+
# service's size limit -- are caught in CI instead of by a customer.
7+
8+
on:
9+
push:
10+
branches: ["mainline"]
11+
pull_request:
12+
branches: ["mainline"]
13+
14+
permissions:
15+
contents: read
16+
17+
concurrency:
18+
group: static-checks-${{ github.ref }}
19+
cancel-in-progress: true
20+
21+
jobs:
22+
static-checks:
23+
name: Static validation
24+
runs-on: ubuntu-latest
25+
steps:
26+
- name: Check out repository
27+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
28+
29+
- name: Set up Python
30+
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
31+
with:
32+
python-version: "3.12"
33+
# Cache pip downloads keyed on the test requirements file so the
34+
# openjd-cli / cfn-lint install is restored from cache on unchanged runs.
35+
cache: pip
36+
cache-dependency-path: tests/requirements.txt
37+
38+
- name: Install test dependencies
39+
run: |
40+
python -m pip install --upgrade pip
41+
python -m pip install -r tests/requirements.txt
42+
43+
- name: Run static checks
44+
working-directory: tests
45+
run: python -m pytest -v

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ conda_recipes/archive_files/vraystd_*
1515
**/*.exe
1616
**/*.run
1717
**/*.pyc
18+
**/__pycache__/
19+
.pytest_cache/
1820
**/*.aex
1921
**/*.conda
2022
**/*.safetensors

tests/README.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Static validation tests
2+
3+
Static, offline checks for the sample assets in this repository. They run in CI
4+
on every pull request (see `.github/workflows/static_checks.yml`) and can be run
5+
locally with:
6+
7+
```bash
8+
python -m pip install -r tests/requirements.txt
9+
python -m pytest tests -v
10+
```
11+
12+
These checks exist to catch problems *before* a customer does. They are
13+
deliberately fast and require no AWS credentials or network access.
14+
15+
## What is checked
16+
17+
| Area | File | Check |
18+
|------|------|-------|
19+
| Open Job Description job & environment templates | `test_openjd_templates.py` | Every `*template*.yaml` with an OpenJD `specificationVersion` passes `openjd check`. |
20+
| Host configuration scripts | `test_host_configuration_scripts.py` | Byte length is within the Deadline Cloud service limit (`HostConfiguration.scriptBody` max **15000**), scripts have a valid shebang / interpreter line, and Linux (`*.sh`) scripts pass `bash -n` syntax checking. |
21+
| Queue environments | `test_openjd_templates.py` | Serialized `environment-2023-09` templates are within the service limit for `EnvironmentTemplate` (max **15000**). |
22+
| CloudFormation templates | `test_cloudformation.py` | Templates parse as CloudFormation YAML (intrinsic tags such as `!Sub`/`!Ref` supported) and, when `cfn-lint` is available, pass linting. |
23+
| Conda recipes | `test_conda_recipes.py` | Each recipe directory has a `deadline-cloud.yaml` with the expected schema, referenced recipe files exist, and recipe YAML parses. |
24+
25+
## Where the limits come from
26+
27+
The numeric limits in `service_limits.py` are taken from the AWS Deadline Cloud
28+
API model (the `deadline` botocore service definition, API version
29+
`2023-10-12`). The most important one for this repository is
30+
`HostConfiguration.scriptBody`, whose maximum length is **15000** characters — a
31+
host configuration script that exceeds it is rejected by `UpdateFleet`, which is
32+
exactly the class of failure these checks are meant to catch early.

tests/conftest.py

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
"""Shared discovery helpers for the static validation tests.
3+
4+
Everything here is filesystem-only so the tests run fast and need no network or
5+
AWS credentials. Discovery walks the repository from its root (the parent of the
6+
``tests`` directory) and deliberately skips directories that hold scratch or
7+
vendored copies (for example ``.claude`` worktrees and ``.git``) so those never
8+
affect CI results.
9+
"""
10+
from __future__ import annotations
11+
12+
import re
13+
from pathlib import Path
14+
15+
REPO_ROOT = Path(__file__).resolve().parent.parent
16+
17+
# Directories anywhere in the tree whose contents are not part of the samples we
18+
# ship and should never be validated.
19+
_EXCLUDED_DIR_NAMES = {".git", ".claude", ".kiro", "node_modules", "__pycache__", "build"}
20+
21+
# Matches the OpenJD ``specificationVersion`` header of a standalone template.
22+
# Anchored at column 0 (no leading whitespace) on purpose: a standalone OpenJD
23+
# template file has this as a top-level key, whereas a template *embedded* inside
24+
# another document (for example an environment template nested in a
25+
# CloudFormation resource) is indented. Only standalone template files are
26+
# validated with ``openjd check``.
27+
_SPEC_VERSION_RE = re.compile(
28+
r"""^specificationVersion\s*:\s*['"]?(?P<version>[A-Za-z0-9._-]+)""",
29+
re.MULTILINE,
30+
)
31+
32+
33+
def _is_excluded(path: Path) -> bool:
34+
return any(part in _EXCLUDED_DIR_NAMES for part in path.parts)
35+
36+
37+
def _iter_yaml_files() -> list[Path]:
38+
files = []
39+
for pattern in ("*.yaml", "*.yml"):
40+
for path in REPO_ROOT.rglob(pattern):
41+
if not _is_excluded(path.relative_to(REPO_ROOT)):
42+
files.append(path)
43+
return sorted(set(files))
44+
45+
46+
def spec_version(path: Path) -> str | None:
47+
"""Return the OpenJD ``specificationVersion`` of a YAML file, or ``None``.
48+
49+
Reads only the head of the file; the header appears at the top of every
50+
OpenJD template.
51+
"""
52+
try:
53+
head = path.read_text(encoding="utf-8", errors="replace")[:4096]
54+
except OSError:
55+
return None
56+
match = _SPEC_VERSION_RE.search(head)
57+
return match.group("version") if match else None
58+
59+
60+
def find_openjd_templates() -> list[Path]:
61+
"""All OpenJD job and environment templates in the repository."""
62+
return [
63+
p
64+
for p in _iter_yaml_files()
65+
if (v := spec_version(p)) is not None
66+
and (v.startswith("jobtemplate-") or v.startswith("environment-"))
67+
]
68+
69+
70+
def find_job_templates() -> list[Path]:
71+
return [p for p in find_openjd_templates() if (spec_version(p) or "").startswith("jobtemplate-")]
72+
73+
74+
def find_environment_templates() -> list[Path]:
75+
return [p for p in find_openjd_templates() if (spec_version(p) or "").startswith("environment-")]
76+
77+
78+
def find_host_configuration_scripts() -> list[Path]:
79+
"""Host configuration shell / PowerShell scripts."""
80+
base = REPO_ROOT / "host_configuration_scripts"
81+
if not base.is_dir():
82+
return []
83+
scripts = []
84+
for pattern in ("*.sh", "*.ps1"):
85+
for path in base.rglob(pattern):
86+
if not _is_excluded(path.relative_to(REPO_ROOT)):
87+
scripts.append(path)
88+
return sorted(set(scripts))
89+
90+
91+
def find_cloudformation_templates() -> list[Path]:
92+
"""CloudFormation templates (YAML files under ``cloudformation/``).
93+
94+
Excludes the OpenJD job templates that happen to live under that tree (for
95+
example the ``test-job.yaml`` samples) since those are validated by the
96+
OpenJD checks instead.
97+
"""
98+
base = REPO_ROOT / "cloudformation"
99+
if not base.is_dir():
100+
return []
101+
templates = []
102+
for pattern in ("*.yaml", "*.yml"):
103+
for path in base.rglob(pattern):
104+
rel = path.relative_to(REPO_ROOT)
105+
if _is_excluded(rel) or spec_version(path) is not None:
106+
continue
107+
templates.append(path)
108+
return sorted(set(templates))
109+
110+
111+
def find_conda_recipe_dirs() -> list[Path]:
112+
"""Directories under ``conda_recipes/`` that contain a ``deadline-cloud.yaml``."""
113+
base = REPO_ROOT / "conda_recipes"
114+
if not base.is_dir():
115+
return []
116+
dirs = []
117+
for path in base.rglob("deadline-cloud.yaml"):
118+
rel = path.relative_to(REPO_ROOT)
119+
if not _is_excluded(rel):
120+
dirs.append(path.parent)
121+
return sorted(set(dirs))
122+
123+
124+
def rel(path: Path) -> str:
125+
"""Repo-relative string form of a path, for readable test ids."""
126+
try:
127+
return str(path.relative_to(REPO_ROOT))
128+
except ValueError:
129+
return str(path)

tests/pytest.ini

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[pytest]
2+
# Run from the tests/ directory so `import conftest`, `service_limits`, etc.
3+
# resolve without packaging the samples repo.
4+
testpaths = .
5+
python_files = test_*.py
6+
addopts = -ra

tests/requirements.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Dependencies for the static validation test suite (tests/).
2+
# Pinned loosely; CI caches the resolved install (see the static_checks workflow).
3+
pytest>=7,<10
4+
PyYAML>=6,<7
5+
openjd-cli>=0.7,<1
6+
cfn-lint>=1,<2

tests/service_limits.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
"""AWS Deadline Cloud service limits relevant to the sample assets.
3+
4+
These values come from the AWS Deadline Cloud API model (the ``deadline``
5+
botocore service definition, API version ``2023-10-12``). They are the hard
6+
limits the service enforces on request inputs; a sample that exceeds one of them
7+
will be rejected by the corresponding API call. Keeping the checks pinned to the
8+
documented service limits is what lets CI catch the problem before a customer
9+
does.
10+
"""
11+
12+
# Maximum length, in characters, of a fleet host configuration script.
13+
# Model shape: HostConfigurationScript (string, max=15000).
14+
# Used by UpdateFleet's hostConfiguration.scriptBody. A script longer than this
15+
# is rejected by the service -- this is the limit that previously slipped
16+
# through to a customer.
17+
HOST_CONFIGURATION_SCRIPT_MAX_CHARS = 15000
18+
19+
# Bounds, in seconds, on HostConfiguration.scriptTimeoutSeconds.
20+
# Model shape: HostConfigurationScriptTimeoutSeconds (integer, min=300, max=3600).
21+
HOST_CONFIGURATION_SCRIPT_TIMEOUT_MIN_SECONDS = 300
22+
HOST_CONFIGURATION_SCRIPT_TIMEOUT_MAX_SECONDS = 3600
23+
24+
# Maximum length, in characters, of a serialized queue environment template
25+
# passed to the service. Model shape: EnvironmentTemplate (string, max=15000).
26+
ENVIRONMENT_TEMPLATE_MAX_CHARS = 15000
27+
28+
# Maximum length, in characters, of a serialized job template passed to
29+
# CreateJob. Model shape: JobTemplate (string, max=1000000).
30+
JOB_TEMPLATE_MAX_CHARS = 1000000

tests/test_cloudformation.py

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
"""Static checks for the CloudFormation templates.
3+
4+
The checks are twofold:
5+
6+
* The template parses as CloudFormation YAML, including the intrinsic-function
7+
short forms (``!Sub``, ``!Ref``, ``!GetAtt``, ...) that plain YAML loaders
8+
choke on, and has the required top-level ``Resources`` section.
9+
* When ``cfn-lint`` is installed it must pass with no errors. ``cfn-lint`` is
10+
the authoritative linter for CloudFormation; if it is not available (for
11+
example in a minimal local environment) that portion is skipped, but CI
12+
installs it so the lint always runs there.
13+
"""
14+
from __future__ import annotations
15+
16+
import json
17+
import shutil
18+
import subprocess
19+
from pathlib import Path
20+
21+
import pytest
22+
import yaml
23+
24+
from conftest import find_cloudformation_templates, rel
25+
26+
_TEMPLATES = find_cloudformation_templates()
27+
_CFN_LINT = shutil.which("cfn-lint")
28+
29+
30+
class _CfnLoader(yaml.SafeLoader):
31+
"""A SafeLoader that understands CloudFormation ``!Tag`` short forms."""
32+
33+
34+
def _cfn_tag_constructor(loader: yaml.Loader, tag_suffix: str, node):
35+
# Represent intrinsics generically; we only care that they parse, not that
36+
# they resolve. Preserve the tag name so the structure round-trips sensibly.
37+
if isinstance(node, yaml.ScalarNode):
38+
return {tag_suffix: loader.construct_scalar(node)}
39+
if isinstance(node, yaml.SequenceNode):
40+
return {tag_suffix: loader.construct_sequence(node)}
41+
return {tag_suffix: loader.construct_mapping(node)}
42+
43+
44+
_CfnLoader.add_multi_constructor("!", _cfn_tag_constructor)
45+
46+
47+
def test_cloudformation_templates_discovered():
48+
assert _TEMPLATES, "no CloudFormation templates were discovered"
49+
50+
51+
@pytest.mark.parametrize("template", _TEMPLATES, ids=rel)
52+
def test_cloudformation_template_parses(template: Path):
53+
try:
54+
doc = yaml.load(template.read_text(encoding="utf-8"), Loader=_CfnLoader)
55+
except yaml.YAMLError as exc:
56+
pytest.fail(f"{rel(template)} is not valid YAML:\n{exc}")
57+
58+
assert isinstance(doc, dict), f"{rel(template)} did not parse to a mapping"
59+
assert "Resources" in doc, (
60+
f"{rel(template)} has no top-level 'Resources' section; is it a "
61+
f"CloudFormation template?"
62+
)
63+
assert doc["Resources"], f"{rel(template)} has an empty 'Resources' section"
64+
65+
66+
@pytest.mark.skipif(_CFN_LINT is None, reason="cfn-lint is not installed")
67+
@pytest.mark.parametrize("template", _TEMPLATES, ids=rel)
68+
def test_cloudformation_template_passes_cfn_lint(template: Path):
69+
result = subprocess.run(
70+
[_CFN_LINT, "--format", "json", str(template)],
71+
capture_output=True,
72+
text=True,
73+
timeout=120,
74+
)
75+
if result.returncode == 0:
76+
return
77+
78+
# cfn-lint exit codes: bit 0x2 => error, 0x4 => warning, 0x8 => informational.
79+
# Only fail the test on errors so style warnings don't block sample PRs.
80+
findings = []
81+
try:
82+
findings = json.loads(result.stdout or "[]")
83+
except json.JSONDecodeError:
84+
pytest.fail(
85+
f"cfn-lint failed for {rel(template)}:\n{result.stdout}\n{result.stderr}"
86+
)
87+
88+
errors = [f for f in findings if f.get("Level") == "Error"]
89+
assert not errors, (
90+
f"cfn-lint reported {len(errors)} error(s) for {rel(template)}:\n"
91+
+ "\n".join(
92+
f" {e.get('Rule', {}).get('Id', '?')}: {e.get('Message', '')}"
93+
for e in errors
94+
)
95+
)

0 commit comments

Comments
 (0)