A repository containing EMODnet-focused demonstration configuration files and the orchestration workflow for the GRMP test system. The workflow runs the GRMP orchestrator, collects JUnit XML reports, publishes them to a dashboard, and optionally creates GitHub issues for failing test suites.
The orchestrator can be found here. Available test implementations can be found here.
.
├── .github/
│ └── workflows/
│ └── orchestrate-config-tests.yml ← main orchestration workflow
├── dashboard/
│ └── index.html ← dashboard (pushed to report branch on each run)
├── tests / ← top-level folder for configuration directories
The following resources are tested
| Resource URI | Resource type | Description | tests |
|---|---|---|---|
| http://vocab.nerc.ac.uk/ldes/P01/ | LDES | Terms built using the BODC parameter semantic model designed to describe individual measured phenomena. May be used to mark up sets of data such as a NetCDF array or spreadsheet column. Units must be specified when using a P01 code. The P06 unit that is linked to individual P01 in the NVS is the one used in BODC's systems but external users can use any appropriate units. | * LDES test |
| http://vocab.nerc.ac.uk/ldes/P06/ | LDES | Terms approved for use by BODC to describe the measurement units for data held in its repositories. | * LDES test |
| http://vocab.nerc.ac.uk/ldes/L22/ | LDES | Terms for distinct sampling or measuring devices that may be identified in the real world in terms of manufacturer and model number. | * LDES test |
| http://vocab.nerc.ac.uk/ldes/S09/ | LDES | Controlled vocabulary defining the terms that may be used for biological entity size (part of the what theme) in the BODC parameter semantic model. | * LDES test |
| http://vocab.nerc.ac.uk/ldes/S25/ | LDES | Terms used to describe biological entities (organisms or parts thereof) in the BODC Parameter Usage Vocabulary | * LDES test |
| https://mtt.vliz.be/api/ldes/data/1/latest.ttl | LDES | BODC P02 translations derived from mtt.vliz.be API | * LDES test |
| https://edmo.seadatanet.org/ldes/feed | LDES | EDMO LDES feed | * LDES test |
| https://edmerp.seadatanet.org/ldes/feed | LDES | EDMERP LDES feed | * LDES test |
| https://edmo.seadatanet.org/sparql/sparql | SPARQL endpoint | EDMO SPARQL endpoint | * content-availability * CORS |
| https://edmerp.seadatanet.org/sparql/sparql | SPARQL endpoint | EDMERP SPARQL endpoint | * content-availability * CORS |
| https://sparql.ifremer.fr/csr/query | SPARQL endpoint | Ifremer CSR SPARQL endpoint | * content-availability * CORS |
| https://vocab.nerc.ac.uk/sparql/sparql | SPARQL endpoint | NVS SPARQL endpoint | * content-availability * CORS |
The workflow is triggered manually via workflow_dispatch. It runs the orchestrator against a selected configuration directory, pushes the resulting report to a dedicated report branch, and performs a series of post-processing steps.
| Input | Required | Default | Description |
|---|---|---|---|
report_branch |
Yes | grmp-reports |
Branch to push reports and dashboard files to |
config_directory |
Yes | input-echo-demo |
Top-level folder containing the YAML config files to run |
dashboard_label |
No | (repository name) | Label shown in the dashboard header |
enable_prometheus |
No | false |
Generate and push a metrics.prom file for Prometheus |
The workflow requires the following repository permissions:
| Permission | Purpose |
|---|---|
contents: write |
Push reports to the report branch |
packages: read |
Pull test container images from GHCR |
issues: write |
Create and comment on GitHub issues |
pages: read |
Fetch the GitHub Pages URL for dashboard deep links in issues |
The optional add-issue-to-project.yml workflow uses vars.OPSCI_PROJECT_URL (with a fallback default) and secrets.ORG_PAT to add opened issues to an organization project. If you do not use that integration, disable or remove the workflow.
The orchestrator container (ghcr.io/vliz-be-opsci/grmp-framework:main) is run with the selected config directory mounted at /config. The orchestrator reads all .yaml and .yml files under that directory, runs the configured test containers, and writes a combined_report.xml to the reports/ directory.
The following GitHub Actions context variables are passed to the orchestrator so it can construct full provenance URLs:
GITHUB_SERVER_URLGITHUB_REPOSITORYGITHUB_SHACONFIG_DIRECTORY
The combined_report.xml is committed to the report branch as report.xml. If the branch does not yet exist, it is created as an orphan branch. The latest dashboard/index.html from the source branch is also copied across on each run, keeping the dashboard up to date. The commit message includes the run number, attempt number, branch name, and short commit SHA.
The combined report is parsed to extract per-suite statistics (total, passed, failed, errored, skipped, pass rate, duration). A new_run.json file is generated containing the full run metadata including run number, run attempt, source branch, triggering actor, and summary statistics.
For each test suite in the report that has create-issue: true set in its configuration, the workflow checks for existing open GitHub issues. Issues are identified by a title of the form [GRMP] {suite_name} #{config_hash} where the hash is a stable 8-character SHA-256 of the suite's input config properties (provenance and create-issue excluded from the hash).
- If the suite is failing or erroring and no open issue exists → a new issue is created with a properties table and a test case summary table, labelled
grmp-test-failure - If the suite is failing or erroring and an open issue exists → a comment is added with the current run's results
- If the suite is passing and an open issue exists → a comment is added noting recovery, suggesting the issue can be closed manually
Each issue comment includes a link to the GitHub Actions run and, if GitHub Pages is configured, a deep link to the dashboard pre-loaded with the corresponding run (#run=<commit_sha>). The GitHub Pages URL is fetched automatically from the GitHub API — no manual configuration is needed. This step runs with continue-on-error: true so a failure here does not block the remaining steps.
Note: The
grmp-test-failurelabel must exist in the repository before the first issue creation run.
The new_run.json metadata is prepended to runs.json on the report branch, which maintains a history of the most recent 50 runs. This file is read by the dashboard to populate the run selector dropdown. Reruns of the same workflow run are stored as separate entries distinguished by the run_attempt field.
A config.json is written to the report branch containing the reportUrl, runsUrl, and label fields needed by the dashboard to locate its data sources. This file is updated on every run so the dashboard always points to the correct branch.
When enable_prometheus is set to true, a metrics.prom file is generated in Prometheus textfile collector format and committed to the report branch. The file contains per-suite metrics with testsuite, run, attempt, and branch labels:
| Metric | Description |
|---|---|
grmp_testsuite_tests_total |
Total test cases in the suite |
grmp_testsuite_passed_total |
Passing test cases |
grmp_testsuite_failed_total |
Failing test cases |
grmp_testsuite_errors_total |
Errored test cases |
grmp_testsuite_skipped_total |
Skipped test cases |
grmp_testsuite_pass_rate |
Pass rate between 0 and 1 |
grmp_testsuite_duration_seconds |
Suite execution time in seconds |
grmp_run_timestamp_seconds |
Unix timestamp of the run |
To ingest these metrics into Prometheus, use the Node Exporter textfile collector. A cron job on the monitoring server can periodically fetch the latest metrics.prom from the report branch:
*/15 * * * * curl -s https://raw.githubusercontent.com/{org}/{repo}/{report_branch}/metrics.prom \
-o /path/to/node_exporter/textfiles/metrics.promThe dashboard (dashboard/index.html) is a self-contained static page that requires no server-side infrastructure. It operates in one of two modes depending on whether a config.json file is present alongside it.
When the dashboard is served from the report branch (via GitHub Pages or directly from raw.githubusercontent.com), it finds a config.json in the same directory containing:
{
"reportUrl": "https://raw.githubusercontent.com/{org}/{repo}/{report_branch}/report.xml",
"runsUrl": "https://raw.githubusercontent.com/{org}/{repo}/{report_branch}/runs.json",
"label": "My Dashboard"
}On load, the dashboard fetches config.json, then fetches runs.json to populate the run selector dropdown with the history of the last 50 runs. Selecting a run fetches the corresponding report.xml at the specific commit SHA for that run. If runs.json is unavailable it falls back to fetching the latest report.xml directly from reportUrl.
This mode is indicated by a Remote badge in the dashboard header.
When the dashboard is opened locally (e.g. by double-clicking index.html) or served without a config.json, it falls back to browse mode. A file picker is shown allowing the user to select a local combined_report.xml file directly from disk. This is useful for quickly inspecting a report without any GitHub infrastructure.
This mode is indicated by a Browse badge in the dashboard header.
- Remote mode: view the current report or any of the last 50 historical runs via a dropdown
- Remote mode: deep-link to a specific run via the
#run=<commit_sha>URL fragment — used by issue comments to link directly to the relevant run - Run selector shows the run number, attempt (if a rerun), source branch, short SHA, pass rate, and date
- Filter test suites by status (passed, failed, errored, skipped)
- Filter test cases by status
- Search test suites by name
- Filter by suite property and value
- Provenance values that are GitHub blob URLs are rendered as clickable links showing only the file path (e.g.
ldes-validation/tests.yaml)
If GitHub Pages is enabled for the repository, the dashboard is served automatically at https://{owner}.github.io/{repo} (or a custom domain if configured) from the report branch. The workflow keeps index.html up to date on the report branch on every run, so no manual deployment is needed.
To add a new set of tests:
- Create a new top-level directory in the repository (e.g.
my-new-checks/) - Add one or more
.yamlconfig files following the grmp-framework configuration format - Trigger the workflow with
config_directoryset to your new directory name
No changes to the workflow itself are needed.
If any test configuration uses the !secret YAML tag to reference sensitive values, the corresponding SECRET_* environment variables must be passed to the orchestrator container. This requires adding them to the Run tests step in the workflow:
-e SECRET_MY_API_KEY="${{ secrets.MY_API_KEY }}" \See the grmp-framework documentation for full details on how secrets are handled.