Skip to content

Commit 703773d

Browse files
Shahalt1Shahalt1
committed
Implement TestZeus Execute GitHub Action with input validation, login, and report generation
1 parent bf4d032 commit 703773d

File tree

4 files changed

+348
-0
lines changed

4 files changed

+348
-0
lines changed

‎README.md‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
# TestZeus Execute · Composite GitHub Action
2+
3+
Generate detailed CTRF reports from your TestZeus test runs directly from a GitHub Workflow.
4+
5+
---
6+
7+
## 📥 Inputs
8+
9+
| Name | Description | Required |
10+
|------|-------------|----------|
11+
| `test_ids_file` | Path to a JSON file that lists the test IDs to execute and the execution mode. | Yes |
12+
| `email` | TestZeus account email used to authenticate via the CLI. | Yes |
13+
| `password` | TestZeus account password used to authenticate via the CLI. | Yes |
14+
15+
---
16+
17+
## 🗂️ Test IDs JSON Format
18+
19+
Your **`test_ids_file`** *must* follow this shape:
20+
21+
```json
22+
{
23+
"execution_mode": "lenient", // or "strict"
24+
"test_ids": [
25+
"TC-101",
26+
"TC-102",
27+
"TC-103"
28+
]
29+
}
30+
```
31+
32+
• **`execution_mode`** — string, either `lenient` or `strict` depending on how you want the tests executed.
33+
• **`test_ids`** — array of TestZeus test IDs.
34+
35+
> The action validates the file and will fail fast if the structure is invalid.
36+
37+
---
38+
39+
## 🚀 Quick Start
40+
41+
```yaml
42+
name: "Run TestZeus Suite"
43+
44+
on: [workflow_dispatch]
45+
46+
jobs:
47+
testzeus:
48+
runs-on: ubuntu-latest
49+
50+
steps:
51+
- uses: actions/checkout@v4
52+
53+
- name: Execute TestZeus Tests
54+
uses: test-zeus-ai/testzeus-execute@v1
55+
with:
56+
test_ids_file: "test_ids.json"
57+
email: ${{ secrets.TESTZEUS_EMAIL }}
58+
password: ${{ secrets.TESTZEUS_PASSWORD }}
59+
60+
# Publish a beautiful GitHub summary using CTRF-io
61+
- name: Publish Test Report
62+
uses: ctrf-io/github-test-reporter@v1
63+
with:
64+
report-path: "ctrf_report.json"
65+
template-path: "templates/testzeus-report.hbs"
66+
custom-report: true
67+
if: always()
68+
```
69+
70+
---
71+
72+
## 📝 Handlebars Template (`templates/testzeus-report.hbs`)
73+
74+
Copy the template below into your repo (for example inside `templates/testzeus-report.hbs`) **or** create your own custom template.
75+
Then reference the file in your workflow step with the `template-path` input (see the *Publish Test Report* example above).
76+
77+
![Test Results Summary](assets/test-results-summary.png)
78+
79+
```hbs
80+
# 🧪 Test Results Summary
81+
82+
| **Tests** | **Passed** | **Failed** | **Skipped** | **Other** | **Flaky** | **Duration** |
83+
|----------|------------|------------|-------------|-----------|-----------|--------------|
84+
| {{ctrf.summary.tests}} | {{ctrf.summary.passed}} | {{ctrf.summary.failed}} | {{add ctrf.summary.skipped ctrf.summary.pending}} | {{ctrf.summary.other}} | {{countFlaky ctrf.tests}} | {{formatDuration ctrf.summary.start ctrf.summary.stop}} |
85+
86+
---
87+
88+
## 📊 Overview
89+
- ✅ Passed: {{ctrf.summary.passed}} / {{ctrf.summary.tests}}
90+
- ❌ Failed: {{ctrf.summary.failed}}
91+
92+
---
93+
94+
## ⚙️ Execution Details
95+
{{#if ctrf.tool.name}}![tool](https://ctrf.io/assets/github/tools.svg) **Tool**: {{ctrf.tool.name}}{{/if}}
96+
🔍 **Branch**: `{{github.branchName}}`
97+
👤 **Triggered by**: `{{github.actor}}`
98+
99+
---
100+
101+
{{#if ctrf.summary.failed}}
102+
## ❌ Failed Tests
103+
104+
{{#each ctrf.tests}}
105+
{{#if (eq this.status "fail")}}
106+
### 🔴 {{this.extra.feature_name}} - {{this.extra.scenario_name}}
107+
- ⏱️ Duration: {{formatDurationMs this.duration}}
108+
- 🔗 TestZeus Run: [View](https://prod.testzeus.app/test-runs/{{this.extra.test_run_id}})
109+
{{#if this.extra.test_data_id}}
110+
- 🧾 Test Data: [View](https://prod.testzeus.app/test-data/{{this.extra.test_data_id.[0]}})
111+
{{/if}}
112+
113+
{{#if (getCollapseLargeReports)}}
114+
<details>
115+
<summary><strong>View Steps</strong></summary>
116+
117+
{{#each this.steps}}
118+
- {{#if (eq this.status "fail")}}❌{{else if (eq this.status "pass")}}✅{{/if}} {{this.name}}
119+
{{/each}}
120+
121+
</details>
122+
{{else}}
123+
- **Steps**:
124+
{{#each this.steps}}
125+
- {{#if (eq this.status "fail")}}❌{{else if (eq this.status "pass")}}✅{{/if}} {{this.name}}
126+
{{/each}}
127+
{{/if}}
128+
129+
{{/if}}
130+
{{/each}}
131+
132+
{{/if}}
133+
```
134+
135+
---
136+
137+
## 🔔 Slack Notification (Optional)
138+
139+
Add a Slack message once the tests finish:
140+
141+
```yaml
142+
- name: Notify Slack
143+
uses: slackapi/slack-github-action@v1
144+
with:
145+
payload: |
146+
{
147+
"text": "🧪 TestZeus run finished – see details: ${{ github.run_url }}"
148+
}
149+
env:
150+
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
151+
if: always()
152+
```
153+
154+
---
155+
156+
## 📦 Upload Artifacts (Optional)
157+
158+
```yaml
159+
- name: Upload CTRF report
160+
uses: actions/upload-artifact@v4
161+
with:
162+
name: ctrf-report
163+
path: ctrf_report.json
164+
```
165+
166+
---
167+
168+
### CTRF Schema
169+
170+
The generated CTRF report follows the **Common Test Report Format (CTRF) v1.0.0** specification. The schema includes:
171+
172+
- **Report metadata**: Format version, specification version, and tool information
173+
- **Test summary**: Aggregate counts (total, passed, failed, pending, skipped, other) and execution timing
174+
- **Individual test results**: Each test includes:
175+
- Test identification (name, status, duration, timing)
176+
- Thread/execution context information
177+
- File attachments (screenshots, logs, artifacts)
178+
- Step-by-step execution details with individual step status
179+
- Extended metadata (tenant IDs, test run identifiers, feature/scenario names)
180+
181+
This standardized format ensures compatibility with CTRF-compliant tools and enables consistent test reporting across different testing frameworks.
182+
183+
#### Schema Example
184+
185+
```json
186+
{
187+
"reportFormat": "CTRF",
188+
"specVersion": "1.0.0",
189+
"results": {
190+
"tool": {
191+
"name": "testzeus",
192+
"version": "1.0.0"
193+
},
194+
"summary": {
195+
"tests": 5,
196+
"passed": 4,
197+
"failed": 1,
198+
"start": 1640995200,
199+
"stop": 1640995800
200+
},
201+
"tests": [
202+
{
203+
"name": "Login Test",
204+
"status": "pass",
205+
"duration": 2500,
206+
"steps": [
207+
{
208+
"name": "Enter credentials",
209+
"status": "pass"
210+
}
211+
],
212+
"attachments": [
213+
{
214+
"name": "image.png",
215+
"contentType": "png",
216+
"path": "<path/to/image>.png"
217+
}
218+
],
219+
"extra": {
220+
"tenantid": "abcd",
221+
"test_run_id": "abcd",
222+
"test_run_dash_id": "abcd",
223+
"agent_config_id": "abcd",
224+
"feature_name": "Authentication",
225+
"scenario_name": "Login to google"
226+
}
227+
}
228+
]
229+
}
230+
}
231+
```
232+
233+
---
234+
235+
## ℹ️ Tips
236+
237+
* Keep your `test_ids_file` under source control so reviewers can see exactly what is being executed.
238+
* Store credentials (`TESTZEUS_EMAIL`, `TESTZEUS_PASSWORD`) as **encrypted repository secrets**.
239+
* Pair this action with schedule triggers to run nightly regression suites.
240+
* The action installs dependencies with **Poetry** and invokes the CLI with `poetry run`.
241+
If you use a **self-hosted runner**, make sure Poetry is available on the machine (GitHub-hosted runners already include it).
242+
243+
Happy testing! 🚀

‎action.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: 'TestZeus Execute'
2+
description: 'Generate report from TestZeus using Test IDs'
3+
4+
inputs:
5+
test_ids_file:
6+
description: 'File containing test IDs'
7+
required: true
8+
email:
9+
description: 'TestZeus login email'
10+
required: true
11+
password:
12+
description: 'TestZeus login password'
13+
required: true
14+
15+
runs:
16+
using: 'composite'
17+
steps:
18+
- name: Install TestZeus
19+
shell: bash
20+
run: |
21+
echo "Installing TestZeus CLI..."
22+
poetry add testzeus-cli jq
23+
24+
- name: Validate Test IDs File
25+
shell: bash
26+
run: |
27+
FILE="${{ inputs.test_ids_file }}"
28+
29+
echo "🔍 Validating test IDs file: $FILE"
30+
31+
if [[ ! -f "$FILE" ]]; then
32+
echo "❌ File not found: $FILE"
33+
exit 1
34+
fi
35+
36+
if ! jq empty "$FILE" 2>/dev/null; then
37+
echo "❌ Invalid JSON in file: $FILE"
38+
exit 1
39+
fi
40+
41+
if [[ $(jq '.test_ids | type' "$FILE") != "\"array\"" ]]; then
42+
echo "❌ 'test_ids' must be an array"
43+
exit 1
44+
fi
45+
46+
if [[ $(jq '.execution_mode | type' "$FILE") != "\"string\"" ]]; then
47+
echo "❌ 'execution_mode' must be a string"
48+
exit 1
49+
fi
50+
51+
echo "✅ Test ID file validation passed."
52+
53+
- name: Login to TestZeus
54+
shell: bash
55+
run: |
56+
echo "🔐 Logging into TestZeus..."
57+
LOGIN_OUTPUT=$(poetry run testzeus login --email "${{ inputs.email }}" --password "${{ inputs.password }}" 2>&1)
58+
59+
if echo "$LOGIN_OUTPUT" | grep -q "Login failed"; then
60+
echo "❌ Login failed: aborting action."
61+
exit 1
62+
fi
63+
64+
echo "✅ Successfully logged into TestZeus."
65+
66+
- name: Extract Test IDs and Mode
67+
shell: bash
68+
run: |
69+
FILE="${{ inputs.test_ids_file }}"
70+
TEST_IDS=$(jq -r '.test_ids | join(",")' "$FILE")
71+
EXECUTION_MODE=$(jq -r '.execution_mode' "$FILE")
72+
73+
echo "TEST_IDS=$TEST_IDS" >> $GITHUB_ENV
74+
echo "EXECUTION_MODE=$EXECUTION_MODE" >> $GITHUB_ENV
75+
76+
echo "🧪 Extracted test IDs: $TEST_IDS"
77+
echo "⚙️ Execution mode: $EXECUTION_MODE"
78+
79+
- name: Generate CTRF Report
80+
shell: bash
81+
run: |
82+
echo "📊 Generating CTRF report..."
83+
poetry run testzeus --format json test-runs run-multiple-tests-and-generate-ctrf $TEST_IDS --mode $EXECUTION_MODE --interval 30 --filename ctrf_report.json
84+
if [[ -f "ctrf_report.json" ]]; then
85+
echo "✅ Report generated as ctrf_report.json"
86+
else
87+
echo "❌ Report generation failed: aborting action."
88+
exit 1
89+
fi

‎assets/test-results-summary.png‎

36.2 KB
Loading

‎pyproject.toml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
[project]
2+
name = "testzeus-execute"
3+
version = "0.1.0"
4+
description = ""
5+
authors = [
6+
{name = "Shahalt1",email = "shahalthayyil2020@gmail.com"}
7+
]
8+
readme = "README.md"
9+
requires-python = ">=3.11,<3.14"
10+
dependencies = [
11+
]
12+
13+
14+
[build-system]
15+
requires = ["poetry-core>=2.0.0,<3.0.0"]
16+
build-backend = "poetry.core.masonry.api"

0 commit comments

Comments
 (0)