Skip to content

Commit e18962e

Browse files
fugazifugazi
committed
Enhance API Testing Framework
- Added a Constitution section to AGENTS.md to define non-negotiable rules for agents. - Updated CLAUDE.md to include new skills for API Testing and CI/CD Pipelines. - Expanded README.md with instructions for adding API Testing skills and CI/CD pipeline configuration. - Created cicd-testing.instructions.md for detailed CI/CD test pipeline setup using GitHub Actions. - Introduced api-testing skill with comprehensive documentation for REST and GraphQL testing. - Added contract-testing.md to outline patterns for verifying API contracts. - Developed playwright-api-testing.md for detailed API testing strategies using Playwright. - Created rest-api-patterns.md to document common REST API testing patterns. - Implemented rest-assured-testing.md for Java-based API testing with REST Assured. - Added schema-validation.md for validation patterns in TypeScript and Java. - Created api-health-check.sh script for validating API endpoint responses. - Developed templates for Playwright and REST Assured API tests to streamline test creation.
1 parent cf76c32 commit e18962e

13 files changed

Lines changed: 1231 additions & 2 deletions

‎AGENTS.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,16 @@ Follow this structure in agent files:
7676

7777
Clear statement of who the agent is and its primary role.
7878

79+
## Constitution (from TOP)
80+
81+
### MUST DO
82+
83+
- [5-6 rules that are NON-NEGOTIABLE for this agent]
84+
85+
### WON'T DO
86+
87+
- [4-5 rules that this agent must NEVER violate]
88+
7989
## Core Responsibilities
8090

8191
- List specific tasks the agent performs
@@ -96,6 +106,10 @@ Clear statement of who the agent is and its primary role.
96106
- Expected format and quality
97107
```
98108

109+
### Constitution Pattern
110+
111+
Agents that generate or modify test code should include a `## Constitution (from TOP)` section defining hard guardrails. The Constitution follows the Test Orchestration Pattern (TOP) and is defined centrally in `agents/qa-orchestrator.agent.md`. Individual agents copy the relevant MUST DO / WON'T DO rules into their own Constitution section so the rules are visible at the agent level.
112+
99113
### Skill Content Structure
100114

101115
```markdown

‎CLAUDE.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,11 @@ It contains custom agent definitions and skill instructions for:
1414

1515
- **Playwright** (TypeScript) - End-to-end browser automation
1616
- **Selenium WebDriver** (Java 21+) - UI testing with JUnit 5 and AssertJ
17+
- **API Testing** - REST/GraphQL testing with Playwright request fixture and REST Assured
1718
- **Accessibility Testing** - WCAG 2.1 AA compliance testing
1819
- **Manual QA** - ISTQB-based testing practices
1920
- **Test Planning** - Test strategy and documentation
21+
- **CI/CD Pipelines** - GitHub Actions test automation workflows
2022

2123
**Important:** This repository has **no build system**. Files are Markdown with YAML frontmatter. No build, lint, or test commands are required.
2224

@@ -208,6 +210,18 @@ When creating orchestrator agents that delegate to sub-agents:
208210
- Modern Java: Records, Streams, Optional, Pattern Matching
209211
- See `instructions/selenium-webdriver-java.instructions.md`
210212

213+
### API Testing
214+
- Playwright request fixture and REST Assured (Java 21+)
215+
- Schema validation (Zod for TS, JSON Schema Validator for Java)
216+
- Contract testing, idempotency, authentication flows
217+
- See `skills/api-testing/`
218+
219+
### CI/CD Test Pipelines
220+
- Tiered test system: smoke → sanity → selective → full regression
221+
- GitHub Actions workflows with sharding and parallel execution
222+
- Deployment gates, flaky test handling, failure notifications
223+
- See `instructions/cicd-testing.instructions.md`
224+
211225
### Accessibility Testing
212226
- WCAG 2.1 AA compliance
213227
- Use axe-core or Playwright's built-in accessibility tree
@@ -244,3 +258,4 @@ When creating new agents or skills:
244258
- [Orchestration Workflow](./instructions/orchestration-workflow.instructions.md) — TOP 8-step workflow for AI-driven test automation
245259
- [File Map Template (Playwright)](./skills/playwright-e2e-testing/references/file-map-template.md) — Recommended project structure for Playwright TypeScript
246260
- [File Map Template (Selenium)](./skills/webapp-selenium-testing/references/file-map-template.md) — Recommended project structure for Selenium Java
261+
- [CI/CD Test Pipelines](./instructions/cicd-testing.instructions.md) — GitHub Actions workflows, test tiers, parallel execution

‎README.md‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,9 @@ npx skills add https://github.com/fugazi/test-automation-skills-agents --skill p
125125
```bash
126126
npx skills add https://github.com/fugazi/test-automation-skills-agents --skill playwright-cli
127127
```
128+
```bash
129+
npx skills add https://github.com/fugazi/test-automation-skills-agents --skill api-testing
130+
```
128131

129132
---
130133

@@ -164,6 +167,7 @@ Instructions are cross-cutting rules that keep outputs consistent:
164167
- Playwright coding standards, locator strategy, POM patterns
165168
- Selenium Java standards, explicit waits, AssertJ, Allure
166169
- Accessibility expectations (WCAG 2.1 AA)
170+
- CI/CD test pipeline configuration (GitHub Actions, test tiers, parallel execution)
167171
- Agent authoring guidelines (frontmatter, handoffs, tool selection)
168172

169173
In practice:
@@ -254,6 +258,8 @@ Recommended pattern:
254258

255259
This repo currently includes the following agents (see `agents/`):
256260

261+
- **QA Orchestrator**: routes test tasks to specialist agents, enforces Test Constitution
262+
- **Architect**: orchestrator-style agent (delegation-focused)
257263
- **API Tester Specialist**: API test creation (REST Assured / Playwright API / Supertest), auth, contracts, schemas
258264
- **Flaky Test Hunter**: identifies root causes of flaky tests, applies stabilization strategies
259265
- **Playwright Test Planner**: explores an app and produces a structured test plan
@@ -265,7 +271,6 @@ This repo currently includes the following agents (see `agents/`):
265271
- **Docs Agent**: technical writer focused on docs output (note: its default paths reference a typical app repo)
266272
- **Implementation Plan Generation Mode**: produces deterministic implementation plans (no code changes)
267273
- **Principal Software Engineer**: principal-level guidance (architecture, quality, pragmatic trade-offs)
268-
- **Architect**: orchestrator-style agent (delegation-focused)
269274

270275
> Note: Some agents are intended for orchestration or cross-repo workflows. Use the specialists for daily QA tasks.
271276
@@ -309,6 +314,7 @@ Typical triggers:
309314
| `playwright-regression-testing` | Regression strategy + test selection + CI/CD optimization | “Organize tests into tiers (smoke, selective, full) and set up GitHub Actions pipeline.” |
310315
| `qa-manual-istqb` | ISTQB-aligned artifacts + test design techniques | “Create a risk-based regression suite and a traceability matrix.” |
311316
| `qa-test-planner` | Test plans + test cases + bug reports + Playwright artifacts | “Use the skill qa-test-planner to create a test plan for payments.” |
317+
| `api-testing` | REST/GraphQL testing with Playwright and REST Assured | “Create API tests for user endpoints with schema validation.” |
312318

313319
> Note: `qa-test-planner` is intentionally strict: it is designed to trigger only when you call it by name.
314320
@@ -371,6 +377,18 @@ If a skill still does not activate automatically:
371377
- Schema/contract checks
372378
- Idempotency where relevant
373379
- Pagination/sorting/filtering edge cases
380+
3. Use `api-testing` skill for schema validation patterns (Zod, JSON Schema) and contract testing.
381+
382+
### Workflow 6 — CI/CD pipeline setup
383+
384+
1. Use `cicd-testing` instruction for pipeline configuration guidance.
385+
2. Set up tiered GitHub Actions workflows:
386+
- Smoke (every commit, < 2 min)
387+
- Sanity (every PR, < 10 min)
388+
- Selective regression (on merge, < 30 min)
389+
- Full regression (nightly, < 60 min)
390+
3. Configure parallel execution with sharding.
391+
4. Add deployment gates, flaky test handling, and failure notifications.
374392

375393
## Contributing
376394

@@ -416,7 +434,7 @@ If a skill still does not activate automatically:
416434
* Job: `SDET - Software Developer Engineer in Test`
417435
* Country: `Colombia`
418436
* City: `Medellin`
419-
* E-mail: `douglas@douglasfugazi.co`
437+
* E-mail: `info@douglasfugazi.co`
420438
* LinkedIn: [https://www.linkedin.com/in/douglasfugazi](https://www.linkedin.com/in/douglasfugazi)
421439
* Contact: [https://douglasfugazi.co](https://douglasfugazi.co)
422440

‎instructions/cicd-testing.instructions.md‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
---
2+
description: "CI/CD test pipeline configuration for automated testing. Defines GitHub Actions workflows, test tiers, parallel execution, reporting, and deployment gates. Apply when creating or modifying CI/CD pipeline files for test automation."
3+
applyTo: "**/.github/workflows/*.yml, **/.github/workflows/*.yaml, **/Jenkinsfile, **/.gitlab-ci.yml"
4+
---
5+
6+
# CI/CD Test Pipeline Configuration
7+
8+
Instructions for setting up automated test pipelines using GitHub Actions. Covers test tier organization, parallel execution, reporting, deployment gates, and flaky test handling.
9+
10+
## Test Tier System
11+
12+
Organize tests into tiers that run from fastest/most-critical to slowest/broadest:
13+
14+
| Tier | Scope | Duration | Trigger | Browsers |
15+
|------|-------|----------|---------|----------|
16+
| **Tier 0: Smoke** | Critical path (5-10 tests) | < 2 min | Every commit | 1 browser (chromium) |
17+
| **Tier 1: Sanity** | Core features (20-50 tests) | < 10 min | Every PR | 2 browsers |
18+
| **Tier 2: Selective** | Change-based | < 30 min | On merge | All browsers |
19+
| **Tier 3: Full** | Complete regression | < 60 min | Nightly / pre-release | All browsers + mobile |
20+
21+
## Tagging Strategy
22+
23+
Tag tests by tier for selective execution:
24+
25+
```typescript
26+
// In test files, tag tests by tier
27+
test('login works @smoke @sanity', async ({ loginPage }) => { ... });
28+
test('full checkout flow @regression', async ({ checkoutPage }) => { ... });
29+
test('search filters @sanity @regression', async ({ searchPage }) => { ... });
30+
```
31+
32+
Run by tag from CLI:
33+
34+
```bash
35+
npx playwright test -g "@smoke" # Tier 0
36+
npx playwright test -g "@sanity" # Tier 1
37+
npx playwright test # All — includes @regression
38+
```
39+
40+
## Smoke Pipeline (Tier 0)
41+
42+
Runs on every commit. Fast feedback loop for developers.
43+
44+
```yaml
45+
name: Smoke Tests
46+
on: [push]
47+
48+
jobs:
49+
smoke:
50+
runs-on: ubuntu-latest
51+
steps:
52+
- uses: actions/checkout@v4
53+
- uses: actions/setup-node@v4
54+
with:
55+
node-version: 20
56+
- run: npm ci
57+
- run: npx playwright install --with-deps chromium
58+
- run: npx playwright test --project=chromium -g "@smoke"
59+
- uses: actions/upload-artifact@v4
60+
if: failure()
61+
with:
62+
name: smoke-results
63+
path: test-results/
64+
```
65+
66+
## Full Regression Pipeline (Tier 3)
67+
68+
Runs nightly and on-demand. Cross-browser, sharded for speed.
69+
70+
```yaml
71+
name: Full Regression
72+
on:
73+
schedule:
74+
- cron: "0 2 * * *" # 2 AM UTC daily
75+
workflow_dispatch:
76+
77+
jobs:
78+
test:
79+
runs-on: ubuntu-latest
80+
strategy:
81+
fail-fast: false
82+
matrix:
83+
project: [chromium, firefox, webkit]
84+
shard: [1/4, 2/4, 3/4, 4/4]
85+
steps:
86+
- uses: actions/checkout@v4
87+
- uses: actions/setup-node@v4
88+
with:
89+
node-version: 20
90+
- run: npm ci
91+
- run: npx playwright install --with-deps
92+
- run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shard }}
93+
- uses: actions/upload-artifact@v4
94+
if: failure()
95+
with:
96+
name: results-${{ matrix.project }}-${{ matrix.shard }}
97+
path: test-results/
98+
99+
report:
100+
needs: test
101+
if: always()
102+
runs-on: ubuntu-latest
103+
steps:
104+
- uses: actions/checkout@v4
105+
- uses: actions/download-artifact@v4
106+
with:
107+
path: test-results/
108+
- run: npx playwright show-report
109+
```
110+
111+
## Parallel Execution with Sharding
112+
113+
Split tests across multiple CI runners for faster execution:
114+
115+
```yaml
116+
strategy:
117+
fail-fast: false
118+
matrix:
119+
shard: [1/4, 2/4, 3/4, 4/4]
120+
```
121+
122+
```bash
123+
# Each runner executes its shard
124+
npx playwright test --shard=${{ matrix.shard }}
125+
```
126+
127+
## Playwright Reporter Configuration
128+
129+
Configure reporters for CI and local environments:
130+
131+
```typescript
132+
// playwright.config.ts
133+
export default defineConfig({
134+
reporter: process.env.CI
135+
? [
136+
['html', { open: 'never' }],
137+
['json', { outputFile: 'test-results/results.json' }],
138+
['github'],
139+
]
140+
: [['list']],
141+
});
142+
```
143+
144+
## Deployment Gates
145+
146+
Enforce test quality before promoting to higher environments:
147+
148+
- **Smoke must pass** before deploy to staging
149+
- **Full regression must pass** before deploy to production
150+
- **Performance thresholds** must be met for production releases
151+
152+
Example gate in workflow:
153+
154+
```yaml
155+
deploy-staging:
156+
needs: smoke
157+
runs-on: ubuntu-latest
158+
steps:
159+
- run: echo "Deploying to staging..."
160+
161+
deploy-production:
162+
needs: [smoke, full-regression]
163+
runs-on: ubuntu-latest
164+
steps:
165+
- run: echo "Deploying to production..."
166+
```
167+
168+
## Test Data Management in CI
169+
170+
- Use **environment variables** for secrets (never hardcode)
171+
- Use **CI-specific test data** (never production data)
172+
- **Clean up test data** after each run
173+
- **Seed database** before test suite, truncate after
174+
175+
```yaml
176+
env:
177+
BASE_URL: http://localhost:3000
178+
TEST_USER_EMAIL: ${{ secrets.TEST_USER_EMAIL }}
179+
TEST_USER_PASSWORD: ${{ secrets.TEST_USER_PASSWORD }}
180+
```
181+
182+
## Flaky Test Handling in CI
183+
184+
Configure retries and artifact capture for CI-only:
185+
186+
```typescript
187+
// playwright.config.ts
188+
export default defineConfig({
189+
retries: process.env.CI ? 2 : 0, // Retry only in CI
190+
use: {
191+
trace: 'on-first-retry', // Capture trace on retry
192+
screenshot: 'only-on-failure',
193+
video: 'retain-on-failure',
194+
},
195+
});
196+
```
197+
198+
## Failure Notifications
199+
200+
Alert the team when tests fail in CI:
201+
202+
```yaml
203+
- name: Notify on failure
204+
if: failure()
205+
uses: slackapi/slack-github-action@v1
206+
with:
207+
payload: |
208+
{
209+
"text": "Test suite failed on ${{ github.ref }}",
210+
"blocks": [
211+
{
212+
"type": "section",
213+
"text": {
214+
"type": "mrkdwn",
215+
"text": "*Test Failure*\nBranch: ${{ github.ref }}\nCommit: ${{ github.sha }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
216+
}
217+
}
218+
]
219+
}
220+
env:
221+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
222+
```

0 commit comments

Comments
 (0)