Skip to content

Commit 9e01dce

Browse files
aqeelataqeelat
committed
feat(matrix): expand SDK generator matrix to 9 tools with parallel runner and CI workflow
Replace sequential 3-tool matrix (Orval, OpenAPI Generator, Swagger Codegen) with parallel 9-tool matrix adding openapi-typescript, @hey-api/openapi-ts, openapi-typescript-codegen, oazapfts, Kiota, and NSwag. - Add scripts/matrix-runner.mjs: parallel generation, typecheck, quality analysis with configurable concurrency, npx cache seeding, --no-generate mode for analysis-only runs - Add .github/workflows/matrix.yml: CI matrix running each (tool, scenario, version) as a separate parallel job with artifact collection and summary - Rewrite scripts/run-matrix.sh as thin wrapper delegating to matrix-runner.mjs - Add automated type quality analysis (PRESERVED/PARTIAL/DEGRADED/LOST/EMPTY) based on key property type fidelity rather than global pattern counts - Update RUNBOOK.md, README.md, state-of-the-union.md for new tools and docs - Add granular JetBrains .gitignore rules replacing blanket .idea/ ignore
1 parent ef524bd commit 9e01dce

8 files changed

Lines changed: 1061 additions & 116 deletions

File tree

‎.github/workflows/matrix.yml‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
name: SDK Generator Matrix
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
workflow_dispatch:
9+
inputs:
10+
tools:
11+
description: 'Comma-separated tool IDs (empty = all)'
12+
required: false
13+
default: ''
14+
scenarios:
15+
description: 'Comma-separated scenarios (empty = all)'
16+
required: false
17+
default: ''
18+
19+
env:
20+
NODE_VERSION: '22'
21+
22+
jobs:
23+
build-specs:
24+
runs-on: ubuntu-latest
25+
steps:
26+
- uses: actions/checkout@v4
27+
- uses: actions/setup-node@v4
28+
with:
29+
node-version: ${{ env.NODE_VERSION }}
30+
- run: npm install --no-save js-yaml
31+
- run: node scripts/build-specs.mjs
32+
- uses: actions/upload-artifact@v4
33+
with:
34+
name: specs
35+
path: specs/
36+
retention-days: 1
37+
38+
generate:
39+
needs: build-specs
40+
runs-on: ubuntu-latest
41+
strategy:
42+
fail-fast: false
43+
matrix:
44+
tool:
45+
- orval
46+
- openapi-generator
47+
- swagger-codegen
48+
- openapi-typescript
49+
- hey-api
50+
- openapi-typescript-codegen
51+
- oazapfts
52+
- kiota
53+
- nswag
54+
scenario:
55+
- baseline-duplicated-pagination
56+
- generic-schema-binding
57+
- paginated-response
58+
- recursive-category-tree
59+
- nested-workspace-resources
60+
version:
61+
- '3.1.0'
62+
- '3.1.1'
63+
- '3.1.2'
64+
- '3.2.0'
65+
66+
steps:
67+
- uses: actions/checkout@v4
68+
- uses: actions/setup-node@v4
69+
with:
70+
node-version: ${{ env.NODE_VERSION }}
71+
72+
- name: Check manual dispatch filter
73+
if: github.event_name == 'workflow_dispatch'
74+
run: |
75+
tools="${{ github.event.inputs.tools }}"
76+
scenarios="${{ github.event.inputs.scenarios }}"
77+
if [ -n "$tools" ]; then
78+
echo "$tools" | tr ',' '\n' | grep -qx "${{ matrix.tool }}" || exit 0
79+
fi
80+
if [ -n "$scenarios" ]; then
81+
echo "$scenarios" | tr ',' '\n' | grep -qx "${{ matrix.scenario }}" || exit 0
82+
fi
83+
84+
- uses: actions/download-artifact@v4
85+
with:
86+
name: specs
87+
path: specs/
88+
89+
- name: Install .NET tools
90+
if: matrix.tool == 'kiota' || matrix.tool == 'nswag'
91+
uses: actions/setup-dotnet@v4
92+
with:
93+
dotnet-version: '8.0.x'
94+
95+
- name: Install Kiota
96+
if: matrix.tool == 'kiota'
97+
run: dotnet tool install --global Microsoft.OpenApi.Kiota
98+
99+
- name: Install NSwag
100+
if: matrix.tool == 'nswag'
101+
run: dotnet tool install --global NSwag.GlobalTool
102+
103+
- name: Install npm dependencies
104+
run: npm install --no-save js-yaml typescript
105+
106+
- name: Generate SDK
107+
id: generate
108+
run: node scripts/matrix-runner.mjs --tools=${{ matrix.tool }} --scenarios=${{ matrix.scenario }} --versions=${{ matrix.version }} --no-typecheck --no-analysis
109+
continue-on-error: true
110+
111+
- name: Typecheck
112+
if: steps.generate.outcome == 'success'
113+
id: typecheck
114+
run: |
115+
output_dir="generated/${{ matrix.tool }}/${{ matrix.scenario }}/${{ matrix.version }}"
116+
if [ ! -d "$output_dir" ]; then
117+
echo "skip=true" >> "$GITHUB_OUTPUT"
118+
exit 0
119+
fi
120+
ts_files=$(find "$output_dir" -name "*.ts" -not -name "*.spec.ts" -not -name "*.test.ts" -not -path "*/node_modules/*" 2>/dev/null || true)
121+
if [ -z "$ts_files" ]; then
122+
echo "skip=true" >> "$GITHUB_OUTPUT"
123+
exit 0
124+
fi
125+
echo "$ts_files" | xargs npx tsc --noEmit --strict --esModuleInterop --moduleResolution node --target ES2020 --module commonjs --skipLibCheck 2>&1 | tee logs/typecheck-${{ matrix.tool }}-${{ matrix.scenario }}-${{ matrix.version }}.log
126+
exit ${PIPESTATUS[0]}
127+
continue-on-error: true
128+
129+
- name: Analyze quality
130+
if: steps.generate.outcome == 'success'
131+
run: node scripts/matrix-runner.mjs --tools=${{ matrix.tool }} --scenarios=${{ matrix.scenario }} --versions=${{ matrix.version }} --no-generate --no-typecheck
132+
133+
- name: Write cell result
134+
if: always()
135+
run: |
136+
mkdir -p cell-results
137+
gen_outcome="${{ steps.generate.outcome }}"
138+
tc_outcome="${{ steps.typecheck.outcome }}"
139+
tc_skip="${{ steps.typecheck.outputs.skip }}"
140+
echo "${{ matrix.tool }}.${{ matrix.scenario }}.${{ matrix.version }}.generate=${gen_outcome:-skipped}" >> cell-results/results.env
141+
if [ "$tc_skip" = "true" ]; then
142+
echo "${{ matrix.tool }}.${{ matrix.scenario }}.${{ matrix.version }}.typecheck=skip" >> cell-results/results.env
143+
else
144+
echo "${{ matrix.tool }}.${{ matrix.scenario }}.${{ matrix.version }}.typecheck=${tc_outcome:-skipped}" >> cell-results/results.env
145+
fi
146+
147+
- name: Upload generated output
148+
if: always()
149+
uses: actions/upload-artifact@v4
150+
with:
151+
name: generated-${{ matrix.tool }}-${{ matrix.scenario }}-${{ matrix.version }}
152+
path: |
153+
generated/${{ matrix.tool }}/${{ matrix.scenario }}/${{ matrix.version }}/
154+
logs/${{ matrix.tool }}-${{ matrix.scenario }}-${{ matrix.version }}.log
155+
logs/typecheck-${{ matrix.tool }}-${{ matrix.scenario }}-${{ matrix.version }}.log
156+
logs/matrix-results.json
157+
cell-results/results.env
158+
retention-days: 7
159+
if-no-files-found: ignore
160+
161+
summary:
162+
needs: generate
163+
runs-on: ubuntu-latest
164+
if: always()
165+
steps:
166+
- uses: actions/checkout@v4
167+
- uses: actions/setup-node@v4
168+
with:
169+
node-version: ${{ env.NODE_VERSION }}
170+
171+
- uses: actions/download-artifact@v4
172+
with:
173+
pattern: generated-*
174+
path: all-results/
175+
merge-multiple: true
176+
177+
- name: Build summary
178+
run: |
179+
echo "## SDK Generator Matrix Results" >> "$GITHUB_STEP_SUMMARY"
180+
echo "" >> "$GITHUB_STEP_SUMMARY"
181+
echo "| Tool | Scenario | Version | Generate | Typecheck |" >> "$GITHUB_STEP_SUMMARY"
182+
echo "|------|----------|---------|----------|-----------|" >> "$GITHUB_STEP_SUMMARY"
183+
184+
for envf in all-results/cell-results/results.env; do
185+
[ -f "$envf" ] || continue
186+
while IFS='=' read -r key value; do
187+
IFS='.' read -r tool scenario version field <<< "$key"
188+
[ "$field" = "generate" ] || continue
189+
tc_key="${tool}.${scenario}.${version}.typecheck"
190+
tc_value=$(grep "^${tc_key}=" all-results/cell-results/results.env 2>/dev/null | cut -d= -f2)
191+
gen_label="${value}"
192+
tc_label="${tc_value:-N/A}"
193+
echo "| ${tool} | ${scenario} | ${version} | ${gen_label} | ${tc_label} |" >> "$GITHUB_STEP_SUMMARY"
194+
done < "$envf"
195+
done
196+
197+
echo "" >> "$GITHUB_STEP_SUMMARY"
198+
echo "### Artifacts" >> "$GITHUB_STEP_SUMMARY"
199+
echo "Generated output and logs are available as workflow artifacts." >> "$GITHUB_STEP_SUMMARY"
200+
201+
- name: Load quality results
202+
run: |
203+
echo "### Type Quality" >> "$GITHUB_STEP_SUMMARY"
204+
echo "" >> "$GITHUB_STEP_SUMMARY"
205+
if [ -f all-results/logs/matrix-results.json ]; then
206+
node -e "
207+
const data = JSON.parse(require('fs').readFileSync('all-results/logs/matrix-results.json', 'utf8'));
208+
console.log('| Tool | Scenario | Version | Fidelity | unknown | any |');
209+
console.log('|------|----------|---------|----------|---------|-----|');
210+
for (const r of (data.results || [])) {
211+
if (r.quality) {
212+
console.log('| ' + [r.tool, r.scenario, r.version, r.quality.fidelity, r.quality.unknownCount, r.quality.anyCount].join(' | ') + ' |');
213+
}
214+
}
215+
" >> "$GITHUB_STEP_SUMMARY"
216+
else
217+
echo "Quality data not available" >> "$GITHUB_STEP_SUMMARY"
218+
fi

‎.gitignore‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,92 @@ node_modules/
44
openapitools.json
55
generated/
66
logs/
7+
all-results/
8+
9+
# Covers JetBrains IDEs: IntelliJ, GoLand, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
10+
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
11+
12+
# User-specific stuff
13+
.idea/**/workspace.xml
14+
.idea/**/tasks.xml
15+
.idea/**/usage.statistics.xml
16+
.idea/**/dictionaries
17+
.idea/**/shelf
18+
19+
# AWS User-specific
20+
.idea/**/aws.xml
21+
22+
# Generated files
23+
.idea/**/contentModel.xml
24+
25+
# Sensitive or high-churn files
26+
.idea/**/dataSources/
27+
.idea/**/dataSources.ids
28+
.idea/**/dataSources.local.xml
29+
.idea/**/sqlDataSources.xml
30+
.idea/**/dynamic.xml
31+
.idea/**/uiDesigner.xml
32+
.idea/**/dbnavigator.xml
33+
34+
# Gradle
35+
.idea/**/gradle.xml
36+
.idea/**/libraries
37+
38+
# Gradle and Maven with auto-import
39+
# When using Gradle or Maven with auto-import, you should exclude module files,
40+
# since they will be recreated, and may cause churn. Uncomment if using
41+
# auto-import.
42+
# .idea/artifacts
43+
# .idea/compiler.xml
44+
# .idea/jarRepositories.xml
45+
# .idea/modules.xml
46+
# .idea/*.iml
47+
# .idea/modules
48+
# *.iml
49+
# *.ipr
50+
51+
# CMake
52+
cmake-build-*/
53+
54+
# Mongo Explorer plugin
55+
.idea/**/mongoSettings.xml
56+
57+
# File-based project format
58+
*.iws
59+
60+
# IntelliJ
61+
out/
62+
63+
# mpeltonen/sbt-idea plugin
64+
.idea_modules/
65+
66+
# JIRA plugin
67+
atlassian-ide-plugin.xml
68+
69+
# Cursive Clojure plugin
70+
.idea/replstate.xml
71+
72+
# SonarLint plugin
73+
.idea/sonarlint/
74+
# see https://community.sonarsource.com/t/is-the-file-idea-idea-idea-sonarlint-xml-intended-to-be-under-source-control/121119
75+
.idea/sonarlint.xml
76+
77+
# Crashlytics plugin (for Android Studio and IntelliJ)
78+
com_crashlytics_export_strings.xml
79+
crashlytics.properties
80+
crashlytics-build.properties
81+
fabric.properties
82+
83+
# Editor-based HTTP Client
84+
.idea/httpRequests
85+
http-client.private.env.json
86+
87+
# Android studio 3.1+ serialized cache file
88+
.idea/caches/build_file_checksums.ser
89+
90+
# Apifox Helper cache
91+
.idea/.cache/.Apifox_Helper
92+
.idea/ApifoxUploaderProjectSetting.xml
93+
94+
# Github Copilot persisted session migrations, see: https://github.com/microsoft/copilot-intellij-feedback/issues/712#issuecomment-3322062215
95+
.idea/**/copilot.data.migration.*.xml

‎README.md‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -127,10 +127,13 @@ specs/ Generated OAS-version matrix specs
127127
<fixture>/oas-<version>.json
128128
scripts/
129129
validate-and-build.sh Stage 1: validate fixtures + rebuild specs (run when fixtures change)
130-
run-matrix.sh Stage 2: SDK generation + typecheck (run when generator versions change)
130+
run-matrix.sh Stage 2: SDK generation + typecheck (parallel, all tools)
131+
matrix-runner.mjs Matrix engine: parallel generation, typecheck, quality analysis
131132
validate-openapi.sh OpenAPI doc validators (Redocly, openapi-spec-validator, Spectral, swagger-cli)
132133
build-specs.mjs Generates specs/ from fixtures/
133134
validate-jsonschema.mjs Standalone: AJV + Hyperjump runtime validation (not in pipeline)
135+
.github/workflows/
136+
matrix.yml CI matrix: each (tool, scenario, version) as a parallel job
134137
LICENSE MIT
135138
orval.config.ts Orval generation matrix config
136139
IMPLEMENTATION_GUIDE.md Agent playbook for fixing generators
@@ -166,24 +169,35 @@ Versioned specs are generated from fixtures into `specs/<fixture>/oas-<version>.
166169
- [x] Add inline-binding pagination variant (type binding at route response level)
167170
- [x] Run SDK generator matrix against all fixtures (Orval, OpenAPI Generator, Swagger Codegen)
168171
- [ ] Investigate AJV's behavior on the pagination/generic-wrapper fixture
169-
- [ ] Add more TypeScript tools (`openapi-typescript-codegen`, `oazapfts`, `@hey-api/openapi-ts`)
170-
- [ ] Add non-TypeScript generators (Java, C#, Python, Go, Rust, Kotlin, Swift, AutoRest, NSwag, Kiota)
172+
- [x] Add more TypeScript tools (`openapi-typescript-codegen`, `oazapfts`, `@hey-api/openapi-ts`, `openapi-typescript`)
173+
- [ ] Add non-TypeScript generators (Java, C#, Python, Go, Rust, Kotlin, Swift, AutoRest)
174+
- [x] Add standalone install generators (Kiota, NSwag)
171175
- [ ] Open focused upstream issues with validator-backed fixtures; include disagreement details for mixed-support fixtures
172176

173177
See the **[Implementation Guide](IMPLEMENTATION_GUIDE.md)** for a step-by-step playbook to implement `$dynamicRef` support in any generator. Use validator-backed fixtures first; include validator disagreement when using mixed-support fixtures.
174178

175179
## 📢 Outreach
176180

177-
Issues and PRs opened in upstream generator repos.
181+
Issues and PRs opened in upstream repos.
182+
183+
### SDK Generators
178184

179185
| Generator | Repo | Issue | PR | Status | Updated |
180186
|---|---|---|---|---|---|
181187
| Orval | [orval-labs/orval](https://github.com/orval-labs/orval) | — | — | not-started | — |
182-
| OpenAPI Generator | [OpenAPITools/openapi-generator](https://github.com/OpenAPITools/openapi-generator) | — | — | not-started | — |
183-
| Swagger Codegen v3 | [swagger-api/swagger-codegen](https://github.com/swagger-api/swagger-codegen) | — | — | not-started | — |
188+
| OpenAPI Generator | [OpenAPITools/openapi-generator](https://github.com/OpenAPITools/openapi-generator) | [#23776](https://github.com/OpenAPITools/openapi-generator/issues/23776) | — | blocked | 2026-05-12 |
189+
| Swagger Codegen v3 | [swagger-api/swagger-codegen](https://github.com/swagger-api/swagger-codegen) | [#12731](https://github.com/swagger-api/swagger-codegen/issues/12731) | — | blocked | 2026-05-12 |
190+
191+
### Validators & Parsers
192+
193+
| Tool | Repo | Issue | PR | Status | Updated |
194+
|---|---|---|---|---|---|
184195
| AJV | [ajv-validator/ajv](https://github.com/ajv-validator/ajv) | [#1573](https://github.com/ajv-validator/ajv/issues/1573), [#1745](https://github.com/ajv-validator/ajv/issues/1745) | [#2615](https://github.com/ajv-validator/ajv/pull/2615) | in-progress | 2026-05-12 |
196+
| swagger-parser | [swagger-api/swagger-parser](https://github.com/swagger-api/swagger-parser) | — | [#2332](https://github.com/swagger-api/swagger-parser/pull/2332) | pr-open | 2026-05-12 |
197+
198+
> **Note:** swagger-parser is a dependency of OpenAPI Generator and Swagger Codegen. Fixing `$dynamicRef`/`$dynamicAnchor` support in swagger-parser is a prerequisite for adding codegen support in those generators.
185199
186-
**Status values:** `not-started` · `in-progress` · `pr-open` · `pr-stale` · `merged` · `rejected` · `superseded` · `wontfix` · `resolved`
200+
**Status values:** `not-started` · `in-progress` · `pr-open` · `pr-stale` · `merged` · `rejected` · `superseded` · `wontfix` · `resolved` · `blocked`
187201

188202
## 🤝 Contributing
189203

0 commit comments

Comments
 (0)