Skip to content

Commit 4691a2b

Browse files
oberstetoberstet
authored
follow up heading up to new release v25.9.1 (crossbario#1678)
* follow up heading up to new release v25.9.1 * add note rgd packaging in project README * Update autobahn-testsuite Docker image to v25.10.1 and remove obsolete Makefiles: - Updated justfile to use crossbario/autobahn-testsuite:25.10.1 instead of :latest - Removed wstest/Makefile (functionality moved to justfile) - Removed docs/Makefile (functionality moved to justfile) - Kept docker/Makefile due to unique multi-arch Docker building capabilities * Refactor client testing to sequential execution and add conformance verification: - Remove matrix strategy from client-conformance job to run all 6 combinations sequentially - This ensures proper consolidation of all client test results in index.html/index.json - Add summary table generation from index.json files in consolidate-reports job - Add automatic PR comment posting with conformance summary tables - Create new verify-reports job that fails only if conformance is not 100% - Keep consolidation separate from verification for better debugging workflow * Fix artifact reconstruction to match new client naming scheme: - Update find patterns from 'client-*' to 'clients-all-*' - Update find patterns from 'servers-*' to 'servers-all-*' - Add better debugging output for artifact structure * also accept INFORMATIONAL/INFORMATIONAL as case success * Extract conformance verification logic into standalone Python script: - Create .github/workflows/verify_conformance.py to avoid shell escaping issues - Script accepts client and server JSON file paths as arguments - Handles both OK and INFORMATIONAL test results as passing - Provides detailed failure information for debugging - Simplify verify-reports workflow step to just call the Python script * Fix Python script paths and extract summary generation logic: - Fix verify_conformance.py path to use github.workspace - Create generate_summary.py script to avoid shell escaping issues - Update summary generation to accept INFORMATIONAL results as success - Simplify workflow steps to use standalone Python scripts - Both scripts now handle OK/OK and INFORMATIONAL/INFORMATIONAL as passing * Add checkout step to verify-reports job: The verify-reports job needs to checkout the code to access the Python verification script, otherwise the script file isn't available in the runner. * Fix f-string linting issue and add debugging for missing clients index.json: - Remove unnecessary f-prefix from empty string print - Add debugging output to understand consolidated reports directory structure - This will help identify why clients/index.json is not found * Fix artifact reconstruction patterns to match current naming: - Update patterns from 'client-*' to 'clients-all-*' - Update patterns from 'servers-*' to 'servers-all-*' - Add debugging to show available artifacts before processing - This should fix the missing clients directory issue * Add GitHub Actions permissions for PR commenting: - Add permissions block with issues:write and pull-requests:write - This fixes the 'Resource not accessible by integration' error - Allows the workflow to post conformance summary comments on PRs * Improve PR comment debugging and file path handling: - Add explicit github-token parameter to github-script action - Use absolute path for summary.md file reading - Add extensive debugging output for file paths and permissions - Better error handling with detailed logging - This should help identify the exact issue with PR commenting * maybe fix PR comment write permissions .. * having a workflow post a PR comment requires a PhD ..
1 parent 12c36f1 commit 4691a2b

11 files changed

Lines changed: 572 additions & 445 deletions

‎.audit/oberstet_rel_v25.9.1_part3.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- [ ] I did **not** use any AI-assistance tools to help create this pull request.
2+
- [x] I **did** use AI-assistance tools to *help* create this pull request.
3+
- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request.
4+
5+
Submitted by: @oberstet
6+
Date: 2025-10-02
7+
Related issue(s): #1677
8+
Branch: oberstet:rel_v25.9.1_part3

‎.github/workflows/generate_summary.py‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/usr/bin/env python3
2+
"""
3+
WebSocket conformance summary generator script for GitHub Actions.
4+
5+
This script processes a WebSocket conformance index.json file and generates
6+
a markdown table summarizing the test results for each testee.
7+
"""
8+
9+
import json
10+
import sys
11+
import argparse
12+
from pathlib import Path
13+
14+
15+
def generate_summary(json_file: Path, title: str) -> str:
16+
"""
17+
Generate a markdown summary table from an index.json file.
18+
19+
Args:
20+
json_file: Path to the index.json file
21+
title: Title for the summary section
22+
23+
Returns:
24+
Markdown formatted summary table as a string
25+
"""
26+
if not json_file.exists():
27+
return f"⚠️ {json_file} not found\n"
28+
29+
try:
30+
with open(json_file, 'r') as f:
31+
data = json.load(f)
32+
except (json.JSONDecodeError, IOError) as e:
33+
return f"❌ Error reading {json_file}: {e}\n"
34+
35+
# Build markdown table
36+
lines = [
37+
"",
38+
f"## {title}",
39+
"",
40+
"| Testee | Cases OK / Total | Status |",
41+
"|--------|------------------|---------|"
42+
]
43+
44+
for testee, cases in data.items():
45+
total_cases = len(cases)
46+
ok_cases = 0
47+
48+
for case_id, case_data in cases.items():
49+
behavior = case_data.get("behavior")
50+
behavior_close = case_data.get("behaviorClose")
51+
52+
# Test passes if both behaviors are OK, or both are INFORMATIONAL
53+
if (behavior == "OK" and behavior_close == "OK") or \
54+
(behavior == "INFORMATIONAL" and behavior_close == "INFORMATIONAL"):
55+
ok_cases += 1
56+
57+
status = '✅' if ok_cases == total_cases else '❌'
58+
lines.append(f'| {testee} | {ok_cases} / {total_cases} | {status} |')
59+
60+
return '\n'.join(lines)
61+
62+
63+
def main():
64+
"""Main entry point."""
65+
parser = argparse.ArgumentParser(description='Generate WebSocket conformance summary table')
66+
parser.add_argument('json_file', type=Path, help='Path to the index.json file')
67+
parser.add_argument('title', help='Title for the summary section')
68+
69+
args = parser.parse_args()
70+
71+
summary = generate_summary(args.json_file, args.title)
72+
print(summary)
73+
74+
75+
if __name__ == "__main__":
76+
main()

‎.github/workflows/post-summary.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# IMPORTANT: This workflow requires to create and add the fine-grained PAT as a secret:
2+
#
3+
# - Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens.
4+
# - Create a token for your repo with “Issues: Read & Write” and “Pull requests: Read & Write”.
5+
# - Copy the generated token value*
6+
# - Add it to your repo as a secret named GH_FINEGRAINED_TOKEN (Settings → Secrets and variables → Actions → New repository secret)
7+
# - Paste the token value* into the “Secret *” field.
8+
9+
on:
10+
workflow_run:
11+
workflows: ["wstest"] # <-- match exactly the name in your wstest.yml 'name:' field
12+
types:
13+
- completed
14+
15+
jobs:
16+
post-summary:
17+
runs-on: ubuntu-latest
18+
if: ${{ github.event.workflow_run.event == 'pull_request' }}
19+
steps:
20+
- name: Download summary artifact
21+
uses: actions/github-script@v7
22+
with:
23+
script: |
24+
const run_id = ${{ github.event.workflow_run.id }};
25+
const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
26+
owner: context.repo.owner,
27+
repo: context.repo.repo,
28+
run_id,
29+
});
30+
// Find the artifact whose name starts with 'conformance-summary-'
31+
const summaryArtifact = artifacts.data.artifacts.find(a => a.name.startsWith('conformance-summary-'));
32+
if (!summaryArtifact) {
33+
core.setFailed('No summary artifact found');
34+
return;
35+
}
36+
const download = await github.rest.actions.downloadArtifact({
37+
owner: context.repo.owner,
38+
repo: context.repo.repo,
39+
artifact_id: summaryArtifact.id,
40+
archive_format: 'zip',
41+
});
42+
const fs = require('fs');
43+
const unzip = require('unzipper');
44+
await fs.promises.writeFile('summary.zip', Buffer.from(download.data));
45+
await fs.createReadStream('summary.zip').pipe(unzip.Extract({ path: 'summary' })).promise();
46+
47+
- name: Post summary as PR comment
48+
uses: actions/github-script@v7
49+
env:
50+
GH_TOKEN: ${{ secrets.GH_FINEGRAINED_TOKEN }}
51+
with:
52+
script: |
53+
const fs = require('fs');
54+
// Find the summary.md file inside the unzipped artifact folder
55+
const summaryDir = 'summary';
56+
let summaryPath = null;
57+
// Find summary.md anywhere in the unzipped folder
58+
const walk = (dir) => {
59+
for (const file of fs.readdirSync(dir)) {
60+
const fullPath = `${dir}/${file}`;
61+
if (fs.statSync(fullPath).isFile() && file === 'summary.md') {
62+
summaryPath = fullPath;
63+
return;
64+
} else if (fs.statSync(fullPath).isDirectory()) {
65+
walk(fullPath);
66+
if (summaryPath) return;
67+
}
68+
}
69+
};
70+
walk(summaryDir);
71+
if (!summaryPath) {
72+
core.setFailed('summary.md not found in artifact');
73+
return;
74+
}
75+
const summary = fs.readFileSync(summaryPath, 'utf8');
76+
// Get PR number from workflow_run event
77+
const prs = github.event.workflow_run.pull_requests;
78+
if (!prs || prs.length === 0) {
79+
core.setFailed('No PR found in workflow_run event');
80+
return;
81+
}
82+
const pr_number = prs[0].number;
83+
await github.rest.issues.createComment({
84+
owner: context.repo.owner,
85+
repo: context.repo.repo,
86+
issue_number: pr_number,
87+
body: summary
88+
});

‎.github/workflows/verify_conformance.py‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/usr/bin/env python3
2+
"""
3+
WebSocket conformance verification script for GitHub Actions.
4+
5+
This script checks if all test cases in a WebSocket conformance index.json file
6+
have passed (behavior: "OK" and behaviorClose: "OK", or both "INFORMATIONAL").
7+
"""
8+
9+
import json
10+
import sys
11+
import argparse
12+
from pathlib import Path
13+
14+
15+
def verify_conformance(json_file: Path, test_type: str) -> bool:
16+
"""
17+
Verify 100% conformance for a given index.json file.
18+
19+
Args:
20+
json_file: Path to the index.json file
21+
test_type: Type of test (e.g., "Client" or "Server")
22+
23+
Returns:
24+
True if all tests passed, False otherwise
25+
"""
26+
if not json_file.exists():
27+
print(f"❌ {json_file} not found")
28+
return False
29+
30+
print(f"==> Checking {test_type} conformance...")
31+
32+
try:
33+
with open(json_file, 'r') as f:
34+
data = json.load(f)
35+
except (json.JSONDecodeError, IOError) as e:
36+
print(f"❌ Error reading {json_file}: {e}")
37+
return False
38+
39+
all_passed = True
40+
total_testees = len(data)
41+
passed_testees = 0
42+
43+
for testee, cases in data.items():
44+
total_cases = len(cases)
45+
ok_cases = 0
46+
failed_cases = []
47+
48+
for case_id, case_data in cases.items():
49+
behavior = case_data.get("behavior")
50+
behavior_close = case_data.get("behaviorClose")
51+
52+
# Test passes if both behaviors are OK, or both are INFORMATIONAL
53+
if (behavior == "OK" and behavior_close == "OK") or \
54+
(behavior == "INFORMATIONAL" and behavior_close == "INFORMATIONAL"):
55+
ok_cases += 1
56+
else:
57+
failed_cases.append({
58+
"case_id": case_id,
59+
"behavior": behavior,
60+
"behaviorClose": behavior_close
61+
})
62+
63+
if ok_cases == total_cases:
64+
print(f'✅ {testee}: {ok_cases}/{total_cases} tests passed')
65+
passed_testees += 1
66+
else:
67+
print(f'❌ {testee}: {ok_cases}/{total_cases} tests passed')
68+
# Show details of first few failed cases for debugging
69+
for i, failed_case in enumerate(failed_cases[:3]):
70+
print(f' Failed case {failed_case["case_id"]}: '
71+
f'behavior={failed_case["behavior"]}, '
72+
f'behaviorClose={failed_case["behaviorClose"]}')
73+
if len(failed_cases) > 3:
74+
print(f' ... and {len(failed_cases) - 3} more failed cases')
75+
all_passed = False
76+
77+
print('')
78+
print(f'{test_type} Summary: {passed_testees}/{total_testees} testees passed all tests')
79+
80+
if not all_passed:
81+
print(f'❌ {test_type} conformance: FAILED - Not all tests passed')
82+
return False
83+
else:
84+
print(f'✅ {test_type} conformance: PASSED - All tests passed')
85+
return True
86+
87+
88+
def main():
89+
"""Main entry point."""
90+
parser = argparse.ArgumentParser(description='Verify WebSocket conformance test results')
91+
parser.add_argument('client_json', type=Path, help='Path to client index.json file')
92+
parser.add_argument('server_json', type=Path, help='Path to server index.json file')
93+
94+
args = parser.parse_args()
95+
96+
print("==> Verifying 100% WebSocket conformance...")
97+
98+
client_passed = verify_conformance(args.client_json, "Client")
99+
print("")
100+
server_passed = verify_conformance(args.server_json, "Server")
101+
102+
print("")
103+
print("==> Overall WebSocket Conformance Verification:")
104+
105+
if client_passed and server_passed:
106+
print("✅ PASSED - Both client and server conformance tests achieved 100% pass rate")
107+
sys.exit(0)
108+
else:
109+
print("❌ FAILED - One or more conformance tests did not achieve 100% pass rate")
110+
print("")
111+
print("This means the WebSocket implementation has conformance issues that need to be addressed.")
112+
print("Download the detailed reports from the workflow artifacts to investigate specific failures.")
113+
sys.exit(1)
114+
115+
116+
if __name__ == "__main__":
117+
main()

0 commit comments

Comments
 (0)