Skip to content

Commit b0bce6f

Browse files
aborrusoclaude
aborruso
and
claude
committed
feat: add schema validation monitoring proposal
OpenSpec proposal for automated weekly validation of: - HTML parameter extraction from normattiva.it - XML Akoma Ntoso schema stability - Automatic GitHub issue creation on failures Simplified approach: validates only structural dependencies, not full conversion. Fast execution (~1min), focused diagnostics. Also updated ROADMAP.md with v2.0.3-v2.0.7 releases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent a26a001 commit b0bce6f

5 files changed

Lines changed: 448 additions & 2 deletions

File tree

‎ROADMAP.md‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,37 @@ akoma2md input.xml output.md
5151

5252
---
5353

54+
## ✅ COMPLETED - v2.0.3 to v2.0.7 (2025-11-04)
55+
56+
### Search & UX Enhancements
57+
58+
**Status**: ✅ Completed
59+
60+
#### Versioni Rilasciate
61+
- **v2.0.7** (2025-11-04): Permanent link field in frontmatter
62+
- **v2.0.6** (2025-11-04): Cleaner search output & improved UX
63+
- **v2.0.5** (2025-11-04): Enhanced Exa search scoring & URL conversion
64+
- **v2.0.4** (2025-11-04): Enhanced article recognition in search
65+
- **v2.0.3** (2025-11-04): README update on PyPI (metadata only)
66+
67+
#### Funzionalità Implementate
68+
- ✅ Campo `url_permanente` nel frontmatter YAML con URN canonico e vigenza
69+
- ✅ Output ricerca più pulito: verbose mode solo con `--debug-search`
70+
- ✅ Scoring Exa migliorato: priorità leggi complete vs articoli specifici
71+
- ✅ Conversione automatica URL articolo→legge completa quando appropriato
72+
- ✅ Riconoscimento articoli esteso: "articolo 7", "art 7", "art. 7", numeri complessi (16bis, 16ter)
73+
- ✅ Bonus/penalità intelligenti per selezione risultati Exa
74+
- ✅ Metadata PyPI aggiornati e corretti
75+
76+
#### Example Output
77+
```yaml
78+
---
79+
url_permanente: https://www.normattiva.it/uri-res/N2Ls?urn:nir:stato:decreto.legislativo:2005-03-07;82!vig=2025-01-30
80+
---
81+
```
82+
83+
---
84+
5485
## 🟡 NEXT RELEASE - v2.1.0 (Prossima Release)
5586

5687
### 1. Refactoring `main()` function
@@ -753,5 +784,5 @@ akoma2md/
753784

754785
---
755786

756-
**Ultimo aggiornamento**: 2025-11-04
757-
**Versione corrente**: v2.0.2
787+
**Ultimo aggiornamento**: 2025-12-03
788+
**Versione corrente**: v2.0.18

‎openspec/changes/add-conversion-validation-monitoring/design.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Design: Schema Validation Monitoring
2+
3+
## Architecture Overview
4+
5+
This change introduces automated monitoring of structural dependencies (HTML parsing, XML schema) through lightweight scheduled validation and issue reporting.
6+
7+
```
8+
┌─────────────────────────────────────────────────────────────┐
9+
│ GitHub Actions │
10+
│ ┌───────────────────────────────────────────────────────┐ │
11+
│ │ schema-validation.yml (weekly schedule) │ │
12+
│ │ │ │
13+
│ │ 1. Checkout code │ │
14+
│ │ 2. Setup Python │ │
15+
│ │ 3. Run schema tests ──────────────┐ │ │
16+
│ │ 4. Parse results │ │ │
17+
│ │ 5. Create issue if failed ────────┼──────┐ │ │
18+
│ └────────────────────────────────────┼──────┼───────────┘ │
19+
└───────────────────────────────────────┼──────┼───────────────┘
20+
│ │
21+
┌─────────────────────┘ │
22+
│ │
23+
▼ ▼
24+
┌──────────────────────────┐ ┌──────────────────────┐
25+
│ test_schema_ │ │ GitHub Issues API │
26+
│ validation.py │ │ │
27+
│ │ │ Title: Schema │
28+
│ 4 tests (~1min): │ │ validation │
29+
│ - HTML params OK? │ │ failed │
30+
│ - XML namespace OK? │ │ Assignee: aborruso │
31+
│ - XML elements OK? │ │ Labels: bug, auto │
32+
│ - Download OK? │ │ │
33+
└──────────┬───────────────┘ └──────────────────────┘
34+
│
35+
▼
36+
┌──────────────────────────┐
37+
│ normattiva.it │
38+
│ (single sample doc) │
39+
│ │
40+
│ - Download HTML page │
41+
│ - Extract params │
42+
│ - Download XML │
43+
│ - Verify schema │
44+
└──────────────────────────┘
45+
```
46+
47+
## Key Design Decisions
48+
49+
### 1. Test Scope: Schema Only, Not Full Conversion
50+
**Decision**: Validate only HTML parsing and XML schema, skip Markdown conversion
51+
52+
**Rationale**:
53+
- Focused validation: catches the actual breaking points (external dependencies)
54+
- Fast execution: < 1 minute, suitable for weekly schedule
55+
- Fewer false positives: conversion bugs are separate from schema changes
56+
- Clear diagnostics: failure points to specific structural change
57+
58+
**Alternatives Considered**:
59+
- Full conversion testing → rejected (too slow, too many failure modes)
60+
- Only unit tests → rejected (doesn't detect normattiva.it changes)
61+
- Manual monitoring → rejected (requires human attention)
62+
63+
### 2. Single Sample Document
64+
**Decision**: Test against one representative legal document (CAD decree)
65+
66+
**Rationale**:
67+
- Sufficient coverage: one document exercises all structural dependencies
68+
- Fast execution: single download + parse
69+
- Stable baseline: same document enables change detection
70+
- Easy debugging: consistent test case
71+
72+
**Alternatives Considered**:
73+
- Multiple documents → rejected (unnecessary, schema is uniform)
74+
- Random selection → rejected (non-deterministic)
75+
- All test_data/ files → rejected (doesn't test live download)
76+
77+
### 3. Issue Creation Strategy
78+
**Decision**: Create one issue per failed workflow run with all failures grouped
79+
80+
**Rationale**:
81+
- Single notification: maintainer gets one alert, not spam
82+
- Context preservation: all related failures in one place
83+
- Easy tracking: one issue to close when fixed
84+
- Workflow correlation: issue links directly to failed run
85+
86+
**Alternatives Considered**:
87+
- One issue per failed test → rejected (too noisy, 5+ issues per run)
88+
- Update existing open issue → rejected (hard to track historical failures)
89+
- Slack/email notification → rejected (GitHub-native is simpler)
90+
91+
### 4. Validation Criteria
92+
**Decision**: Four focused validation points (HTML params, XML namespace, XML elements, download)
93+
94+
**Rationale**:
95+
- HTML parameter extraction: detects normattiva.it HTML structure changes
96+
- XML namespace: verifies Akoma Ntoso 3.0 compatibility
97+
- Essential XML elements: ensures required structure present
98+
- Download success: catches API/network issues
99+
100+
**What to validate**:
101+
```python
102+
1. HTML parameter extraction:
103+
- dataGU found in hidden input
104+
- codiceRedaz found in hidden input
105+
- dataVigenza found in hidden input (optional)
106+
- Values are non-empty
107+
108+
2. XML namespace:
109+
- Akoma Ntoso 3.0 namespace present
110+
- Namespace URL unchanged
111+
112+
3. Essential XML elements:
113+
- <akomaNtoso> root element exists
114+
- <meta> section exists
115+
- <body> section exists
116+
- At least one <article> exists
117+
118+
4. Download success:
119+
- HTTP 200 status
120+
- Response is parseable XML
121+
- Not HTML error page
122+
```
123+
124+
### 5. Workflow Schedule
125+
**Decision**: Weekly on Monday 09:00 UTC with manual trigger option
126+
127+
**Rationale**:
128+
- Weekly frequency: balances monitoring vs noise (not too frequent)
129+
- Monday morning: gives full week to address issues
130+
- UTC timezone: clear reference time
131+
- Manual trigger: allows on-demand validation before releases
132+
133+
**Alternatives Considered**:
134+
- Daily → rejected (too frequent, alert fatigue)
135+
- After each commit → rejected (not integration test purpose)
136+
- Monthly → rejected (too infrequent, delays problem detection)
137+
138+
## Error Handling
139+
140+
### Test Failures
141+
- Each test is independent, failures don't stop execution
142+
- All failures collected and reported together
143+
- Stack traces captured for debugging
144+
145+
### Network Failures
146+
- Retry mechanism for transient failures (3 attempts, 5s backoff)
147+
- Distinguish between network errors vs structural changes
148+
- Report network failures separately from conversion errors
149+
150+
### Issue Creation Failures
151+
- Workflow doesn't fail if issue creation fails
152+
- Log error to workflow output
153+
- Maintainer still sees workflow failure in GitHub UI
154+
155+
## Testing Strategy
156+
157+
### Local Testing
158+
```bash
159+
# Run all schema validation tests
160+
pytest tests/test_schema_validation.py -v
161+
162+
# Test HTML parameter extraction only
163+
pytest tests/test_schema_validation.py::TestSchemaValidation::test_html_parameter_extraction -v
164+
165+
# Test XML schema only
166+
pytest tests/test_schema_validation.py::TestSchemaValidation::test_xml_namespace_stable -v
167+
pytest tests/test_schema_validation.py::TestSchemaValidation::test_xml_essential_elements -v
168+
```
169+
170+
### Workflow Testing
171+
1. Manual trigger with success scenario
172+
2. Temporarily break HTML parsing, verify issue created
173+
3. Revert break, verify issue closure (if implemented)
174+
4. Wait for scheduled run (first Monday)
175+
176+
## Performance Considerations
177+
178+
- Target execution time: < 1 minute
179+
- Single document test
180+
- Minimal computation: parse HTML, validate XML structure
181+
- No Markdown conversion (saves time)
182+
- No heavy analysis
183+
184+
## Future Enhancements (Not in Scope)
185+
186+
1. **Multiple Sample Documents**: Test against 3-5 different document types
187+
2. **Historical Tracking**: Store validation results over time, trend analysis
188+
3. **Auto-Recovery**: Retry with different parameters if extraction fails
189+
4. **Full Conversion Testing**: Add optional full MD conversion validation
190+
5. **Performance Benchmarking**: Track download/parse speed over time

‎openspec/changes/add-conversion-validation-monitoring/proposal.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Proposal: add-conversion-validation-monitoring
2+
3+
## Summary
4+
Implement automated weekly testing to validate that Akoma Ntoso → Markdown conversion rules remain correct over time. Monitor for structural changes in normattiva.it data or conversion breakage, automatically opening issues when failures are detected.
5+
6+
## Background
7+
The conversion from Akoma Ntoso XML to Markdown relies on specific XML structures and normattiva.it API behavior. These can change without notice:
8+
- normattiva.it may alter XML schema or HTML structure
9+
- New Akoma Ntoso elements may appear in documents
10+
- Conversion logic may produce different output over time
11+
12+
Currently, there's no automated monitoring to detect when conversion rules break or need updates.
13+
14+
## Goals
15+
1. Verify HTML parameter extraction from normattiva.it pages still works (dataGU, codiceRedaz, dataVigenza)
16+
2. Verify XML Akoma Ntoso schema stability (namespace, base elements unchanged)
17+
3. Run validation weekly via GitHub Actions scheduled workflow
18+
4. Automatically open GitHub issues assigned to @aborruso when validation fails
19+
20+
## Non-Goals
21+
- Not testing full Markdown conversion (only data extraction and XML schema)
22+
- Not validating Markdown output quality or formatting
23+
- Not testing all possible legal documents (focus on single representative sample)
24+
- Not implementing auto-fix mechanisms (manual review required)
25+
26+
## Success Criteria
27+
- Weekly workflow runs successfully on schedule
28+
- Test failures produce actionable GitHub issues
29+
- Issues include specific failure details and affected documents
30+
- Workflow can be manually triggered for on-demand validation
31+
32+
## Open Questions
33+
- Should the workflow also test PyPI-installed version vs local code?
34+
- How many documents to test (balance coverage vs execution time)?
35+
- Should we cache successful conversion outputs for diff comparison?

‎openspec/changes/add-conversion-validation-monitoring/specs/conversion-validation/spec.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# conversion-validation Specification
2+
3+
## Purpose
4+
Provide automated monitoring and validation of Akoma Ntoso to Markdown conversion rules through scheduled testing and issue reporting.
5+
6+
## ADDED Requirements
7+
8+
### Requirement: Weekly Schema Validation
9+
The system SHALL run automated schema and extraction validation tests weekly to detect breaking changes in normattiva.it HTML structure or XML schema.
10+
11+
#### Scenario: Scheduled Weekly Execution
12+
- **WHEN** the workflow is scheduled to run (every Monday at 09:00 UTC)
13+
- **THEN** the system SHALL execute validation tests against a sample legal document URL
14+
- **AND** report results to GitHub Actions logs
15+
- **AND** create issues if validation fails
16+
17+
#### Scenario: Manual Workflow Trigger
18+
- **WHEN** a maintainer manually triggers the workflow via GitHub Actions UI
19+
- **THEN** the system SHALL execute the same validation tests on-demand
20+
- **AND** report results immediately
21+
22+
### Requirement: HTML Parameter Extraction Validation
23+
The system SHALL verify that parameter extraction from normattiva.it HTML pages continues to work correctly.
24+
25+
#### Scenario: HTML Form Parameter Extraction
26+
- **WHEN** running validation against a normattiva.it page URL
27+
- **THEN** the system SHALL download the HTML page
28+
- **AND** verify that hidden input fields for dataGU, codiceRedaz, dataVigenza are found
29+
- **AND** verify extracted values are non-empty and in expected format
30+
31+
#### Scenario: HTML Structure Change Detection
32+
- **WHEN** HTML parsing fails to extract required parameters
33+
- **THEN** the test SHALL fail with specific error indicating which parameter is missing
34+
- **AND** report that normattiva.it HTML structure may have changed
35+
36+
### Requirement: XML Schema Stability Validation
37+
The system SHALL verify that Akoma Ntoso XML schema remains stable and compatible with conversion logic.
38+
39+
#### Scenario: Namespace Validation
40+
- **WHEN** running validation against downloaded XML document
41+
- **THEN** the system SHALL verify Akoma Ntoso 3.0 namespace is present
42+
- **AND** verify namespace URL is unchanged: `http://docs.oasis-open.org/legaldocml/ns/akn/3.0`
43+
44+
#### Scenario: Essential Elements Validation
45+
- **WHEN** validating XML structure
46+
- **THEN** the system SHALL verify presence of essential elements:
47+
- `<akomaNtoso>` root element
48+
- `<meta>` metadata section
49+
- `<body>` document body
50+
- At least one `<article>` element
51+
- **AND** report which elements are missing if validation fails
52+
53+
### Requirement: Automated Issue Creation
54+
The system SHALL automatically create GitHub issues when validation tests fail, assigned to the project maintainer.
55+
56+
#### Scenario: Test Failure Issue Creation
57+
- **WHEN** one or more integration tests fail in the weekly workflow
58+
- **THEN** the system SHALL create a GitHub issue with:
59+
- Title: "Conversion validation failed - [date]"
60+
- Body containing summary of all failures
61+
- Labels: "bug", "automated"
62+
- Assignee: @aborruso
63+
64+
#### Scenario: Issue Content Detail
65+
- **WHEN** creating an automated issue
66+
- **THEN** the issue body SHALL include:
67+
- List of failed document URLs/identifiers
68+
- Specific error messages or validation failures
69+
- Link to failed workflow run
70+
- Comparison diff if output structure changed
71+
72+
#### Scenario: Success Without Issue
73+
- **WHEN** all validation tests pass
74+
- **THEN** no GitHub issue SHALL be created
75+
- **AND** workflow SHALL complete with success status
76+
77+
### Requirement: Download Stability Validation
78+
The system SHALL verify that XML document download from normattiva.it continues to work correctly.
79+
80+
#### Scenario: XML Download Success
81+
- **WHEN** testing document download from normattiva.it
82+
- **THEN** the system SHALL verify HTTP response is successful (200)
83+
- **AND** verify response content-type indicates XML
84+
- **AND** verify response body is valid XML (parseable)
85+
- **AND** report failure if download fails or returns HTML error page

0 commit comments

Comments
 (0)