Skip to content

Commit dc1efab

Browse files
pwt-cdclaude
andauthored
Clarify LCFS credit integration with biomass tracking (#263)
* DOCUMENTATION: Clarify LCFS credit integration with biomass tracking Enhanced regulatory compliance documentation to demonstrate how BOOST biomass tracking provides essential data for LCFS credit calculations. Key improvements: - Added comprehensive tracking integration narrative to regulatory-compliance.md - Included Pacific Renewable Fuels real-world example (4.2M records → 54.58M credits) - Enhanced use-cases.inc.md with concrete workflow demonstration - Updated documentation with prose style over bullet points - Fixed version placeholders across specification files This directly addresses @colinmccormick concern that "tracking biomass supply doesn't seem to enter into this at all" by showing concrete data flow from BOOST tracking systems into LCFS credit generation. Closes #246 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * SCHEMA: Comprehensive integrity improvements and Python model synchronization Enhanced BOOST schema integrity with normalization fixes, Python model synchronization, and comprehensive validation coverage analysis. Schema Improvements: - Updated TraceableUnit Python model with 4 missing fields: * productClassification (enum with 7 values for market classification) * physicalArrangement (object for spatial organization and LCA analysis) * alternativeFateMetrics (object for BECCS analysis and carbon impact) * identificationMethodId (FK to IdentificationMethod entity) Normalization Fixes: - Removed redundant arrays from Organization schema (equipmentIds, operatorIds, harvestSites, traceableUnitIds) - Removed redundant arrays from SupplyBase schema (skidRoads, forestRoads, equipmentDeployment, traceableUnitIds) - Added normalization documentation explaining FK-based relationship management - Follows 3NF principles for improved data integrity and maintainability Validation Coverage Analysis: - Added comprehensive FK analysis tool (fk_analysis.py) - Generated integrity analysis reports identifying 111 FK relationships - Created normalization issues analysis and recommendations - Generated missing Python models for all 36 entities Documentation Updates: - Regenerated all entity tables with schema improvements - Updated HTML and PDF documentation (96 pages, 1.1MB HTML) - Synchronized documentation across all formats - Maintained single source of truth from JSON schemas Technical Benefits: - Eliminates data redundancy and maintenance overhead - Ensures referential integrity through proper FK relationships - Improves query performance and reduces storage requirements - Provides comprehensive validation framework for 36 entities All changes validated through automated build system and integrity analysis tools. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * CLEANUP: Remove temporary analysis markdown files The NORMALIZATION_ISSUES_ANALYSIS.md and SCHEMA_INTEGRITY_ANALYSIS_REPORT.md files were temporary analysis artifacts that served their purpose. The actual improvements have been implemented in the schemas and Python models. Keeping the useful tools: - fk_analysis.py (foreign key analysis tool) - generated_models.py (Python models for all entities) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * CLEANUP: Remove static generated_models.py file The BOOST Python implementation uses dynamic schema-driven model generation, not static model files. Pydantic models should be generated at runtime from JSON schemas to maintain the dynamic, schema-driven architecture. This aligns with the design principles documented in README.md: - Dynamic Schema-Driven Architecture - Pydantic models generated directly from JSON schemas - Automatically adapts to schema changes without code modifications The generate_missing_models.py script was helpful for analysis but the static output file contradicts the dynamic design approach. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * REFACTOR: Complete migration to dynamic schema-driven architecture Eliminated static Pydantic models in favor of true dynamic model generation from JSON schemas, delivering on the README's promise of "Dynamic Schema-Driven Architecture" where models are generated directly from JSON schemas at runtime. Major Changes: - **Removed static model files**: * models.py (853 lines) - static models for 8 entities * models_backup.py, models_fixed.py - outdated artifacts * generate_missing_models.py - static generation tool - **Added dynamic_models.py**: Clean facade for dynamic model access * Runtime generation of all 36 BOOST entity models from schemas * Backward compatibility for existing enum imports * Dynamic validation against current schema definitions * Complete entity coverage (was 8, now all 36 entities) - **Updated all dependencies**: * test_bioram_validation.py - uses dynamic models * bioram_validation.py - removed static model imports * examples/*.py - use dynamic enums with backward compatibility Technical Benefits: - **Always current**: Models automatically reflect schema changes - **No synchronization issues**: Single source of truth from JSON schemas - **Complete coverage**: All 36 entities available, not just 8 hand-coded ones - **Reduced codebase**: Eliminated 850+ lines of redundant static code - **True dynamic architecture**: Runtime model generation as promised Documentation: - **MIGRATION_GUIDE.md**: Comprehensive migration instructions - **README.md**: Updated to highlight full dynamic architecture delivery - **Backward compatibility**: Existing code continues to work Testing confirmed all dynamic models work correctly with full enum support and validation against current schema definitions. This delivers the true "Dynamic Schema-Driven Architecture" promised in the README where Pydantic models are generated directly from JSON schemas. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * FIX: Update GitHub Actions to use dynamic models instead of static models.py Fixed Python validation workflow that was failing because it still expected the old static models.py file that was removed in the dynamic architecture migration. Changes: - **Validate Python model syntax**: Now compiles dynamic_models.py, schema_loader.py, and boost_client.py instead of models.py - **Test Pydantic model imports**: Replaced with dynamic model loading test using get_models() - **Test model validation**: Updated to use dynamic model classes with proper required fields The workflow now correctly validates: ✅ Dynamic model loading from JSON schemas ✅ All 36 entities accessible dynamically (not just 8 static ones) ✅ Enum value loading from schemas ✅ Model validation with complete required field sets This ensures CI/CD validates the actual dynamic architecture rather than the old static models approach that was removed. Testing confirmed all workflow steps pass locally with the new dynamic models. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>
1 parent a91c790 commit dc1efab

File tree

63 files changed

+2109
-2771
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+2109
-2771
lines changed

.github/workflows/schema-validation.yml

Lines changed: 68 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -404,76 +404,104 @@ jobs:
404404
- name: Validate Python model syntax
405405
working-directory: drafts/current/reference-implementations/python
406406
run: |
407-
echo "🐍 Validating Python model syntax..."
408-
python3 -m py_compile models.py
407+
echo "🐍 Validating Python dynamic model syntax..."
408+
python3 -m py_compile dynamic_models.py
409+
python3 -m py_compile schema_loader.py
410+
python3 -m py_compile boost_client.py
409411
echo "✅ Python syntax validation passed"
410412
411-
- name: Test Pydantic model imports
413+
- name: Test dynamic model loading
412414
working-directory: drafts/current/reference-implementations/python
413415
run: |
414-
echo "🔍 Testing Pydantic model imports..."
416+
echo "🔍 Testing dynamic model loading..."
415417
python3 -c "
416418
import sys
417-
from models import *
418-
419-
# Test that key models can be imported
420-
required_models = [
421-
'Organization',
422-
'TraceableUnit',
423-
'Transaction',
424-
'MaterialProcessing',
425-
'Claim'
419+
from dynamic_models import get_models
420+
421+
# Test dynamic model loading
422+
models = get_models()
423+
print('✅ Dynamic models loaded successfully')
424+
425+
# Test that key entities can be loaded dynamically
426+
required_entities = [
427+
'organization',
428+
'traceable_unit',
429+
'transaction',
430+
'material_processing'
426431
]
427432
428-
for model_name in required_models:
429-
if model_name not in globals():
430-
print(f'❌ Model {model_name} not found')
433+
for entity_name in required_entities:
434+
try:
435+
model_class = models.get_model(entity_name)
436+
print(f'✅ {entity_name} model loaded successfully')
437+
except Exception as e:
438+
print(f'❌ Failed to load {entity_name}: {e}')
431439
sys.exit(1)
432-
else:
433-
print(f'✅ {model_name} imported successfully')
434440
435-
print('✅ All required Pydantic models imported successfully')
441+
# Test enum loading
442+
org_types = models.get_enum_values('organization', 'organizationType')
443+
if org_types:
444+
print(f'✅ Organization types loaded: {org_types[:3]}...')
445+
else:
446+
print('❌ Failed to load organization types')
447+
sys.exit(1)
448+
449+
print('✅ All dynamic models and enums loaded successfully')
436450
"
437451
438452
- name: Test model validation
439453
working-directory: drafts/current/reference-implementations/python
440454
run: |
441-
echo "🧪 Testing model validation..."
455+
echo "🧪 Testing dynamic model validation..."
442456
python3 -c "
443457
import sys
444-
from models import Organization, TraceableUnit
458+
from dynamic_models import get_models
459+
460+
models = get_models()
445461
446-
# Test Organization model
462+
# Test Organization model validation
447463
try:
448-
org = Organization(
449-
context={},
450-
type='Organization',
451-
id='https://example.com/org/1',
452-
organization_id='ORG-TEST-001',
453-
organization_name='Test Organization',
454-
organization_type='harvester'
455-
)
464+
Organization = models.get_model('organization')
465+
org_data = {
466+
'@context': {},
467+
'@type': 'Organization',
468+
'@id': 'https://example.com/org/1',
469+
'organizationId': 'ORG-TEST-001',
470+
'organizationName': 'Test Organization',
471+
'organizationType': 'harvester'
472+
}
473+
org = Organization(**org_data)
456474
print('✅ Organization model validation passed')
457475
except Exception as e:
458476
print(f'❌ Organization model validation failed: {e}')
459477
sys.exit(1)
460478
461-
# Test TraceableUnit model
479+
# Test TraceableUnit model validation
462480
try:
463-
tru = TraceableUnit(
464-
context={},
465-
type='TraceableUnit',
466-
id='https://example.com/tru/1',
467-
traceable_unit_id='TRU-TEST-001',
468-
unit_type='log',
469-
harvester_id='ORG-HARVEST-001'
470-
)
481+
TraceableUnit = models.get_model('traceable_unit')
482+
tru_data = {
483+
'@context': {},
484+
'@type': 'TraceableUnit',
485+
'@id': 'https://example.com/tru/1',
486+
'traceableUnitId': 'TRU-TEST-001',
487+
'unitType': 'individual_log',
488+
'harvesterId': 'ORG-HARVEST-001',
489+
'uniqueIdentifier': 'RFID-TEST-001',
490+
'identificationMethodId': 'IM-RFID-001',
491+
'identificationConfidence': 0.95,
492+
'totalVolumeM3': 2.5,
493+
'harvestGeographicDataId': 'GEO-HARVEST-001',
494+
'createdTimestamp': '2025-08-22T12:00:00Z',
495+
'materialTypeId': 'MAT-PINE-001',
496+
'isMultiSpecies': False
497+
}
498+
tru = TraceableUnit(**tru_data)
471499
print('✅ TraceableUnit model validation passed')
472500
except Exception as e:
473501
print(f'❌ TraceableUnit model validation failed: {e}')
474502
sys.exit(1)
475503
476-
print('✅ Model validation tests passed')
504+
print('✅ Dynamic model validation tests passed')
477505
"
478506
479507
validation-summary:
Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# BOOST Python Migration Guide: Static to Dynamic Models
2+
3+
This guide documents the migration from static Pydantic models to the dynamic, schema-driven architecture as promised in the README.
4+
5+
## What Changed
6+
7+
### Before: Static Models (Removed)
8+
```python
9+
from models import Organization, TraceableUnit, OrganizationType, UnitType
10+
```
11+
12+
### After: Dynamic Models
13+
```python
14+
from dynamic_models import get_models, OrganizationType, UnitType
15+
16+
# Get models dynamically from schemas
17+
models = get_models()
18+
Organization = models.get_model('organization')
19+
TraceableUnit = models.get_model('traceable_unit')
20+
```
21+
22+
## Migration Steps
23+
24+
### 1. Update Imports
25+
26+
**Old approach:**
27+
```python
28+
from models import Organization, TraceableUnit, Transaction
29+
from models import OrganizationType, UnitType, ProcessType, ClaimType
30+
```
31+
32+
**New approach:**
33+
```python
34+
from dynamic_models import get_models, OrganizationType, UnitType, ProcessType
35+
# For backward compatibility, enums are still available as classes
36+
```
37+
38+
### 2. Get Model Classes Dynamically
39+
40+
**Old approach:**
41+
```python
42+
org = Organization(
43+
organizationId="ORG-001",
44+
organizationName="Forest Co",
45+
organizationType=OrganizationType.HARVESTER
46+
)
47+
```
48+
49+
**New approach:**
50+
```python
51+
models = get_models()
52+
Organization = models.get_model('organization')
53+
54+
org = Organization(
55+
organizationId="ORG-001",
56+
organizationName="Forest Co",
57+
organizationType=OrganizationType.HARVESTER # Still works!
58+
)
59+
```
60+
61+
### 3. Use String Values Directly (Recommended)
62+
63+
**Even better approach:**
64+
```python
65+
models = get_models()
66+
Organization = models.get_model('organization')
67+
68+
# Use strings directly - always valid with current schema
69+
org = Organization(
70+
organizationId="ORG-001",
71+
organizationName="Forest Co",
72+
organizationType="harvester" # String values from schema
73+
)
74+
```
75+
76+
### 4. Validate Enum Values
77+
78+
```python
79+
models = get_models()
80+
81+
# Get valid enum values for validation
82+
org_types = models.get_enum_values('organization', 'organizationType')
83+
print(f"Valid org types: {org_types}") # ['harvester', 'processor', ...]
84+
85+
# Validate before creating
86+
if org_type in org_types:
87+
org = models.get_model('organization')(organizationType=org_type, ...)
88+
```
89+
90+
## Benefits of Dynamic Models
91+
92+
### 1. Always Current
93+
Models are generated from JSON schemas at runtime, so they always reflect the latest schema definitions.
94+
95+
### 2. No Synchronization Issues
96+
No more mismatches between static Python models and JSON schemas.
97+
98+
### 3. Complete Coverage
99+
All 36 BOOST entities are available dynamically, not just the 8 that were hand-coded.
100+
101+
### 4. Easier Maintenance
102+
Schema changes automatically propagate to Python models without code updates.
103+
104+
## Backward Compatibility
105+
106+
The new `dynamic_models.py` provides backward compatibility:
107+
108+
```python
109+
# These still work for existing code:
110+
from dynamic_models import OrganizationType, UnitType, ProcessType
111+
112+
org_type = OrganizationType.HARVESTER # Returns "harvester"
113+
unit_type = UnitType.INDIVIDUAL_LOG # Returns "individual_log"
114+
```
115+
116+
## Available Entities
117+
118+
The dynamic system provides access to all 36 BOOST entities:
119+
120+
```python
121+
models = get_models()
122+
123+
# Core entities
124+
Organization = models.get_model('organization')
125+
TraceableUnit = models.get_model('traceable_unit')
126+
Transaction = models.get_model('transaction')
127+
MaterialProcessing = models.get_model('material_processing')
128+
129+
# All other entities are now available too:
130+
Certificate = models.get_model('certificate')
131+
Equipment = models.get_model('equipment')
132+
GeographicData = models.get_model('geographic_data')
133+
# ... and 29 more
134+
```
135+
136+
## Best Practices
137+
138+
### 1. Use String Values
139+
Prefer string enum values over enum classes for better schema alignment:
140+
141+
```python
142+
# Good
143+
org = Organization(organizationType="harvester")
144+
145+
# Still works, but string is better
146+
org = Organization(organizationType=OrganizationType.HARVESTER)
147+
```
148+
149+
### 2. Validate Dynamically
150+
Use schema-based validation instead of static type checking:
151+
152+
```python
153+
models = get_models()
154+
155+
# Validate enum values dynamically
156+
valid_types = models.get_enum_values('organization', 'organizationType')
157+
if org_type not in valid_types:
158+
raise ValueError(f"Invalid org type. Valid types: {valid_types}")
159+
```
160+
161+
### 3. Cache Model Classes
162+
If creating many instances, cache the model class:
163+
164+
```python
165+
models = get_models()
166+
Organization = models.get_model('organization') # Cache this
167+
168+
# Create many instances
169+
orgs = [Organization(...) for _ in range(100)]
170+
```
171+
172+
## Testing Updates
173+
174+
Test files now import from `dynamic_models` instead of `models`:
175+
176+
```python
177+
# test_bioram_validation.py
178+
from dynamic_models import get_models
179+
180+
def test_validation():
181+
models = get_models()
182+
BioramPathway = models.get_model('bioram_pathway')
183+
pathway = BioramPathway(...)
184+
```
185+
186+
## Schema Path Configuration
187+
188+
By default, dynamic models load schemas from `../../schema` relative to the Python files. You can specify a different path:
189+
190+
```python
191+
from dynamic_models import get_models
192+
193+
# Use custom schema path
194+
models = get_models(schema_path="/path/to/boost/schemas")
195+
```
196+
197+
## Troubleshooting
198+
199+
### Entity Not Found
200+
```python
201+
# If you get "Entity 'foo' not found"
202+
models = get_models()
203+
available = models.list_entities()
204+
print(f"Available entities: {available}")
205+
```
206+
207+
### Enum Not Found
208+
```python
209+
# If you get "Enum 'entity.field' not found"
210+
models = get_models()
211+
available_enums = models.list_enums()
212+
print(f"Available enums: {available_enums}")
213+
```
214+
215+
### Schema Loading Errors
216+
Make sure the schema path is correct and contains valid JSON schema files:
217+
218+
```python
219+
from pathlib import Path
220+
221+
schema_path = Path("../../schema").resolve()
222+
print(f"Schema path exists: {schema_path.exists()}")
223+
print(f"Schema contents: {list(schema_path.iterdir())}")
224+
```
225+
226+
This migration ensures the BOOST Python implementation truly delivers on its promise of being a "Dynamic Schema-Driven Architecture."

drafts/current/reference-implementations/python/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,17 @@ This reference implementation demonstrates how to use the BOOST standard in Pyth
1616
- **🌐 JSON-LD Export/Import**: Full semantic web compatibility
1717
- **🛡️ Schema Version Compatibility**: Graceful handling of schema evolution
1818

19+
## Recent Updates: Full Dynamic Architecture
20+
21+
**🎉 The Python implementation now fully delivers on its promise of "Dynamic Schema-Driven Architecture"!**
22+
23+
- **✅ All models generated dynamically** from JSON schemas at runtime
24+
- **✅ No more static model files** to maintain or synchronize
25+
- **✅ Always current** - models automatically reflect schema changes
26+
- **✅ Complete entity coverage** - all 36 BOOST entities available dynamically
27+
28+
See [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md) for details on migrating from the old static models approach.
29+
1930
## Installation
2031

2132
### Prerequisites

0 commit comments

Comments
 (0)