Skip to content

Commit 33194ad

Browse files
reidspencerclaude
reidspencer
and
claude
committed
Consolidate session documentation into top-level NOTEBOOK.md
- Move bast/NOTEBOOK.md to top-level NOTEBOOK.md - Merge SESSION_HANDOFF.md content into NOTEBOOK.md - Update CLAUDE.md with BAST test counts (54 tests) - Add deprecation note about bast/jvm/src/test/ directory Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 4686f22 commit 33194ad

3 files changed

Lines changed: 198 additions & 138 deletions

File tree

‎CLAUDE.md‎

Lines changed: 52 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -58,20 +58,23 @@ utils → language → bast → passes → diagrams → commands → riddlc
5858
testkit
5959
```
6060

61-
### New Module: bast/
62-
**Purpose**: Binary AST (BAST) serialization for fast module imports
61+
### BAST Module (Binary AST)
62+
**Purpose**: Binary AST serialization for fast module imports
6363

64-
- **Location**: `bast/` (top-level directory)
65-
- **Dependencies**: `language`, `passes`
64+
- **Location**: `language/shared/src/main/scala/com/ossuminc/riddl/language/bast/` (inside language module)
65+
- **Package**: `com.ossuminc.riddl.language.bast`
6666
- **Cross-platform**: JVM, JS, Native
67-
- **Status**: In development (as of Jan 2026)
67+
- **Status**: Core functionality complete (as of Jan 2026)
6868

69-
**Key files**:
70-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/package.scala` - Constants and node type tags
71-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/BinaryFormat.scala` - Format specification
72-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/VarIntCodec.scala` - LEB128 varint encoding
73-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/ByteBufferWriter.scala` - Binary writer
74-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/ByteBufferReader.scala` - Binary reader
69+
**Note**: There is a legacy standalone `bast/` directory at the project root. This is deprecated - all authoritative BAST code lives in the `language` module's `bast` package.
70+
71+
**Key files** (all in `language/shared/src/main/scala/com/ossuminc/riddl/language/bast/`):
72+
- `package.scala` - Constants and node type tags (NODE_*, TYPE_*, STREAMLET_*, etc.)
73+
- `BASTWriter.scala` - Serialization pass (extends HierarchyPass)
74+
- `BASTReader.scala` - Deserialization
75+
- `BASTLoader.scala` - Import loading utility
76+
- `BASTUtils.scala` - Shared utilities
77+
- `StringTable.scala` - String interning for compression
7578

7679
## NPM Packaging (JavaScript/TypeScript API)
7780

@@ -162,9 +165,9 @@ class MyPass extends HierarchyPass {
162165
```
163166

164167
**BAST Writer Pattern**:
165-
- BASTWriter will be a Pass subclass
168+
- `BASTWriterPass` (in passes module) extends `HierarchyPass`
169+
- Uses `BASTWriter` utilities (in language module) for byte writing
166170
- Sacrifice write speed for read speed
167-
- Use `ByteBufferWriter` for output
168171
- String interning for deduplication
169172

170173
## GitHub Workflows
@@ -410,37 +413,44 @@ sbt riddlc/stage
410413
- Inside domains: for domain-specific imports
411414
- Not allowed elsewhere (contexts, entities, etc.)
412415

413-
3. **Writer as Pass** - BASTWriter extends `HierarchyPass`
416+
3. **Writer as Pass** - `BASTWriterPass` (in passes module) extends `HierarchyPass`
414417
- Idiomatic RIDDL architecture
415418
- Automatic traversal infrastructure
416-
- Minimal performance overhead
419+
- Uses `BASTWriter` utilities from language module for actual byte writing
417420

418421
4. **BASTImport as Container** - Resolution works naturally
419422
- ResolutionPass traverses Container.contents automatically
420423
- No special handling needed in resolution pass
421424

422425
#### Files Created
423426

424-
**Infrastructure (Phase 1)**:
425-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/package.scala` - Constants, node tags, flags
426-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/BinaryFormat.scala` - Format spec, Header
427-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/VarIntCodec.scala` - LEB128 encoding/decoding
428-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/ByteBufferWriter.scala` - Binary writer
429-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/ByteBufferReader.scala` - Binary reader
427+
All BAST source files are in `language/shared/src/main/scala/com/ossuminc/riddl/language/bast/`:
428+
429+
**Core** (in `language/shared/.../bast/`):
430+
- `package.scala` - Constants, node tags (NODE_*, TYPE_*), flags
431+
- `StringTable.scala` - String interning for compression
432+
- `BASTWriter.scala` - Writing utilities (not a Pass)
433+
- `BASTReader.scala` - Deserialization with `readNode()` and `readTypeExpression()`
434+
- `BASTLoader.scala` - Import loading utility
435+
- `BASTUtils.scala` - Shared utilities
430436

431-
**Serialization/Deserialization (Phases 2-3)**:
432-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/StringTable.scala` - String interning
433-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/BASTWriter.scala` - Serialization pass
434-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/BASTReader.scala` - Deserialization
437+
**Pass** (in `passes/shared/.../passes/`):
438+
- `BASTWriterPass.scala` - Serialization pass (extends HierarchyPass)
435439

436-
**Import Integration (Phase 4)**:
437-
- `bast/shared/src/main/scala/com/ossuminc/riddl/bast/BASTLoader.scala` - Import loading utility
440+
**Tests** (in `passes/jvm/src/test/scala/com/ossuminc/riddl/passes/`):
441+
- `BASTMinimalTest.scala` - Basic serialization test (1 test)
442+
- `BASTIncrementalTest.scala` - 37 incremental round-trip tests covering all AST structures
443+
- `BASTWriterSpec.scala` - Serialization tests (5 tests)
444+
- `BASTRoundTripTest.scala` - Round-trip tests (3 tests)
445+
- `BASTPerformanceBenchmark.scala` - Performance benchmarks comparing parse vs load (3 tests)
446+
- `BASTLoaderTest.scala` - Import integration tests (4 tests)
447+
- `BASTDebugTest.scala` - Byte-level debugging test (1 test)
448+
- `BASTBenchmarkRunner.scala` - Standalone benchmark runner (runMain)
449+
- `DeepASTComparison.scala` - AST comparison utility for round-trip verification
438450

439-
**Tests**:
440-
- `bast/jvm/src/test/scala/com/ossuminc/riddl/bast/BASTWriterSpec.scala` - Serialization tests
441-
- `bast/jvm/src/test/scala/com/ossuminc/riddl/bast/BASTRoundTripTest.scala` - Round-trip tests
442-
- `bast/jvm/src/test/scala/com/ossuminc/riddl/bast/BASTPerformanceTest.scala` - Performance tests
443-
- `bast/jvm/src/test/scala/com/ossuminc/riddl/bast/BASTLoaderTest.scala` - Import integration tests
451+
**Total: 54 BAST tests** (as of Jan 2026)
452+
453+
**DEPRECATED**: The `bast/jvm/src/test/` directory contains older test files that use the deprecated `BASTWriter.creator()` instead of `BASTWriterPass.creator()`. These should NOT be used - all authoritative tests are in the passes module.
444454

445455
#### Key Implementation Notes
446456

@@ -465,6 +475,14 @@ sbt riddlc/stage
465475
- Checksum validation on load
466476
- Graceful degradation for unknown node types
467477

478+
5. **CRITICAL: readNode() vs readTypeExpression() in BASTReader**:
479+
- `readNode()` handles **definition-level tags**: NODE_TYPE, NODE_DOMAIN, NODE_CONTEXT, NODE_ENTITY, NODE_FIELD, NODE_ENUMERATOR, etc.
480+
- `readTypeExpression()` handles **type expression tags**: TYPE_REF, TYPE_ALTERNATION, TYPE_AGGREGATION, TYPE_STRING, TYPE_NUMBER, etc.
481+
- **These are DISJOINT sets** - readNode() does NOT handle TYPE_* tags!
482+
- When reading contents that contain type expressions (e.g., `Alternation.of` which contains `AliasedTypeExpression`), you MUST use `readTypeExpression()`, not `readNode()` via `readContentsDeferred()`
483+
- **Bug pattern**: If you see "Invalid string table index" errors with huge counts like `metadata[1000009]`, it usually means the reader is misaligned because it tried to read a TYPE_* tag as a NODE_* tag
484+
- **Fix pattern**: Create specialized reader methods like `readTypeExpressionContents()` that read count + call `readTypeExpression()` for each item
485+
468486
#### Related Issues & TODOs
469487

470488
- **Issue #72**: Implement import functionality (currently stub)
@@ -658,3 +676,5 @@ Then add to root aggregation: `.aggregate(..., mymodule, mymoduleJS, mymoduleNat
658676
8. **RiddlAPI origin parameters must be URLs** - Use `originToURL()` helper for String origins
659677
9. **Scala 3 lambda syntax required** - `foreach(line => func(line))` not `foreach(func)`
660678
10. **Share code via utils/shared/** - For code used by both JVM and JS variants
679+
11. **BAST code lives in language module** - `language/shared/.../bast/`, NOT the standalone `bast/` directory
680+
12. **BAST readNode() vs readTypeExpression()** - Disjoint tag sets; see "Key Implementation Notes" for details on avoiding deserialization bugs

‎NOTEBOOK.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Engineering Notebook: RIDDL
2+
3+
This is the central engineering notebook for the RIDDL project. It tracks current status, work completed, design decisions, and next steps across all modules.
4+
5+
---
6+
7+
## Current Status
8+
9+
**Last Updated**: January 16, 2026
10+
11+
The RIDDL project is a mature compiler and toolchain for the Reactive Interface to Domain Definition Language. Recent work has focused on BAST (Binary AST) serialization for fast module imports.
12+
13+
---
14+
15+
## BAST Module (Binary AST Serialization)
16+
17+
### Status
18+
19+
**BAST serialization is fully integrated into the language and passes modules.**
20+
21+
The standalone `bast/` module has been **removed from build.sbt** and code reorganized:
22+
- **Utility code** → `language/shared/src/main/scala/com/ossuminc/riddl/language/bast/`
23+
- **BASTWriterPass** → `passes/shared/src/main/scala/com/ossuminc/riddl/passes/`
24+
25+
This enables BAST to work like Python's `.pyc` files - automatic loading from cache when available.
26+
27+
### Work Completed
28+
29+
- [x] **Phase 1**: Infrastructure - Module structure, format spec, varint codec, byte buffer reader/writer
30+
- [x] **Phase 2**: Core Serialization - BASTWriter pass, all node types, string interning
31+
- [x] **Phase 3**: Deserialization - BASTReader, round-trip verification, performance tests
32+
- [x] **Phase 4**: Import Integration (Jan 15, 2026)
33+
- Simplified syntax: `import "file.bast"` (removed `as namespace` clause)
34+
- Support imports at root level AND inside domains
35+
- BASTLoader utility to load and populate contents
36+
- Path resolution verified: `ImportedDomain.SomeType` resolves correctly
37+
- [x] **Phase 5**: Module Reorganization & Auto-Generation (Jan 15, 2026)
38+
- Removed standalone `bast/` module from build.sbt
39+
- Split BASTWriter: utility class in `language/bast/`, Pass wrapper in `passes/`
40+
- Added `BASTUtils.scala` with `checkForBastFile()`, `loadBAST()`, `tryLoadBastOrParseRiddl()`
41+
- Integrated automatic BAST loading into `TopLevelParser.parseURL()` and `parseInput()`
42+
- Added `--auto-generate-bast` / `-B` CLI option to riddlc
43+
- Implemented auto-generation in `Riddl.parse()` when option enabled
44+
- [x] **Phase 6**: Bug Fixes & Test Consolidation (Jan 16, 2026)
45+
- Fixed Alternation deserialization bug (`readTypeExpressionContents` helper)
46+
- Created BASTIncrementalTest with 37 test cases
47+
- Verified all BAST tests migrated to passes module (54 tests)
48+
- Documented deprecated `bast/jvm/src/test/` files
49+
50+
### Key Technical Insight
51+
52+
**CRITICAL: `readNode()` vs `readTypeExpression()` in BASTReader**:
53+
- `readNode()` handles **definition-level tags**: NODE_TYPE, NODE_DOMAIN, NODE_CONTEXT, etc.
54+
- `readTypeExpression()` handles **type expression tags**: TYPE_REF, TYPE_ALTERNATION, etc.
55+
- **These are DISJOINT sets** - readNode() does NOT handle TYPE_* tags!
56+
- When reading contents containing type expressions (e.g., `Alternation.of`), use `readTypeExpression()`, not `readNode()`
57+
- **Bug pattern**: "Invalid string table index" with huge counts usually means byte stream misalignment from reading TYPE_* tag as NODE_* tag
58+
59+
### Test Status (54 tests, all passing)
60+
61+
| Test Suite | Tests |
62+
|------------|-------|
63+
| BASTMinimalTest | 1 |
64+
| BASTIncrementalTest | 37 |
65+
| BASTWriterSpec | 5 |
66+
| BASTRoundTripTest | 3 |
67+
| BASTPerformanceBenchmark | 3 |
68+
| BASTLoaderTest | 4 |
69+
| BASTDebugTest | 1 |
70+
71+
**Note**: All tests are in `passes/jvm/src/test/`. The `bast/jvm/src/test/` directory contains deprecated duplicates using old API.
72+
73+
### Key Code Locations
74+
75+
**Language Module** (`language/shared/.../language/bast/`):
76+
- `package.scala` - Constants, node type tags (NODE_*, TYPE_*)
77+
- `BASTWriter.scala` - Serialization utilities
78+
- `BASTReader.scala` - Deserialization with `readNode()` and `readTypeExpression()`
79+
- `BASTLoader.scala` - Import loading utility
80+
- `BASTUtils.scala` - File checking, BAST loading helpers
81+
82+
**Passes Module** (`passes/shared/.../passes/`):
83+
- `BASTWriterPass.scala` - Pass wrapper using AST traversal framework
84+
85+
**Commands Module** (`commands/jvm/.../commands/`):
86+
- `BastGenCommand.scala` - `riddlc bast-gen` command
87+
88+
### Next Steps
89+
90+
1. Performance benchmarking (parse RIDDL vs load BAST)
91+
2. Cross-platform testing (JS, Native variants)
92+
3. Clean up deprecated `bast/jvm/src/test/` directory
93+
94+
### Open Questions
95+
96+
- How should BAST versioning handle breaking format changes?
97+
98+
---
99+
100+
## Design Decisions Log
101+
102+
| Decision | Rationale | Alternatives | Date |
103+
|----------|-----------|--------------|------|
104+
| No namespace syntax for imports | RIDDL uses nested domains for namespacing | `import "x.bast" as ns` | 2026-01-15 |
105+
| Imports at root + domain only | Simplest useful locations | Root only, All containers | 2026-01-15 |
106+
| BASTImport as Container | Resolution pass naturally traverses contents | Special handling | 2026-01-15 |
107+
| Custom binary format | Memory-mappable, ~10x faster than parsing | FlatBuffers, Protobuf | 2026-01-10 |
108+
| String interning | Deduplicates common strings | No interning | 2026-01-10 |
109+
| Delta-encoded locations | ~70% space savings | Full coordinates | 2026-01-10 |
110+
111+
---
112+
113+
## Session Log
114+
115+
### January 16, 2026 (Continuation)
116+
117+
**Focus**: Test migration verification
118+
119+
**Completed**:
120+
- Confirmed all BAST test files already exist in `passes/jvm/src/test/` with correct imports
121+
- Files in `bast/jvm/src/test/` are deprecated duplicates using old `BASTWriter.creator()` API
122+
- Authoritative tests use `BASTWriterPass.creator()` from the passes module
123+
- All 54 BAST tests pass, all 244 passes module tests pass
124+
125+
### January 16, 2026 (Earlier)
126+
127+
**Focus**: Debugging "Invalid string table index" deserialization errors
128+
129+
**Completed**:
130+
- Created BASTIncrementalTest with 37 test cases building from simple to complex structures
131+
- Identified root cause: `Alternation` types caused byte stream misalignment
132+
- Fixed by adding `readTypeExpressionContents()` helper method to BASTReader
133+
- Updated `TYPE_ALTERNATION` case to use new helper instead of `readContentsDeferred()`
134+
135+
**Root Cause Analysis**:
136+
- Writer serializes `AliasedTypeExpression` items using `TYPE_REF` tags
137+
- Reader's `readContentsDeferred()` called `readNode()`
138+
- `readNode()` only handles NODE_* tags, not TYPE_* tags
139+
- This caused byte misinterpretation leading to invalid string table indices
140+
141+
---
142+
143+
## Git Information
144+
145+
**Branch**: `development`
146+
**Main branch**: `main`

‎bast/NOTEBOOK.md‎

Lines changed: 0 additions & 106 deletions
This file was deleted.

0 commit comments

Comments
 (0)