From d3d58caf32da58b2120a2ecf0228e24d7f9645d6 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:06:24 +0000 Subject: [PATCH 01/12] Add plan 163: schema-driven data extraction (mdsmith extract) Introduces a `bind:` projection layer on schema scopes and content entries plus an `extract` subcommand that turns a kind-conformant Markdown file into JSON/YAML/Lua/msgpack. Sequenced on top of the schema-composition rework (plan 156) and content-schema work (plan 149). https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- PLAN.md | 1 + plan/163_schema-driven-data-extraction.md | 174 ++++++++++++++++++++++ 2 files changed, 175 insertions(+) create mode 100644 plan/163_schema-driven-data-extraction.md diff --git a/PLAN.md b/PLAN.md index 10ae82a12..1125a8204 100644 --- a/PLAN.md +++ b/PLAN.md @@ -90,5 +90,6 @@ footer: | | 161 | πŸ”³ | sonnet | [Expose rule maintainability patterns via CLI help and LSP](plan/161_rule-pattern-metadata.md) | | 162 | πŸ”² | sonnet | [Split the overloaded `meta` rule category](plan/162_rule-category-cleanup.md) | | 163 | πŸ”² | | [Extract mdsmith Markdown parse/produce as a public Go library](plan/163_public-markdown-library.md) | +| 163 | πŸ”² | opus | [Schema-driven data extraction (mdsmith extract)](plan/163_schema-driven-data-extraction.md) | | 164 | βœ… | | [GitHub-UI-triggered releases and a split website deploy](plan/164_github-ui-releases-and-split-website.md) | diff --git a/plan/163_schema-driven-data-extraction.md b/plan/163_schema-driven-data-extraction.md new file mode 100644 index 000000000..52b80b496 --- /dev/null +++ b/plan/163_schema-driven-data-extraction.md @@ -0,0 +1,174 @@ +--- +id: 163 +title: Schema-driven data extraction (mdsmith extract) +status: "πŸ”²" +model: opus +depends-on: [149, 156] +summary: >- + Add a bind layer to schemas and an `extract` + subcommand that turns a kind-conformant Markdown + file into JSON/YAML/Lua/msgpack following the + schema. +--- +# Schema-driven data extraction (mdsmith extract) + +## Goal + +Let a kind's schema double as an extraction contract. +Once `mdsmith check` confirms a file conforms, `mdsmith +extract --format json|yaml|lua|msgpack ` +emits a data tree. Its shape is exactly what the schema +declares. + +## Why a new concept is needed + +Today the schema engine +([internal/schema](../internal/schema)) validates +*structure*: the heading tree, repeat cardinality, +content-node kinds, and front-matter CUE constraints. +But its [Scope](../internal/schema/schema.go) and +`ContentEntry` nodes are **anonymous**. The matcher knows +a `## {id}` heading must exist, yet not what key `id` +becomes in output, and it never captures node bodies. +Front matter is already YAML-typed and passes through +unchanged. The gap is the document body. + +So this plan adds exactly one new schema concept β€” a +**binding / projection layer** β€” plus a thin extractor +that walks the existing validated match. No second +parser, and no schema-to-type inference. + +## Sequencing + +The schema engine is mid-rework. This plan must land +after, and consume the outputs of, that work rather than +the legacy single-source model. + +- **Plan 156 / PR #288 (schema composition).** A file can + resolve to multiple kinds whose schemas compose via + `schema.Compose()`. The extractor consumes the composed + `Schema`, never a single `Rule.Schema`. This forces a + new rule: `bind:` names must compose too. Identical + headings from different kinds merge their bound + children. Two kinds binding the same node to different + names is a schema error raised at compose time. +- **Plan 149 (section-content schema).** Body extraction + rides on the `ContentEntry` model from the + content-schema work. This plan adds no content matcher + of its own and is blocked until that model is stable. +- **Plan 147 / PR #284 (actionable schema diagnostics).** + If landed, bind validation and conformance failures + reuse the `SchemaDiagnostic` formatter. + +## The `bind:` concept + +`bind:` is an optional string key on any schema scope or +content entry. It names the value that node contributes +to the extracted tree. Unbound nodes are structural only +and emit nothing, keeping output intentional. + +Mapping rules: + +- **Front matter** β†’ top-level `frontmatter` object, + passed through from the existing decode. +- **Non-repeating scope, `bind: x`** β†’ object `x` holding + its bound children. +- **Repeating scope, `bind: xs`** (a `{placeholder}` + heading) β†’ array `xs`. Each element is an object whose + fields are the captured placeholders plus bound + children. +- **`code-block`, `bind: c`** β†’ string `c` (raw body); + optional `parse: yaml|json` embeds the decoded value. +- **`list`, `bind: items`** β†’ array of item strings. +- **`table` with `columns`, `bind: rows`** β†’ array of row + objects keyed by column header. +- **`paragraph`, `bind: t`** β†’ its text. + +Extraction is gated on a successful schema match. A +non-conformant file makes `extract` report the same +diagnostics as `check` and exit non-zero. It never emits +partial data. + +## Tasks + +1. **Parse `bind:` (red/green).** Add a `Bind string` + field to `Scope` and `ContentEntry` in + [schema.go](../internal/schema/schema.go); parse it in + [parse_inline.go](../internal/schema/parse_inline.go) + and [parse_file.go](../internal/schema/parse_file.go). + Unit-test round-trip and that an empty bind means + "structural only". +2. **Validate `bind:` names.** Reject duplicate sibling + binds, and binds whose value is unreachable because a + parent is unbound. Surface via the `SchemaDiagnostic` + path (plan 147) if landed, else a plain parse error. +3. **Compose `bind:` across kinds.** Extend + `schema.Compose()` (plan 156) so merged headings union + their bound children. Binding one composed node to two + different names is a compose-time error. Add + `compose_test.go` cases for union and conflict. +4. **Extractor package.** Add `internal/extract` with + `Extract(f *lint.File, sch *schema.Schema, match …) + (any, []lint.Diagnostic)`. `sch` is the composed + schema. It consumes the existing schema-validation + walk (extend `schema.Validate` / the content matcher + to expose the scopeβ†’nodes match tree) rather than + re-matching. +5. **Capture placeholder values.** When a repeating scope + matches a `{field}` heading pattern, record each + captured field into the element object, reusing + [fieldinterp](../internal/fieldinterp/fieldinterp.go). +6. **Format encoders.** Add `internal/extract/encode` + with json (stdlib), yaml (existing dep), msgpack, and + lua (table literal) encoders behind a `Format` enum. +7. **`extract` subcommand.** Register `extract` in + [main.go](../cmd/mdsmith/main.go); signature `mdsmith + extract --format `. Reuse the + config-load and kind-resolution helpers from + [kinds.go](../cmd/mdsmith/kinds.go). Validate that + `` is one of the file's resolved kinds. Run + schema validation first and abort on failure. +8. **Fixtures and integration test.** Add a kind with a + bound schema under `testdata/`, a conformant sample, + and golden outputs per format. Assert non-conformant + input exits non-zero with check diagnostics. +9. **Docs.** Add a section under + [schemas.md](../docs/guides/schemas.md) and a + `docs/reference/cli/extract.md` page. Both are picked + up by existing catalog directives. Run `mdsmith fix` + so catalogs and PLAN.md regenerate. + +## Acceptance Criteria + +- [ ] `bind:` parses on scopes and content entries from + inline and `proto.md` schemas. Duplicate and + unreachable binds are rejected with actionable + diagnostics. +- [ ] `mdsmith extract --format json ` on a + conformant file emits a tree matching the bound + schema, with front matter under `frontmatter`. +- [ ] A file resolving to multiple kinds composes its + binds; conflicting binds on one composed node are a + compose-time error. +- [ ] Repeating sections become arrays. Placeholder + captures and bound children appear as element + fields. Code-block, list, table, and paragraph + binds extract as specified. +- [ ] `yaml`, `lua`, and `msgpack` produce equivalent + data; golden fixtures cover all four formats. +- [ ] A non-conformant file makes `extract` exit non-zero + and print the same diagnostics as `mdsmith check`. +- [ ] An unknown kind, or a kind not assigned to the + file, exits non-zero with a clear message. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes + +## Open questions + +- Should `bind:` default to the slugified heading text, + or stay strictly opt-in? The plan assumes opt-in. +- Lua output: bare `return { … }` table to start, not a + named module. +- Exposing extraction over the LSP or a `query`-style + selector is out of scope here. From f19b69c85a7512b7db63b26bb81a73004f408eaf Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:13:10 +0000 Subject: [PATCH 02/12] Rework plan 163: default binding layer; split custom bindings to plan 164 - Default projection derives the data tree from the schema hierarchy; no annotations required. - Custom bindings move to follow-up plan 164, layered on a single keyFor() seam. - Address Copilot review: disambiguate the duplicate plan-156 id via explicit file link; name the concrete schema.MatchTree type returned by an extended schema.Validate. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- PLAN.md | 1 + plan/163_schema-driven-data-extraction.md | 235 ++++++++++++---------- plan/164_custom-binding-overrides.md | 64 ++++++ 3 files changed, 190 insertions(+), 110 deletions(-) create mode 100644 plan/164_custom-binding-overrides.md diff --git a/PLAN.md b/PLAN.md index 1125a8204..f9fae8e40 100644 --- a/PLAN.md +++ b/PLAN.md @@ -91,5 +91,6 @@ footer: | | 162 | πŸ”² | sonnet | [Split the overloaded `meta` rule category](plan/162_rule-category-cleanup.md) | | 163 | πŸ”² | | [Extract mdsmith Markdown parse/produce as a public Go library](plan/163_public-markdown-library.md) | | 163 | πŸ”² | opus | [Schema-driven data extraction (mdsmith extract)](plan/163_schema-driven-data-extraction.md) | +| 164 | πŸ”² | opus | [Custom binding overrides for mdsmith extract](plan/164_custom-binding-overrides.md) | | 164 | βœ… | | [GitHub-UI-triggered releases and a split website deploy](plan/164_github-ui-releases-and-split-website.md) | diff --git a/plan/163_schema-driven-data-extraction.md b/plan/163_schema-driven-data-extraction.md index 52b80b496..fdf0fb0e5 100644 --- a/plan/163_schema-driven-data-extraction.md +++ b/plan/163_schema-driven-data-extraction.md @@ -5,10 +5,9 @@ status: "πŸ”²" model: opus depends-on: [149, 156] summary: >- - Add a bind layer to schemas and an `extract` - subcommand that turns a kind-conformant Markdown - file into JSON/YAML/Lua/msgpack following the - schema. + Derive a default data tree from the hierarchical + schema and add an `extract` subcommand that emits a + kind-conformant file as JSON/YAML/Lua/msgpack. --- # Schema-driven data extraction (mdsmith extract) @@ -17,72 +16,81 @@ summary: >- Let a kind's schema double as an extraction contract. Once `mdsmith check` confirms a file conforms, `mdsmith extract --format json|yaml|lua|msgpack ` -emits a data tree. Its shape is exactly what the schema -declares. - -## Why a new concept is needed - -Today the schema engine -([internal/schema](../internal/schema)) validates -*structure*: the heading tree, repeat cardinality, -content-node kinds, and front-matter CUE constraints. -But its [Scope](../internal/schema/schema.go) and -`ContentEntry` nodes are **anonymous**. The matcher knows -a `## {id}` heading must exist, yet not what key `id` -becomes in output, and it never captures node bodies. -Front matter is already YAML-typed and passes through -unchanged. The gap is the document body. - -So this plan adds exactly one new schema concept β€” a -**binding / projection layer** β€” plus a thin extractor -that walks the existing validated match. No second -parser, and no schema-to-type inference. +emits a data tree. Its shape is derived from the schema +hierarchy itself β€” no annotations required. -## Sequencing +## Why a default binding layer first -The schema engine is mid-rework. This plan must land -after, and consume the outputs of, that work rather than -the legacy single-source model. +The schema is already a hierarchy: front matter, then a +tree of scopes (sections), each with child scopes and +content entries. That hierarchy *is* the data shape. So +the first deliverable is a **default binding layer** that +projects the schema tree into a data tree directly, +mirroring its nesting. No new schema concept is needed +for the common case. -- **Plan 156 / PR #288 (schema composition).** A file can - resolve to multiple kinds whose schemas compose via - `schema.Compose()`. The extractor consumes the composed - `Schema`, never a single `Rule.Schema`. This forces a - new rule: `bind:` names must compose too. Identical - headings from different kinds merge their bound - children. Two kinds binding the same node to different - names is a schema error raised at compose time. -- **Plan 149 (section-content schema).** Body extraction - rides on the `ContentEntry` model from the - content-schema work. This plan adds no content matcher - of its own and is blocked until that model is stable. -- **Plan 147 / PR #284 (actionable schema diagnostics).** - If landed, bind validation and conformance failures - reuse the `SchemaDiagnostic` formatter. - -## The `bind:` concept +Custom shaping is *not* in this plan. It is a separate +follow-up β€” [plan 164](164_custom-binding-overrides.md) β€” +and we keep it cheap by design: every key flows through +one `keyFor(node)` seam (task 3), so the override plan is +a focused change there plus parsing `bind:`. Until then, +renaming or restructuring is the job of a downstream tool +(`jq`, `yq`, a Lua script) over the standard-format +output. -`bind:` is an optional string key on any schema scope or -content entry. It names the value that node contributes -to the extracted tree. Unbound nodes are structural only -and emit nothing, keeping output intentional. +## Default projection rules -Mapping rules: +The projection walks the composed schema in lockstep with +the validated match and mirrors the hierarchy: - **Front matter** β†’ top-level `frontmatter` object, passed through from the existing decode. -- **Non-repeating scope, `bind: x`** β†’ object `x` holding - its bound children. -- **Repeating scope, `bind: xs`** (a `{placeholder}` - heading) β†’ array `xs`. Each element is an object whose - fields are the captured placeholders plus bound - children. -- **`code-block`, `bind: c`** β†’ string `c` (raw body); - optional `parse: yaml|json` embeds the decoded value. -- **`list`, `bind: items`** β†’ array of item strings. -- **`table` with `columns`, `bind: rows`** β†’ array of row - objects keyed by column header. -- **`paragraph`, `bind: t`** β†’ its text. +- **Literal-heading scope** (`## Goal`) β†’ object keyed by + the slugified heading (`goal`), reusing the existing + anchor slugifier. Its value holds child scopes and + content, recursively. +- **Repeating scope** (`## {id}`, `repeats: true`) β†’ an + array keyed by the slug of the heading's literal stem + (or, if none, the placeholder name). Each element is an + object whose fields are the captured placeholders plus + the element's own child scopes and content. +- **`code-block`** β†’ string under `code` (raw body); + multiple blocks get `code`, `code-2`, … +- **`list`** β†’ array of item strings under `items`. +- **`table` with `columns`** β†’ array of row objects keyed + by column header, under `rows`. +- **`paragraph`** β†’ its text under `text`. + +Sibling key collisions (two `## Goal` headings, or a +content default that shadows a child scope slug) are a +schema error reported at extract time, pointing at the +schema source. Empty/optional sections that did not match +are omitted rather than emitted as null. + +## Sequencing + +The schema engine is mid-rework. This plan lands after, +and consumes the outputs of, that work β€” not the legacy +single-source model. + +- **[Plan 156 β€” kind-schema + composition](156_kind-schema-composition.md) / PR + #288.** (Disambiguation: two plan files share id 156; + this dependency is the composition one, not + `156_schema-entry-unification.md`.) A file can resolve + to multiple kinds whose schemas compose via + `schema.Compose()`. The extractor consumes the composed + `Schema`. Default keys derive from heading text, so + identical headings from two kinds merge to the same key + with no conflict; only genuinely divergent shapes + surface as a collision. +- **Plan 149 (section-content schema).** Content + projection rides on the `ContentEntry` model from the + content-schema work. This plan adds no content matcher + of its own and is blocked until that model is stable. +- **Plan 147 / PR #284 (actionable schema diagnostics).** + If landed, collision and conformance failures reuse the + `SchemaDiagnostic` formatter. Extraction is gated on a successful schema match. A non-conformant file makes `extract` report the same @@ -91,48 +99,52 @@ partial data. ## Tasks -1. **Parse `bind:` (red/green).** Add a `Bind string` - field to `Scope` and `ContentEntry` in - [schema.go](../internal/schema/schema.go); parse it in - [parse_inline.go](../internal/schema/parse_inline.go) - and [parse_file.go](../internal/schema/parse_file.go). - Unit-test round-trip and that an empty bind means - "structural only". -2. **Validate `bind:` names.** Reject duplicate sibling - binds, and binds whose value is unreachable because a - parent is unbound. Surface via the `SchemaDiagnostic` - path (plan 147) if landed, else a plain parse error. -3. **Compose `bind:` across kinds.** Extend - `schema.Compose()` (plan 156) so merged headings union - their bound children. Binding one composed node to two - different names is a compose-time error. Add - `compose_test.go` cases for union and conflict. -4. **Extractor package.** Add `internal/extract` with - `Extract(f *lint.File, sch *schema.Schema, match …) - (any, []lint.Diagnostic)`. `sch` is the composed - schema. It consumes the existing schema-validation - walk (extend `schema.Validate` / the content matcher - to expose the scopeβ†’nodes match tree) rather than - re-matching. -5. **Capture placeholder values.** When a repeating scope - matches a `{field}` heading pattern, record each - captured field into the element object, reusing +1. **Expose the match tree.** Refactor `schema.Validate` + (and the content matcher) to also return a new + `*schema.MatchTree` in `internal/schema`: for each + `Scope` / `ContentEntry`, the matched AST nodes, their + source lines, and captured `{field}` values. `Validate` + keeps its diagnostic return; the tree is an added + result so MDS020 is unaffected. Unit-test the tree on + the existing schema fixtures. +2. **Extractor skeleton (red/green).** Add + `internal/extract` with `Extract(f *lint.File, sch + *schema.Schema, m *schema.MatchTree) (any, + []lint.Diagnostic)`. `sch` is the composed schema; `m` + is the tree from task 1 β€” no re-matching. +3. **Default scope projection.** Walk the scope tree and + build the nested object/array structure per the rules + above. Route every key through one `keyFor(node)` + function β€” the single seam a future custom-binding plan + overrides. Reuse the existing anchor slugifier. Unit- + test literal, nested, and optional-omitted scopes. +4. **Repeating scopes and placeholders.** Project + `repeats: true` scopes as arrays; record each captured + `{field}` into the element object, reusing [fieldinterp](../internal/fieldinterp/fieldinterp.go). -6. **Format encoders.** Add `internal/extract/encode` +5. **Default content projection.** Project `code-block`, + `list`, `table`, and `paragraph` entries (plan 149) + with their default keys. Detect sibling key collisions + and emit a schema diagnostic. +6. **Composition behavior.** Add `compose_test.go` / + extractor tests proving a file under two kinds yields a + merged tree, and that a real shape divergence is + reported as a collision, not silently dropped. +7. **Format encoders.** Add `internal/extract/encode` with json (stdlib), yaml (existing dep), msgpack, and lua (table literal) encoders behind a `Format` enum. -7. **`extract` subcommand.** Register `extract` in +8. **`extract` subcommand.** Register `extract` in [main.go](../cmd/mdsmith/main.go); signature `mdsmith extract --format `. Reuse the config-load and kind-resolution helpers from [kinds.go](../cmd/mdsmith/kinds.go). Validate that `` is one of the file's resolved kinds. Run schema validation first and abort on failure. -8. **Fixtures and integration test.** Add a kind with a - bound schema under `testdata/`, a conformant sample, - and golden outputs per format. Assert non-conformant - input exits non-zero with check diagnostics. -9. **Docs.** Add a section under +9. **Fixtures and integration test.** Add a kind with a + schema under `testdata/`, a conformant sample, and + golden outputs per format. Assert non-conformant input + exits non-zero with check diagnostics. +10. **Docs.** Add a section under [schemas.md](../docs/guides/schemas.md) and a `docs/reference/cli/extract.md` page. Both are picked up by existing catalog directives. Run `mdsmith fix` @@ -140,20 +152,19 @@ partial data. ## Acceptance Criteria -- [ ] `bind:` parses on scopes and content entries from - inline and `proto.md` schemas. Duplicate and - unreachable binds are rejected with actionable - diagnostics. - [ ] `mdsmith extract --format json ` on a - conformant file emits a tree matching the bound - schema, with front matter under `frontmatter`. -- [ ] A file resolving to multiple kinds composes its - binds; conflicting binds on one composed node are a - compose-time error. -- [ ] Repeating sections become arrays. Placeholder - captures and bound children appear as element - fields. Code-block, list, table, and paragraph - binds extract as specified. + conformant file emits a tree whose nesting mirrors + the schema hierarchy, with front matter under + `frontmatter` β€” no schema annotations required. +- [ ] Literal headings key by slug; repeating sections + become arrays; captured placeholders and child + scopes/content appear as element fields. +- [ ] Code-block, list, table, and paragraph entries + project under their default keys; sibling key + collisions are reported as schema diagnostics. +- [ ] A file resolving to multiple kinds yields a merged + tree; a genuine shape divergence is reported, not + silently dropped. - [ ] `yaml`, `lua`, and `msgpack` produce equivalent data; golden fixtures cover all four formats. - [ ] A non-conformant file makes `extract` exit non-zero @@ -166,8 +177,12 @@ partial data. ## Open questions -- Should `bind:` default to the slugified heading text, - or stay strictly opt-in? The plan assumes opt-in. +- Repeating-scope array key: slug of the literal stem vs. + the placeholder name. Plan assumes literal stem, else + placeholder name. +- Custom bindings (rename/restructure) ship in [plan + 164](164_custom-binding-overrides.md), layered on the + `keyFor` seam; out of scope here. - Lua output: bare `return { … }` table to start, not a named module. - Exposing extraction over the LSP or a `query`-style diff --git a/plan/164_custom-binding-overrides.md b/plan/164_custom-binding-overrides.md new file mode 100644 index 000000000..b12fc3ddb --- /dev/null +++ b/plan/164_custom-binding-overrides.md @@ -0,0 +1,64 @@ +--- +id: 164 +title: Custom binding overrides for mdsmith extract +status: "πŸ”²" +model: opus +depends-on: [163] +summary: >- + Add an opt-in `bind:` key that overrides the default + schema-derived key in `mdsmith extract`, layered on + the `keyFor` seam from plan 163. +--- +# Custom binding overrides for mdsmith extract + +## Goal + +[Plan 163](163_schema-driven-data-extraction.md) derives +the extracted data tree from the schema hierarchy with no +annotations. This plan adds an opt-in `bind:` key that +renames or restructures a node when the default key is +wrong, without changing the default behavior. + +## Why this is a small change + +Plan 163 routes every key through one `keyFor(node)` +function. This plan only changes that function and adds +parsing. The walk, encoders, and CLI are untouched. + +- **`keyFor(node)`** returns `node.Bind` when set, else + the default slug. +- A node with `bind: ""` (explicit empty) is hoisted: + its children merge into the parent instead of nesting. +- Composition rule: two kinds binding one composed node + to different names is a compose-time error, reusing the + collision diagnostic from plan 163. + +## Tasks + +1. **Parse `bind:`.** Add `Bind string` to `Scope` and + `ContentEntry`; parse in `parse_inline.go` and + `parse_file.go`. Unit-test round-trip. +2. **Override `keyFor`.** Return the bind name when set; + implement hoist for `bind: ""`. +3. **Validate binds.** Reject duplicate sibling binds and + unreachable binds via the schema diagnostic path. +4. **Compose binds.** Extend `schema.Compose()` so merged + headings union bound children; conflicting names are a + compose-time error. +5. **Fixtures and docs.** Add bind-override golden cases; + document `bind:` under + [schemas.md](../docs/guides/schemas.md). + +## Acceptance Criteria + +- [ ] `bind:` overrides the default key; output is + otherwise identical to plan 163. +- [ ] `bind: ""` hoists a node's children into its + parent. +- [ ] Duplicate or unreachable binds are rejected with + actionable diagnostics. +- [ ] Conflicting binds across composed kinds are a + compose-time error. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes From 36aff7f7663e0afc30e67dd22da90e9a1e1d61e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:16:58 +0000 Subject: [PATCH 03/12] plan 164: use *string for Bind to distinguish unset vs explicit-empty Addresses Copilot review: a plain string cannot represent "unset" vs `bind: ""` (hoist), so Bind is now *string. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/164_custom-binding-overrides.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/plan/164_custom-binding-overrides.md b/plan/164_custom-binding-overrides.md index b12fc3ddb..f684b5219 100644 --- a/plan/164_custom-binding-overrides.md +++ b/plan/164_custom-binding-overrides.md @@ -25,21 +25,24 @@ Plan 163 routes every key through one `keyFor(node)` function. This plan only changes that function and adds parsing. The walk, encoders, and CLI are untouched. -- **`keyFor(node)`** returns `node.Bind` when set, else - the default slug. -- A node with `bind: ""` (explicit empty) is hoisted: - its children merge into the parent instead of nesting. +- **`keyFor(node)`** returns the bind value when present, + else the default slug. `Bind` is a `*string` so an + unset key and an explicit empty one are distinct. +- A node with `bind: ""` (present, empty) is hoisted: its + children merge into the parent instead of nesting. - Composition rule: two kinds binding one composed node to different names is a compose-time error, reusing the collision diagnostic from plan 163. ## Tasks -1. **Parse `bind:`.** Add `Bind string` to `Scope` and - `ContentEntry`; parse in `parse_inline.go` and - `parse_file.go`. Unit-test round-trip. -2. **Override `keyFor`.** Return the bind name when set; - implement hoist for `bind: ""`. +1. **Parse `bind:`.** Add `Bind *string` to `Scope` and + `ContentEntry` (nil = unset, non-nil = present, so + `bind: ""` is distinguishable); parse in + `parse_inline.go` and `parse_file.go`. Unit-test + round-trip including unset vs. explicit-empty. +2. **Override `keyFor`.** Return the bind value when + present; implement hoist for `bind: ""`. 3. **Validate binds.** Reject duplicate sibling binds and unreachable binds via the schema diagnostic path. 4. **Compose binds.** Extend `schema.Compose()` so merged From 554626d71a1242ad06d373da759d6f98dfc95e53 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:24:11 +0000 Subject: [PATCH 04/12] plan 163: resolve open questions - Repeating-scope elements always retain each placeholder as a name:value field. - Root holds `frontmatter` plus projected sections beside it (no nesting under a single key). - Preamble projected under `preamble`; wildcard/unlisted skipped. - Defer Lua; ship json/yaml/msgpack. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/163_schema-driven-data-extraction.md | 103 ++++++++++++++-------- 1 file changed, 66 insertions(+), 37 deletions(-) diff --git a/plan/163_schema-driven-data-extraction.md b/plan/163_schema-driven-data-extraction.md index fdf0fb0e5..66288fdfa 100644 --- a/plan/163_schema-driven-data-extraction.md +++ b/plan/163_schema-driven-data-extraction.md @@ -7,7 +7,7 @@ depends-on: [149, 156] summary: >- Derive a default data tree from the hierarchical schema and add an `extract` subcommand that emits a - kind-conformant file as JSON/YAML/Lua/msgpack. + kind-conformant file as JSON/YAML/msgpack. --- # Schema-driven data extraction (mdsmith extract) @@ -15,9 +15,9 @@ summary: >- Let a kind's schema double as an extraction contract. Once `mdsmith check` confirms a file conforms, `mdsmith -extract --format json|yaml|lua|msgpack ` -emits a data tree. Its shape is derived from the schema -hierarchy itself β€” no annotations required. +extract --format json|yaml|msgpack ` emits a +data tree. Its shape is derived from the schema hierarchy +itself β€” no annotations required. ## Why a default binding layer first @@ -35,25 +35,35 @@ and we keep it cheap by design: every key flows through one `keyFor(node)` seam (task 3), so the override plan is a focused change there plus parsing `bind:`. Until then, renaming or restructuring is the job of a downstream tool -(`jq`, `yq`, a Lua script) over the standard-format -output. +(`jq`, `yq`) over the standard-format output. ## Default projection rules The projection walks the composed schema in lockstep with the validated match and mirrors the hierarchy: -- **Front matter** β†’ top-level `frontmatter` object, - passed through from the existing decode. +- **Root shape.** The root object holds a `frontmatter` + object (the decoded front matter, unchanged) *and* the + projected sections beside it at the same level. Front + matter stays grouped so it never collides with a + section slug. - **Literal-heading scope** (`## Goal`) β†’ object keyed by the slugified heading (`goal`), reusing the existing anchor slugifier. Its value holds child scopes and content, recursively. - **Repeating scope** (`## {id}`, `repeats: true`) β†’ an - array keyed by the slug of the heading's literal stem - (or, if none, the placeholder name). Each element is an - object whose fields are the captured placeholders plus - the element's own child scopes and content. + array keyed by the slug of the heading's literal stem, + or the placeholder name if the heading is only a + placeholder. Each element is an object that **always + retains every captured placeholder as a `name: value` + field** (both the placeholder name and its value + survive), plus the element's own child scopes and + content. +- **Preamble** (content before the first heading, when + the schema declares a preamble scope) β†’ projected under + a `preamble` key. Wildcard slots and unlisted/closed + headings are skipped: the output is a faithful + projection of the *declared* schema only. - **`code-block`** β†’ string under `code` (raw body); multiple blocks get `code`, `code-2`, … - **`list`** β†’ array of item strings under `items`. @@ -113,14 +123,18 @@ partial data. []lint.Diagnostic)`. `sch` is the composed schema; `m` is the tree from task 1 β€” no re-matching. 3. **Default scope projection.** Walk the scope tree and - build the nested object/array structure per the rules - above. Route every key through one `keyFor(node)` - function β€” the single seam a future custom-binding plan - overrides. Reuse the existing anchor slugifier. Unit- - test literal, nested, and optional-omitted scopes. + build the nested structure per the rules above: + `frontmatter` plus sections at the root, literal scopes + keyed by slug, preamble under `preamble`, wildcard / + unlisted skipped. Route every key through one + `keyFor(node)` function β€” the single seam a future + custom-binding plan overrides. Reuse the existing + anchor slugifier. Unit-test literal, nested, preamble, + and optional-omitted scopes. 4. **Repeating scopes and placeholders.** Project - `repeats: true` scopes as arrays; record each captured - `{field}` into the element object, reusing + `repeats: true` scopes as arrays; each element retains + every captured `{field}` as a `name: value` field, + reusing [fieldinterp](../internal/fieldinterp/fieldinterp.go). 5. **Default content projection.** Project `code-block`, `list`, `table`, and `paragraph` entries (plan 149) @@ -131,8 +145,8 @@ partial data. merged tree, and that a real shape divergence is reported as a collision, not silently dropped. 7. **Format encoders.** Add `internal/extract/encode` - with json (stdlib), yaml (existing dep), msgpack, and - lua (table literal) encoders behind a `Format` enum. + with json (stdlib), yaml (existing dep), and msgpack + encoders behind a `Format` enum. (Lua is deferred.) 8. **`extract` subcommand.** Register `extract` in [main.go](../cmd/mdsmith/main.go); signature `mdsmith extract --format `. Reuse the @@ -154,19 +168,24 @@ partial data. - [ ] `mdsmith extract --format json ` on a conformant file emits a tree whose nesting mirrors - the schema hierarchy, with front matter under - `frontmatter` β€” no schema annotations required. + the schema hierarchy β€” no schema annotations + required. +- [ ] The root holds a `frontmatter` object and the + projected sections beside it at the same level. - [ ] Literal headings key by slug; repeating sections - become arrays; captured placeholders and child - scopes/content appear as element fields. + become arrays; each element retains every captured + placeholder as a `name: value` field plus its child + scopes/content. +- [ ] Preamble is projected under `preamble`; wildcard + and unlisted/closed headings are skipped. - [ ] Code-block, list, table, and paragraph entries project under their default keys; sibling key collisions are reported as schema diagnostics. - [ ] A file resolving to multiple kinds yields a merged tree; a genuine shape divergence is reported, not silently dropped. -- [ ] `yaml`, `lua`, and `msgpack` produce equivalent - data; golden fixtures cover all four formats. +- [ ] `json`, `yaml`, and `msgpack` produce equivalent + data; golden fixtures cover all three formats. - [ ] A non-conformant file makes `extract` exit non-zero and print the same diagnostics as `mdsmith check`. - [ ] An unknown kind, or a kind not assigned to the @@ -175,15 +194,25 @@ partial data. - [ ] `go tool golangci-lint run` reports no issues - [ ] `mdsmith check .` passes -## Open questions - -- Repeating-scope array key: slug of the literal stem vs. - the placeholder name. Plan assumes literal stem, else - placeholder name. -- Custom bindings (rename/restructure) ship in [plan +## Decisions + +- **Repeating-scope key.** Array key is the slug of the + heading's literal stem, or the placeholder name when + the heading is only a placeholder. Each element always + retains every captured placeholder as a `name: value` + field, so both the name and the value survive. +- **Front matter placement.** The root holds a + `frontmatter` object and the projected sections beside + it at the same level. Grouping front matter avoids + collisions with section slugs. +- **Non-listed nodes.** Preamble is projected under + `preamble`; wildcard slots and unlisted/closed headings + are skipped. +- **Lua deferred.** Ship json, yaml, and msgpack. A Lua + encoder can be added later behind the same `Format` + enum. +- **Custom bindings** ship in [plan 164](164_custom-binding-overrides.md), layered on the `keyFor` seam; out of scope here. -- Lua output: bare `return { … }` table to start, not a - named module. -- Exposing extraction over the LSP or a `query`-style - selector is out of scope here. +- **LSP / `query`-style selector** for extraction is out + of scope here. From fa36f44f145361c9f0c300fff5ec3be526d06ca5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:27:42 +0000 Subject: [PATCH 05/12] plan 163: use current repeat:{min,max} terminology, not repeats:true The schema parser rejects the legacy `repeats` key; repeating cardinality is `repeat: {min, max}`. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/163_schema-driven-data-extraction.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/plan/163_schema-driven-data-extraction.md b/plan/163_schema-driven-data-extraction.md index 66288fdfa..e1f2b772d 100644 --- a/plan/163_schema-driven-data-extraction.md +++ b/plan/163_schema-driven-data-extraction.md @@ -51,8 +51,9 @@ the validated match and mirrors the hierarchy: the slugified heading (`goal`), reusing the existing anchor slugifier. Its value holds child scopes and content, recursively. -- **Repeating scope** (`## {id}`, `repeats: true`) β†’ an - array keyed by the slug of the heading's literal stem, +- **Repeating scope** (`## {id}` with a `repeat: {min, + max}` cardinality) β†’ an array keyed by the slug of the + heading's literal stem, or the placeholder name if the heading is only a placeholder. Each element is an object that **always retains every captured placeholder as a `name: value` @@ -131,8 +132,9 @@ partial data. custom-binding plan overrides. Reuse the existing anchor slugifier. Unit-test literal, nested, preamble, and optional-omitted scopes. -4. **Repeating scopes and placeholders.** Project - `repeats: true` scopes as arrays; each element retains +4. **Repeating scopes and placeholders.** Project scopes + with a `repeat: {min, max}` cardinality as arrays; each + element retains every captured `{field}` as a `name: value` field, reusing [fieldinterp](../internal/fieldinterp/fieldinterp.go). From 4c985265a1c6c423307b0d057397e6ed1cc8921e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 11:22:44 +0000 Subject: [PATCH 06/12] Add plan 165: portable Markdown export (mdsmith export) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A source-to-source `export` subcommand that strips directive markers, keeps generated bodies, and inlines includes β€” distinct from schema extraction (163/164). https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- PLAN.md | 1 + plan/165_portable-markdown-export.md | 123 +++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 plan/165_portable-markdown-export.md diff --git a/PLAN.md b/PLAN.md index f9fae8e40..1a35114f7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -93,4 +93,5 @@ footer: | | 163 | πŸ”² | opus | [Schema-driven data extraction (mdsmith extract)](plan/163_schema-driven-data-extraction.md) | | 164 | πŸ”² | opus | [Custom binding overrides for mdsmith extract](plan/164_custom-binding-overrides.md) | | 164 | βœ… | | [GitHub-UI-triggered releases and a split website deploy](plan/164_github-ui-releases-and-split-website.md) | +| 165 | πŸ”² | opus | [Portable Markdown export (mdsmith export)](plan/165_portable-markdown-export.md) | diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md new file mode 100644 index 000000000..3ea7a27ab --- /dev/null +++ b/plan/165_portable-markdown-export.md @@ -0,0 +1,123 @@ +--- +id: 165 +title: Portable Markdown export (mdsmith export) +status: "πŸ”²" +model: opus +depends-on: [] +summary: >- + Add an `export` subcommand that writes a portable, + directive-free copy of a Markdown file: markers + removed, generated bodies kept, includes inlined. +--- +# Portable Markdown export (mdsmith export) + +## Goal + +`mdsmith export ` writes a portable copy of a +Markdown file with every `` directive marker +removed. Generated bodies stay as plain Markdown and +`` content is inlined. The result renders +identically on any Markdown tool with no mdsmith +knowledge. + +## Why a separate command + +This is not schema extraction. `extract` (plan 163) +projects a kind's schema into a data tree. `export` is a +source-to-source transform of the document itself. It +needs no kind, schema, or conformance gate β€” only that +the file parses and its directive bodies are fresh. + +Mixing it into `extract --format markdown` would couple a +plain-document transform onto the schema-projection +command. A dedicated `export` keeps the two concerns +apart and leaves room to grow (output path, later batch). + +## Behavior + +- Regenerate directive bodies in memory first, reusing + the same engine as `mdsmith fix`, so output is never + stale. The source file is never modified. +- Drop the opening and closing marker lines of every + directive region; keep the body text between them + verbatim. +- `` bodies are already expanded by + regeneration, so keeping the body inlines the included + content (recursively). +- Markerless directives with no body (for example + ``, ``) are removed + outright. +- Nested same-type markers that the engine treats as + literal content of an outer directive are preserved, + by reusing the engine's directive-range detection + rather than a separate scan. +- After stripping, normalize blank lines so the output is + stable and lint-clean. Front matter is kept as-is. +- Exporting an already directive-free file is a no-op; + `export` is idempotent. + +## Tasks + +1. **Export core (red/green).** Add `internal/export` + with `Export(f *lint.File) ([]byte, error)`: + regenerate directive bodies in memory via the `fix` + directive engine, then remove marker lines while + keeping bodies. Unit-test marker removal, body + retention, include inlining, and the no-directive + no-op. +2. **Nested / literal-content markers.** Drive removal + off the engine's directive ranges (e.g. + `lint.File` generated ranges) so inner same-type + markers that are literal content survive. Add a test. +3. **Whitespace normalization.** Collapse the blank + lines left by removed markers so output is stable and + passes `mdsmith check`. Test idempotence: export of + export equals export. +4. **`export` subcommand.** Register `export` in + [main.go](../cmd/mdsmith/main.go); `mdsmith export + ` writes to stdout, `-o/--output ` writes + a file. Never mutate the source. Reuse the config and + file-load helpers that back `fix` in + [main.go](../cmd/mdsmith/main.go). Exit non-zero with + a clear message on parse errors. +5. **Fixtures and integration test.** Add `testdata` + inputs covering include, catalog, toc, and build + directives with golden directive-free outputs. Assert + idempotence and that the output passes `mdsmith + check`. +6. **Docs.** Add `docs/reference/cli/export.md` and link + it from the CLI reference catalog. Run `mdsmith fix` + so catalogs and PLAN.md regenerate. + +## Acceptance Criteria + +- [ ] `mdsmith export ` emits the file with all + directive markers removed and generated bodies + kept; `` content is inlined. +- [ ] The source file is never modified. +- [ ] Stale directive bodies are regenerated before + stripping, so the output is never stale. +- [ ] Nested same-type literal-content markers are + preserved. +- [ ] Output is idempotent and passes `mdsmith check`. +- [ ] `-o ` writes to a file; stdout is the + default. +- [ ] A parse error or missing file exits non-zero with + a clear message. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes + +## Decisions + +- **Keep generated bodies.** Markers are stripped but + TOC, catalog, and included content stay as plain + Markdown; includes are inlined for a portable copy. +- **New `export` subcommand.** Not a fourth `extract` + format and not a `fix` flag; a dedicated command keeps + the source-to-source transform separate from schema + extraction. +- **Front matter retained.** It is not a directive; + stripping it is out of scope. +- **Single file first.** Directory or glob batch export + is a possible follow-up, not in this plan. From e199eb26bc0b853af2ac16baf0fba388851e2751 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:11:16 +0000 Subject: [PATCH 07/12] =?UTF-8?q?plan=20163:=20align=20with=20landed=20sch?= =?UTF-8?q?ema=20syntax=20=E2=80=94=20no-heading=20section,=20no=20preambl?= =?UTF-8?q?e=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan 156 entry-unification landed in main (PR #295): the standalone preamble: key is gone, replaced by heading: null. Project the no-heading section's content into the enclosing object instead of a preamble wrapper key. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/163_schema-driven-data-extraction.md | 57 +++++++++++++++-------- 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/plan/163_schema-driven-data-extraction.md b/plan/163_schema-driven-data-extraction.md index e1f2b772d..fd0edee59 100644 --- a/plan/163_schema-driven-data-extraction.md +++ b/plan/163_schema-driven-data-extraction.md @@ -60,11 +60,15 @@ the validated match and mirrors the hierarchy: field** (both the placeholder name and its value survive), plus the element's own child scopes and content. -- **Preamble** (content before the first heading, when - the schema declares a preamble scope) β†’ projected under - a `preamble` key. Wildcard slots and unlisted/closed - headings are skipped: the output is a faithful - projection of the *declared* schema only. +- **No-heading section** (`heading: null` β€” content + before the first child heading) has no heading text and + therefore no slug. Its content entries project + **directly into the enclosing object** (root, or the + parent section) beside the headed-section keys β€” there + is no `preamble` wrapper key. Wildcard slots + (`regex: '.+'`) and unlisted/closed headings are + skipped: the output is a faithful projection of the + *declared* schema only. - **`code-block`** β†’ string under `code` (raw body); multiple blocks get `code`, `code-2`, … - **`list`** β†’ array of item strings under `items`. @@ -80,16 +84,23 @@ are omitted rather than emitted as null. ## Sequencing -The schema engine is mid-rework. This plan lands after, -and consumes the outputs of, that work β€” not the legacy -single-source model. - +This plan consumes the reworked schema engine, not the +legacy single-source model. + +- **Entry-shape unification (`156_schema-entry-unification` + / PR #295) β€” landed in main.** Every `sections:` entry + is discriminated by its `heading:` value: a string or + `{regex, repeat?, sequential?}` mapping for headed + sections, and `heading: null` for the no-heading section + (content before the first child heading). There is no + standalone `preamble:` key. The projection rules above + target this shape directly. - **[Plan 156 β€” kind-schema composition](156_kind-schema-composition.md) / PR #288.** (Disambiguation: two plan files share id 156; - this dependency is the composition one, not - `156_schema-entry-unification.md`.) A file can resolve - to multiple kinds whose schemas compose via + this dependency is the composition one, not the + now-landed `156_schema-entry-unification`.) A file can + resolve to multiple kinds whose schemas compose via `schema.Compose()`. The extractor consumes the composed `Schema`. Default keys derive from heading text, so identical headings from two kinds merge to the same key @@ -126,12 +137,13 @@ partial data. 3. **Default scope projection.** Walk the scope tree and build the nested structure per the rules above: `frontmatter` plus sections at the root, literal scopes - keyed by slug, preamble under `preamble`, wildcard / + keyed by slug, the `heading: null` no-heading section's + content hoisted into the enclosing object, wildcard / unlisted skipped. Route every key through one `keyFor(node)` function β€” the single seam a future custom-binding plan overrides. Reuse the existing - anchor slugifier. Unit-test literal, nested, preamble, - and optional-omitted scopes. + anchor slugifier. Unit-test literal, nested, + no-heading-section, and optional-omitted scopes. 4. **Repeating scopes and placeholders.** Project scopes with a `repeat: {min, max}` cardinality as arrays; each element retains @@ -178,8 +190,10 @@ partial data. become arrays; each element retains every captured placeholder as a `name: value` field plus its child scopes/content. -- [ ] Preamble is projected under `preamble`; wildcard - and unlisted/closed headings are skipped. +- [ ] A `heading: null` no-heading section's content + projects into its enclosing object (no `preamble` + wrapper key); wildcard and unlisted/closed headings + are skipped. - [ ] Code-block, list, table, and paragraph entries project under their default keys; sibling key collisions are reported as schema diagnostics. @@ -207,9 +221,12 @@ partial data. `frontmatter` object and the projected sections beside it at the same level. Grouping front matter avoids collisions with section slugs. -- **Non-listed nodes.** Preamble is projected under - `preamble`; wildcard slots and unlisted/closed headings - are skipped. +- **No-heading section.** A `heading: null` entry has no + slug; its content projects directly into the enclosing + object rather than under a `preamble` wrapper key. The + sibling-collision rule covers any clash with a section + slug. Wildcard slots and unlisted/closed headings are + skipped. - **Lua deferred.** Ship json, yaml, and msgpack. A Lua encoder can be added later behind the same `Format` enum. From 790426511edb5c42fc7e4fc63135bb88be8125d0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:17:58 +0000 Subject: [PATCH 08/12] Renumber plans 163/164 -> 166/167; fix gensection API ref in 165 Rebasing onto main pulled in plan/163_public-markdown-library and plan/164_github-ui-releases-and-split-website, colliding with the new plans. Renumber schema-extraction 163->166 and custom-bindings 164->167 (export stays 165, no collision); update cross-references and depends-on. Plan 165: marker stripping is driven by gensection.FindMarkerPairs (MarkerPair.StartLine/EndLine), not lint.File.GeneratedRanges (which only records include/catalog body ranges). Tighten the acceptance criteria to distinguish engine-recognized markers from literal-content marker-like text. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- PLAN.md | 4 +-- plan/165_portable-markdown-export.md | 33 ++++++++++++------- ...d => 166_schema-driven-data-extraction.md} | 6 ++-- ...des.md => 167_custom-binding-overrides.md} | 8 ++--- 4 files changed, 31 insertions(+), 20 deletions(-) rename plan/{163_schema-driven-data-extraction.md => 166_schema-driven-data-extraction.md} (98%) rename plan/{164_custom-binding-overrides.md => 167_custom-binding-overrides.md} (95%) diff --git a/PLAN.md b/PLAN.md index 1a35114f7..ecf7dfa5d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -90,8 +90,8 @@ footer: | | 161 | πŸ”³ | sonnet | [Expose rule maintainability patterns via CLI help and LSP](plan/161_rule-pattern-metadata.md) | | 162 | πŸ”² | sonnet | [Split the overloaded `meta` rule category](plan/162_rule-category-cleanup.md) | | 163 | πŸ”² | | [Extract mdsmith Markdown parse/produce as a public Go library](plan/163_public-markdown-library.md) | -| 163 | πŸ”² | opus | [Schema-driven data extraction (mdsmith extract)](plan/163_schema-driven-data-extraction.md) | -| 164 | πŸ”² | opus | [Custom binding overrides for mdsmith extract](plan/164_custom-binding-overrides.md) | | 164 | βœ… | | [GitHub-UI-triggered releases and a split website deploy](plan/164_github-ui-releases-and-split-website.md) | | 165 | πŸ”² | opus | [Portable Markdown export (mdsmith export)](plan/165_portable-markdown-export.md) | +| 166 | πŸ”² | opus | [Schema-driven data extraction (mdsmith extract)](plan/166_schema-driven-data-extraction.md) | +| 167 | πŸ”² | opus | [Custom binding overrides for mdsmith extract](plan/167_custom-binding-overrides.md) | diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md index 3ea7a27ab..839d40203 100644 --- a/plan/165_portable-markdown-export.md +++ b/plan/165_portable-markdown-export.md @@ -22,7 +22,7 @@ knowledge. ## Why a separate command -This is not schema extraction. `extract` (plan 163) +This is not schema extraction. `extract` (plan 166) projects a kind's schema into a data tree. `export` is a source-to-source transform of the document itself. It needs no kind, schema, or conformance gate β€” only that @@ -47,10 +47,11 @@ apart and leaves room to grow (output path, later batch). - Markerless directives with no body (for example ``, ``) are removed outright. -- Nested same-type markers that the engine treats as - literal content of an outer directive are preserved, - by reusing the engine's directive-range detection - rather than a separate scan. +- Only lines the engine's marker-pair detection + recognizes as real directive start/end markers are + removed. Marker-like text the engine treats as literal + content (for example inner same-type markers nested in + an outer directive) is left untouched. - After stripping, normalize blank lines so the output is stable and lint-clean. Front matter is kept as-is. - Exporting an already directive-free file is a no-op; @@ -66,9 +67,17 @@ apart and leaves room to grow (output path, later batch). retention, include inlining, and the no-directive no-op. 2. **Nested / literal-content markers.** Drive removal - off the engine's directive ranges (e.g. - `lint.File` generated ranges) so inner same-type - markers that are literal content survive. Add a test. + off the engine's own marker-pair detection β€” + `gensection.FindMarkerPairs` in + [internal/archetype/gensection](../internal/archetype/gensection/parse.go), + whose `MarkerPair.StartLine`/`EndLine` give the exact + start- and end-marker line for every directive (not + just the include/catalog *body* ranges that + `lint.File.GeneratedRanges` records for diagnostic + suppression). Only lines the engine recognizes as real + markers are removed, so inner same-type markers that + the engine treats as literal content survive. Add a + test. 3. **Whitespace normalization.** Collapse the blank lines left by removed markers so output is stable and passes `mdsmith check`. Test idempotence: export of @@ -91,9 +100,11 @@ apart and leaves room to grow (output path, later batch). ## Acceptance Criteria -- [ ] `mdsmith export ` emits the file with all - directive markers removed and generated bodies - kept; `` content is inlined. +- [ ] `mdsmith export ` removes every line the + engine recognizes as a real directive start/end + marker, keeps generated bodies, and inlines + `` content. Marker-like text treated as + literal content is left in place. - [ ] The source file is never modified. - [ ] Stale directive bodies are regenerated before stripping, so the output is never stale. diff --git a/plan/163_schema-driven-data-extraction.md b/plan/166_schema-driven-data-extraction.md similarity index 98% rename from plan/163_schema-driven-data-extraction.md rename to plan/166_schema-driven-data-extraction.md index fd0edee59..dfc6ff9de 100644 --- a/plan/163_schema-driven-data-extraction.md +++ b/plan/166_schema-driven-data-extraction.md @@ -1,5 +1,5 @@ --- -id: 163 +id: 166 title: Schema-driven data extraction (mdsmith extract) status: "πŸ”²" model: opus @@ -30,7 +30,7 @@ mirroring its nesting. No new schema concept is needed for the common case. Custom shaping is *not* in this plan. It is a separate -follow-up β€” [plan 164](164_custom-binding-overrides.md) β€” +follow-up β€” [plan 167](167_custom-binding-overrides.md) β€” and we keep it cheap by design: every key flows through one `keyFor(node)` seam (task 3), so the override plan is a focused change there plus parsing `bind:`. Until then, @@ -231,7 +231,7 @@ partial data. encoder can be added later behind the same `Format` enum. - **Custom bindings** ship in [plan - 164](164_custom-binding-overrides.md), layered on the + 167](167_custom-binding-overrides.md), layered on the `keyFor` seam; out of scope here. - **LSP / `query`-style selector** for extraction is out of scope here. diff --git a/plan/164_custom-binding-overrides.md b/plan/167_custom-binding-overrides.md similarity index 95% rename from plan/164_custom-binding-overrides.md rename to plan/167_custom-binding-overrides.md index f684b5219..c3844e191 100644 --- a/plan/164_custom-binding-overrides.md +++ b/plan/167_custom-binding-overrides.md @@ -1,19 +1,19 @@ --- -id: 164 +id: 167 title: Custom binding overrides for mdsmith extract status: "πŸ”²" model: opus -depends-on: [163] +depends-on: [166] summary: >- Add an opt-in `bind:` key that overrides the default schema-derived key in `mdsmith extract`, layered on - the `keyFor` seam from plan 163. + the `keyFor` seam from plan 166. --- # Custom binding overrides for mdsmith extract ## Goal -[Plan 163](163_schema-driven-data-extraction.md) derives +[Plan 166](166_schema-driven-data-extraction.md) derives the extracted data tree from the schema hierarchy with no annotations. This plan adds an opt-in `bind:` key that renames or restructures a node when the default key is From 46af20348160945a0d30c7a2d287060fa2d290f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:22:05 +0000 Subject: [PATCH 09/12] plan 167: fix remaining stale "plan 163" refs -> 166; plan 166 depends-on Renumbering leftovers: three "plan 163" references in plan 167 (keyFor seam origin, collision diagnostic, acceptance criterion) now point to the renumbered extraction plan 166. Drop ambiguous id 156 from plan 166's depends-on (two plan files share id 156); the composition dependency stays expressed as a filename link in the Sequencing section. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/166_schema-driven-data-extraction.md | 11 ++++++----- plan/167_custom-binding-overrides.md | 6 +++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/plan/166_schema-driven-data-extraction.md b/plan/166_schema-driven-data-extraction.md index dfc6ff9de..c68dae419 100644 --- a/plan/166_schema-driven-data-extraction.md +++ b/plan/166_schema-driven-data-extraction.md @@ -3,7 +3,7 @@ id: 166 title: Schema-driven data extraction (mdsmith extract) status: "πŸ”²" model: opus -depends-on: [149, 156] +depends-on: [149] summary: >- Derive a default data tree from the hierarchical schema and add an `extract` subcommand that emits a @@ -97,10 +97,11 @@ legacy single-source model. target this shape directly. - **[Plan 156 β€” kind-schema composition](156_kind-schema-composition.md) / PR - #288.** (Disambiguation: two plan files share id 156; - this dependency is the composition one, not the - now-landed `156_schema-entry-unification`.) A file can - resolve to multiple kinds whose schemas compose via + #288.** (Two plan files share id 156, so this + dependency is named by filename here rather than in + numeric `depends-on:`: it is the composition one, not + the now-landed `156_schema-entry-unification`.) A file + can resolve to multiple kinds whose schemas compose via `schema.Compose()`. The extractor consumes the composed `Schema`. Default keys derive from heading text, so identical headings from two kinds merge to the same key diff --git a/plan/167_custom-binding-overrides.md b/plan/167_custom-binding-overrides.md index c3844e191..82fe3193d 100644 --- a/plan/167_custom-binding-overrides.md +++ b/plan/167_custom-binding-overrides.md @@ -21,7 +21,7 @@ wrong, without changing the default behavior. ## Why this is a small change -Plan 163 routes every key through one `keyFor(node)` +Plan 166 routes every key through one `keyFor(node)` function. This plan only changes that function and adds parsing. The walk, encoders, and CLI are untouched. @@ -32,7 +32,7 @@ parsing. The walk, encoders, and CLI are untouched. children merge into the parent instead of nesting. - Composition rule: two kinds binding one composed node to different names is a compose-time error, reusing the - collision diagnostic from plan 163. + collision diagnostic from plan 166. ## Tasks @@ -55,7 +55,7 @@ parsing. The walk, encoders, and CLI are untouched. ## Acceptance Criteria - [ ] `bind:` overrides the default key; output is - otherwise identical to plan 163. + otherwise identical to plan 166. - [ ] `bind: ""` hoists a node's children into its parent. - [ ] Duplicate or unreachable binds are rejected with From ecdcf13c35b3682f4f015e653952525045dcf46a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:30:48 +0000 Subject: [PATCH 10/12] plan 165: export checks staleness by default, never auto-fixes Auto-regenerating directive bodies on export is surprising and masks drift. Default mode now fails on a stale body (exit non-zero, no output); --fix opts into in-memory regeneration; --no-check skips the check. The two flags are mutually exclusive. Tasks, acceptance criteria, and decisions updated. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/165_portable-markdown-export.md | 93 +++++++++++++++++++++------- 1 file changed, 70 insertions(+), 23 deletions(-) diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md index 839d40203..8e828e2f8 100644 --- a/plan/165_portable-markdown-export.md +++ b/plan/165_portable-markdown-export.md @@ -33,14 +33,37 @@ plain-document transform onto the schema-projection command. A dedicated `export` keeps the two concerns apart and leaves room to grow (output path, later batch). +## Staleness: check by default, never auto-fix + +`export` does **not** silently regenerate directive +bodies. Auto-fixing on export is surprising and would +mask drift between a directive and its rendered body. +The default is to *check*, not to *fix*: + +- **Default (check).** Before stripping, verify each + directive body equals what the engine would generate. + If any body is stale, export writes nothing and exits + non-zero with a diagnostic naming the stale directive + and advising `mdsmith fix` or `--fix`. The export is + faithful β€” it never papers over drift. +- **`--no-check`.** Skip the staleness check and export + bodies exactly as they appear in the file. For callers + who know the file is fresh or deliberately want the + on-disk bytes. +- **`--fix`.** Regenerate stale bodies in memory (same + engine as `mdsmith fix`) before stripping. Opt-in + convenience for a one-shot fresh export. + +`--fix` and `--no-check` are mutually exclusive (one +regenerates, the other trusts as-is); passing both is a +usage error. In every mode the source file is never +modified. + ## Behavior -- Regenerate directive bodies in memory first, reusing - the same engine as `mdsmith fix`, so output is never - stale. The source file is never modified. - Drop the opening and closing marker lines of every directive region; keep the body text between them - verbatim. + verbatim (regenerated first only under `--fix`). - `` bodies are already expanded by regeneration, so keeping the body inlines the included content (recursively). @@ -60,12 +83,11 @@ apart and leaves room to grow (output path, later batch). ## Tasks 1. **Export core (red/green).** Add `internal/export` - with `Export(f *lint.File) ([]byte, error)`: - regenerate directive bodies in memory via the `fix` - directive engine, then remove marker lines while - keeping bodies. Unit-test marker removal, body - retention, include inlining, and the no-directive - no-op. + with `Export(f *lint.File) ([]byte, error)`: remove + marker lines while keeping the on-disk body bytes + verbatim β€” no regeneration. Unit-test marker removal, + body retention, include-body inlining, and the + no-directive no-op. 2. **Nested / literal-content markers.** Drive removal off the engine's own marker-pair detection β€” `gensection.FindMarkerPairs` in @@ -82,19 +104,34 @@ apart and leaves room to grow (output path, later batch). lines left by removed markers so output is stable and passes `mdsmith check`. Test idempotence: export of export equals export. -4. **`export` subcommand.** Register `export` in +4. **Staleness check and modes.** Add a checker that + compares each directive's on-disk body to what the + engine would generate, reusing the `mdsmith fix` + directive engine. Default: a stale body makes + `Export` return a diagnostic (naming the directive) + and no output. `--fix`: regenerate stale bodies in + memory before stripping. `--no-check`: skip the + check entirely. The two flags are mutually exclusive. + Unit-test all three modes on a stale fixture. +5. **`export` subcommand.** Register `export` in [main.go](../cmd/mdsmith/main.go); `mdsmith export ` writes to stdout, `-o/--output ` writes - a file. Never mutate the source. Reuse the config and - file-load helpers that back `fix` in - [main.go](../cmd/mdsmith/main.go). Exit non-zero with - a clear message on parse errors. -5. **Fixtures and integration test.** Add `testdata` + a file, `--fix` and `--no-check` select the staleness + mode (rejecting the combination). Never mutate the + source. Reuse the config and file-load helpers that + back `fix` in [main.go](../cmd/mdsmith/main.go). Exit + non-zero with a clear message on parse errors and on a + stale body in the default mode. +6. **Fixtures and integration test.** Add `testdata` inputs covering include, catalog, toc, and build - directives with golden directive-free outputs. Assert - idempotence and that the output passes `mdsmith + directives with golden directive-free outputs. Add a + stale-body fixture: assert default mode exits non-zero + with no output, `--fix` produces the fresh golden, and + `--no-check` exports the stale bytes as-is. Assert + idempotence and that fresh output passes `mdsmith check`. -6. **Docs.** Add `docs/reference/cli/export.md` and link +7. **Docs.** Add `docs/reference/cli/export.md` (covering + the default check, `--fix`, and `--no-check`) and link it from the CLI reference catalog. Run `mdsmith fix` so catalogs and PLAN.md regenerate. @@ -105,12 +142,17 @@ apart and leaves room to grow (output path, later batch). marker, keeps generated bodies, and inlines `` content. Marker-like text treated as literal content is left in place. -- [ ] The source file is never modified. -- [ ] Stale directive bodies are regenerated before - stripping, so the output is never stale. +- [ ] The source file is never modified in any mode. +- [ ] Default mode: a stale directive body makes + `export` exit non-zero with a diagnostic naming the + directive and writes no output. +- [ ] `--fix` regenerates stale bodies in memory before + stripping; `--no-check` exports on-disk bytes as-is; + passing both is a usage error. - [ ] Nested same-type literal-content markers are preserved. -- [ ] Output is idempotent and passes `mdsmith check`. +- [ ] Output is idempotent and (when fresh) passes + `mdsmith check`. - [ ] `-o ` writes to a file; stdout is the default. - [ ] A parse error or missing file exits non-zero with @@ -128,6 +170,11 @@ apart and leaves room to grow (output path, later batch). format and not a `fix` flag; a dedicated command keeps the source-to-source transform separate from schema extraction. +- **Check by default, never auto-fix.** A stale body + fails the export rather than being silently + regenerated, so the output faithfully reflects the + file. `--fix` opts into regeneration; `--no-check` + opts out of the check. - **Front matter retained.** It is not a directive; stripping it is out of scope. - **Single file first.** Directory or glob batch export From beaa9ffc912c2d51279c865c9bcb7eec1cf8972f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:33:47 +0000 Subject: [PATCH 11/12] =?UTF-8?q?plan=20165:=20align=20Export=20signature?= =?UTF-8?q?=20=E2=80=94=20([]byte,=20[]lint.Diagnostic)=20+=20Mode?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tasks 1 and 4 disagreed on the return shape. Settle on Export(f, mode) ([]byte, []lint.Diagnostic), mirroring plan 166's Extract: a non-empty diagnostic slice means refusal (nil bytes); a stale body in Check mode appends a directive-positioned diagnostic; hard error is reserved for I/O only. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/165_portable-markdown-export.md | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md index 8e828e2f8..cb9364c3e 100644 --- a/plan/165_portable-markdown-export.md +++ b/plan/165_portable-markdown-export.md @@ -83,9 +83,15 @@ modified. ## Tasks 1. **Export core (red/green).** Add `internal/export` - with `Export(f *lint.File) ([]byte, error)`: remove - marker lines while keeping the on-disk body bytes - verbatim β€” no regeneration. Unit-test marker removal, + with `Export(f *lint.File, mode Mode) ([]byte, + []lint.Diagnostic)` β€” mirroring plan 166's `Extract` + signature. It removes marker lines while keeping the + on-disk body bytes verbatim β€” no regeneration. A + non-empty diagnostic slice means refusal: bytes are + `nil` and the caller exits non-zero (a `nil` slice and + bytes is success). `Mode` is the staleness mode from + task 4. Reserve a hard `error` only for I/O failures, + not document-level problems. Unit-test marker removal, body retention, include-body inlining, and the no-directive no-op. 2. **Nested / literal-content markers.** Drive removal @@ -107,12 +113,15 @@ modified. 4. **Staleness check and modes.** Add a checker that compares each directive's on-disk body to what the engine would generate, reusing the `mdsmith fix` - directive engine. Default: a stale body makes - `Export` return a diagnostic (naming the directive) - and no output. `--fix`: regenerate stale bodies in - memory before stripping. `--no-check`: skip the - check entirely. The two flags are mutually exclusive. - Unit-test all three modes on a stale fixture. + directive engine. `Mode` is `Check` (default), `Fix`, + or `NoCheck`. In `Check`, each stale body appends one + `lint.Diagnostic` (naming the directive, positioned at + its start marker) and `Export` returns `nil` bytes. + `Fix` regenerates stale bodies in memory before + stripping. `NoCheck` skips the comparison. The CLI + maps `--fix`/`--no-check` to the mode and rejects the + combination. Unit-test all three modes on a stale + fixture. 5. **`export` subcommand.** Register `export` in [main.go](../cmd/mdsmith/main.go); `mdsmith export ` writes to stdout, `-o/--output ` writes From c0b5d689bb189ff35f3c6671751423c0687325ce Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 16 May 2026 10:37:19 +0000 Subject: [PATCH 12/12] =?UTF-8?q?plan=20165:=20clarify=20Export=20contract?= =?UTF-8?q?=20=E2=80=94=20no=20error=20return,=20exactly=20one=20value=20s?= =?UTF-8?q?et?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Export operates on an in-memory *lint.File, so it does no I/O and returns no error; reads/writes are the CLI layer's job (real error there). Spell out the success vs refusal contract: success = non-nil bytes + nil diagnostics; refusal = nil bytes + non-empty diagnostics. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR --- plan/165_portable-markdown-export.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md index cb9364c3e..ff52f18f5 100644 --- a/plan/165_portable-markdown-export.md +++ b/plan/165_portable-markdown-export.md @@ -85,15 +85,20 @@ modified. 1. **Export core (red/green).** Add `internal/export` with `Export(f *lint.File, mode Mode) ([]byte, []lint.Diagnostic)` β€” mirroring plan 166's `Extract` - signature. It removes marker lines while keeping the - on-disk body bytes verbatim β€” no regeneration. A - non-empty diagnostic slice means refusal: bytes are - `nil` and the caller exits non-zero (a `nil` slice and - bytes is success). `Mode` is the staleness mode from - task 4. Reserve a hard `error` only for I/O failures, - not document-level problems. Unit-test marker removal, - body retention, include-body inlining, and the - no-directive no-op. + signature. It operates purely on the already-parsed + in-memory `*lint.File`, so it performs no I/O and + returns no `error`; file reads and `-o` writes are the + CLI layer's job (task 5) and surface as a real `error` + there. Contract: exactly one of the two return values + is populated. **Success** β†’ the exported bytes (which + are never `nil`, since a directive-free file still + yields its own content) and a `nil` diagnostic slice. + **Refusal** (stale body in `Check` mode, or any + document-level problem) β†’ `nil` bytes and a non-empty + diagnostic slice; the caller exits non-zero. `Mode` is + the staleness mode from task 4. Unit-test marker + removal, body retention, include-body inlining, and + the no-directive no-op. 2. **Nested / literal-content markers.** Drive removal off the engine's own marker-pair detection β€” `gensection.FindMarkerPairs` in