diff --git a/PLAN.md b/PLAN.md index 10ae82a12..ecf7dfa5d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -91,4 +91,7 @@ footer: | | 162 | 🔲 | sonnet | [Split the overloaded `meta` rule category](plan/162_rule-category-cleanup.md) | | 163 | 🔲 | | [Extract mdsmith Markdown parse/produce as a public Go library](plan/163_public-markdown-library.md) | | 164 | ✅ | | [GitHub-UI-triggered releases and a split website deploy](plan/164_github-ui-releases-and-split-website.md) | +| 165 | 🔲 | opus | [Portable Markdown export (mdsmith export)](plan/165_portable-markdown-export.md) | +| 166 | 🔲 | opus | [Schema-driven data extraction (mdsmith extract)](plan/166_schema-driven-data-extraction.md) | +| 167 | 🔲 | opus | [Custom binding overrides for mdsmith extract](plan/167_custom-binding-overrides.md) | diff --git a/plan/165_portable-markdown-export.md b/plan/165_portable-markdown-export.md new file mode 100644 index 000000000..ff52f18f5 --- /dev/null +++ b/plan/165_portable-markdown-export.md @@ -0,0 +1,195 @@ +--- +id: 165 +title: Portable Markdown export (mdsmith export) +status: "🔲" +model: opus +depends-on: [] +summary: >- + Add an `export` subcommand that writes a portable, + directive-free copy of a Markdown file: markers + removed, generated bodies kept, includes inlined. +--- +# Portable Markdown export (mdsmith export) + +## Goal + +`mdsmith export ` writes a portable copy of a +Markdown file with every `` directive marker +removed. Generated bodies stay as plain Markdown and +`` content is inlined. The result renders +identically on any Markdown tool with no mdsmith +knowledge. + +## Why a separate command + +This is not schema extraction. `extract` (plan 166) +projects a kind's schema into a data tree. `export` is a +source-to-source transform of the document itself. It +needs no kind, schema, or conformance gate — only that +the file parses and its directive bodies are fresh. + +Mixing it into `extract --format markdown` would couple a +plain-document transform onto the schema-projection +command. A dedicated `export` keeps the two concerns +apart and leaves room to grow (output path, later batch). + +## Staleness: check by default, never auto-fix + +`export` does **not** silently regenerate directive +bodies. Auto-fixing on export is surprising and would +mask drift between a directive and its rendered body. +The default is to *check*, not to *fix*: + +- **Default (check).** Before stripping, verify each + directive body equals what the engine would generate. + If any body is stale, export writes nothing and exits + non-zero with a diagnostic naming the stale directive + and advising `mdsmith fix` or `--fix`. The export is + faithful — it never papers over drift. +- **`--no-check`.** Skip the staleness check and export + bodies exactly as they appear in the file. For callers + who know the file is fresh or deliberately want the + on-disk bytes. +- **`--fix`.** Regenerate stale bodies in memory (same + engine as `mdsmith fix`) before stripping. Opt-in + convenience for a one-shot fresh export. + +`--fix` and `--no-check` are mutually exclusive (one +regenerates, the other trusts as-is); passing both is a +usage error. In every mode the source file is never +modified. + +## Behavior + +- Drop the opening and closing marker lines of every + directive region; keep the body text between them + verbatim (regenerated first only under `--fix`). +- `` bodies are already expanded by + regeneration, so keeping the body inlines the included + content (recursively). +- Markerless directives with no body (for example + ``, ``) are removed + outright. +- Only lines the engine's marker-pair detection + recognizes as real directive start/end markers are + removed. Marker-like text the engine treats as literal + content (for example inner same-type markers nested in + an outer directive) is left untouched. +- After stripping, normalize blank lines so the output is + stable and lint-clean. Front matter is kept as-is. +- Exporting an already directive-free file is a no-op; + `export` is idempotent. + +## Tasks + +1. **Export core (red/green).** Add `internal/export` + with `Export(f *lint.File, mode Mode) ([]byte, + []lint.Diagnostic)` — mirroring plan 166's `Extract` + signature. It operates purely on the already-parsed + in-memory `*lint.File`, so it performs no I/O and + returns no `error`; file reads and `-o` writes are the + CLI layer's job (task 5) and surface as a real `error` + there. Contract: exactly one of the two return values + is populated. **Success** → the exported bytes (which + are never `nil`, since a directive-free file still + yields its own content) and a `nil` diagnostic slice. + **Refusal** (stale body in `Check` mode, or any + document-level problem) → `nil` bytes and a non-empty + diagnostic slice; the caller exits non-zero. `Mode` is + the staleness mode from task 4. Unit-test marker + removal, body retention, include-body inlining, and + the no-directive no-op. +2. **Nested / literal-content markers.** Drive removal + off the engine's own marker-pair detection — + `gensection.FindMarkerPairs` in + [internal/archetype/gensection](../internal/archetype/gensection/parse.go), + whose `MarkerPair.StartLine`/`EndLine` give the exact + start- and end-marker line for every directive (not + just the include/catalog *body* ranges that + `lint.File.GeneratedRanges` records for diagnostic + suppression). Only lines the engine recognizes as real + markers are removed, so inner same-type markers that + the engine treats as literal content survive. Add a + test. +3. **Whitespace normalization.** Collapse the blank + lines left by removed markers so output is stable and + passes `mdsmith check`. Test idempotence: export of + export equals export. +4. **Staleness check and modes.** Add a checker that + compares each directive's on-disk body to what the + engine would generate, reusing the `mdsmith fix` + directive engine. `Mode` is `Check` (default), `Fix`, + or `NoCheck`. In `Check`, each stale body appends one + `lint.Diagnostic` (naming the directive, positioned at + its start marker) and `Export` returns `nil` bytes. + `Fix` regenerates stale bodies in memory before + stripping. `NoCheck` skips the comparison. The CLI + maps `--fix`/`--no-check` to the mode and rejects the + combination. Unit-test all three modes on a stale + fixture. +5. **`export` subcommand.** Register `export` in + [main.go](../cmd/mdsmith/main.go); `mdsmith export + ` writes to stdout, `-o/--output ` writes + a file, `--fix` and `--no-check` select the staleness + mode (rejecting the combination). Never mutate the + source. Reuse the config and file-load helpers that + back `fix` in [main.go](../cmd/mdsmith/main.go). Exit + non-zero with a clear message on parse errors and on a + stale body in the default mode. +6. **Fixtures and integration test.** Add `testdata` + inputs covering include, catalog, toc, and build + directives with golden directive-free outputs. Add a + stale-body fixture: assert default mode exits non-zero + with no output, `--fix` produces the fresh golden, and + `--no-check` exports the stale bytes as-is. Assert + idempotence and that fresh output passes `mdsmith + check`. +7. **Docs.** Add `docs/reference/cli/export.md` (covering + the default check, `--fix`, and `--no-check`) and link + it from the CLI reference catalog. Run `mdsmith fix` + so catalogs and PLAN.md regenerate. + +## Acceptance Criteria + +- [ ] `mdsmith export ` removes every line the + engine recognizes as a real directive start/end + marker, keeps generated bodies, and inlines + `` content. Marker-like text treated as + literal content is left in place. +- [ ] The source file is never modified in any mode. +- [ ] Default mode: a stale directive body makes + `export` exit non-zero with a diagnostic naming the + directive and writes no output. +- [ ] `--fix` regenerates stale bodies in memory before + stripping; `--no-check` exports on-disk bytes as-is; + passing both is a usage error. +- [ ] Nested same-type literal-content markers are + preserved. +- [ ] Output is idempotent and (when fresh) passes + `mdsmith check`. +- [ ] `-o ` writes to a file; stdout is the + default. +- [ ] A parse error or missing file exits non-zero with + a clear message. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes + +## Decisions + +- **Keep generated bodies.** Markers are stripped but + TOC, catalog, and included content stay as plain + Markdown; includes are inlined for a portable copy. +- **New `export` subcommand.** Not a fourth `extract` + format and not a `fix` flag; a dedicated command keeps + the source-to-source transform separate from schema + extraction. +- **Check by default, never auto-fix.** A stale body + fails the export rather than being silently + regenerated, so the output faithfully reflects the + file. `--fix` opts into regeneration; `--no-check` + opts out of the check. +- **Front matter retained.** It is not a directive; + stripping it is out of scope. +- **Single file first.** Directory or glob batch export + is a possible follow-up, not in this plan. diff --git a/plan/166_schema-driven-data-extraction.md b/plan/166_schema-driven-data-extraction.md new file mode 100644 index 000000000..c68dae419 --- /dev/null +++ b/plan/166_schema-driven-data-extraction.md @@ -0,0 +1,238 @@ +--- +id: 166 +title: Schema-driven data extraction (mdsmith extract) +status: "🔲" +model: opus +depends-on: [149] +summary: >- + Derive a default data tree from the hierarchical + schema and add an `extract` subcommand that emits a + kind-conformant file as JSON/YAML/msgpack. +--- +# Schema-driven data extraction (mdsmith extract) + +## Goal + +Let a kind's schema double as an extraction contract. +Once `mdsmith check` confirms a file conforms, `mdsmith +extract --format json|yaml|msgpack ` emits a +data tree. Its shape is derived from the schema hierarchy +itself — no annotations required. + +## Why a default binding layer first + +The schema is already a hierarchy: front matter, then a +tree of scopes (sections), each with child scopes and +content entries. That hierarchy *is* the data shape. So +the first deliverable is a **default binding layer** that +projects the schema tree into a data tree directly, +mirroring its nesting. No new schema concept is needed +for the common case. + +Custom shaping is *not* in this plan. It is a separate +follow-up — [plan 167](167_custom-binding-overrides.md) — +and we keep it cheap by design: every key flows through +one `keyFor(node)` seam (task 3), so the override plan is +a focused change there plus parsing `bind:`. Until then, +renaming or restructuring is the job of a downstream tool +(`jq`, `yq`) over the standard-format output. + +## Default projection rules + +The projection walks the composed schema in lockstep with +the validated match and mirrors the hierarchy: + +- **Root shape.** The root object holds a `frontmatter` + object (the decoded front matter, unchanged) *and* the + projected sections beside it at the same level. Front + matter stays grouped so it never collides with a + section slug. +- **Literal-heading scope** (`## Goal`) → object keyed by + the slugified heading (`goal`), reusing the existing + anchor slugifier. Its value holds child scopes and + content, recursively. +- **Repeating scope** (`## {id}` with a `repeat: {min, + max}` cardinality) → an array keyed by the slug of the + heading's literal stem, + or the placeholder name if the heading is only a + placeholder. Each element is an object that **always + retains every captured placeholder as a `name: value` + field** (both the placeholder name and its value + survive), plus the element's own child scopes and + content. +- **No-heading section** (`heading: null` — content + before the first child heading) has no heading text and + therefore no slug. Its content entries project + **directly into the enclosing object** (root, or the + parent section) beside the headed-section keys — there + is no `preamble` wrapper key. Wildcard slots + (`regex: '.+'`) and unlisted/closed headings are + skipped: the output is a faithful projection of the + *declared* schema only. +- **`code-block`** → string under `code` (raw body); + multiple blocks get `code`, `code-2`, … +- **`list`** → array of item strings under `items`. +- **`table` with `columns`** → array of row objects keyed + by column header, under `rows`. +- **`paragraph`** → its text under `text`. + +Sibling key collisions (two `## Goal` headings, or a +content default that shadows a child scope slug) are a +schema error reported at extract time, pointing at the +schema source. Empty/optional sections that did not match +are omitted rather than emitted as null. + +## Sequencing + +This plan consumes the reworked schema engine, not the +legacy single-source model. + +- **Entry-shape unification (`156_schema-entry-unification` + / PR #295) — landed in main.** Every `sections:` entry + is discriminated by its `heading:` value: a string or + `{regex, repeat?, sequential?}` mapping for headed + sections, and `heading: null` for the no-heading section + (content before the first child heading). There is no + standalone `preamble:` key. The projection rules above + target this shape directly. +- **[Plan 156 — kind-schema + composition](156_kind-schema-composition.md) / PR + #288.** (Two plan files share id 156, so this + dependency is named by filename here rather than in + numeric `depends-on:`: it is the composition one, not + the now-landed `156_schema-entry-unification`.) A file + can resolve to multiple kinds whose schemas compose via + `schema.Compose()`. The extractor consumes the composed + `Schema`. Default keys derive from heading text, so + identical headings from two kinds merge to the same key + with no conflict; only genuinely divergent shapes + surface as a collision. +- **Plan 149 (section-content schema).** Content + projection rides on the `ContentEntry` model from the + content-schema work. This plan adds no content matcher + of its own and is blocked until that model is stable. +- **Plan 147 / PR #284 (actionable schema diagnostics).** + If landed, collision and conformance failures reuse the + `SchemaDiagnostic` formatter. + +Extraction is gated on a successful schema match. A +non-conformant file makes `extract` report the same +diagnostics as `check` and exit non-zero. It never emits +partial data. + +## Tasks + +1. **Expose the match tree.** Refactor `schema.Validate` + (and the content matcher) to also return a new + `*schema.MatchTree` in `internal/schema`: for each + `Scope` / `ContentEntry`, the matched AST nodes, their + source lines, and captured `{field}` values. `Validate` + keeps its diagnostic return; the tree is an added + result so MDS020 is unaffected. Unit-test the tree on + the existing schema fixtures. +2. **Extractor skeleton (red/green).** Add + `internal/extract` with `Extract(f *lint.File, sch + *schema.Schema, m *schema.MatchTree) (any, + []lint.Diagnostic)`. `sch` is the composed schema; `m` + is the tree from task 1 — no re-matching. +3. **Default scope projection.** Walk the scope tree and + build the nested structure per the rules above: + `frontmatter` plus sections at the root, literal scopes + keyed by slug, the `heading: null` no-heading section's + content hoisted into the enclosing object, wildcard / + unlisted skipped. Route every key through one + `keyFor(node)` function — the single seam a future + custom-binding plan overrides. Reuse the existing + anchor slugifier. Unit-test literal, nested, + no-heading-section, and optional-omitted scopes. +4. **Repeating scopes and placeholders.** Project scopes + with a `repeat: {min, max}` cardinality as arrays; each + element retains + every captured `{field}` as a `name: value` field, + reusing + [fieldinterp](../internal/fieldinterp/fieldinterp.go). +5. **Default content projection.** Project `code-block`, + `list`, `table`, and `paragraph` entries (plan 149) + with their default keys. Detect sibling key collisions + and emit a schema diagnostic. +6. **Composition behavior.** Add `compose_test.go` / + extractor tests proving a file under two kinds yields a + merged tree, and that a real shape divergence is + reported as a collision, not silently dropped. +7. **Format encoders.** Add `internal/extract/encode` + with json (stdlib), yaml (existing dep), and msgpack + encoders behind a `Format` enum. (Lua is deferred.) +8. **`extract` subcommand.** Register `extract` in + [main.go](../cmd/mdsmith/main.go); signature `mdsmith + extract --format `. Reuse the + config-load and kind-resolution helpers from + [kinds.go](../cmd/mdsmith/kinds.go). Validate that + `` is one of the file's resolved kinds. Run + schema validation first and abort on failure. +9. **Fixtures and integration test.** Add a kind with a + schema under `testdata/`, a conformant sample, and + golden outputs per format. Assert non-conformant input + exits non-zero with check diagnostics. +10. **Docs.** Add a section under + [schemas.md](../docs/guides/schemas.md) and a + `docs/reference/cli/extract.md` page. Both are picked + up by existing catalog directives. Run `mdsmith fix` + so catalogs and PLAN.md regenerate. + +## Acceptance Criteria + +- [ ] `mdsmith extract --format json ` on a + conformant file emits a tree whose nesting mirrors + the schema hierarchy — no schema annotations + required. +- [ ] The root holds a `frontmatter` object and the + projected sections beside it at the same level. +- [ ] Literal headings key by slug; repeating sections + become arrays; each element retains every captured + placeholder as a `name: value` field plus its child + scopes/content. +- [ ] A `heading: null` no-heading section's content + projects into its enclosing object (no `preamble` + wrapper key); wildcard and unlisted/closed headings + are skipped. +- [ ] Code-block, list, table, and paragraph entries + project under their default keys; sibling key + collisions are reported as schema diagnostics. +- [ ] A file resolving to multiple kinds yields a merged + tree; a genuine shape divergence is reported, not + silently dropped. +- [ ] `json`, `yaml`, and `msgpack` produce equivalent + data; golden fixtures cover all three formats. +- [ ] A non-conformant file makes `extract` exit non-zero + and print the same diagnostics as `mdsmith check`. +- [ ] An unknown kind, or a kind not assigned to the + file, exits non-zero with a clear message. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes + +## Decisions + +- **Repeating-scope key.** Array key is the slug of the + heading's literal stem, or the placeholder name when + the heading is only a placeholder. Each element always + retains every captured placeholder as a `name: value` + field, so both the name and the value survive. +- **Front matter placement.** The root holds a + `frontmatter` object and the projected sections beside + it at the same level. Grouping front matter avoids + collisions with section slugs. +- **No-heading section.** A `heading: null` entry has no + slug; its content projects directly into the enclosing + object rather than under a `preamble` wrapper key. The + sibling-collision rule covers any clash with a section + slug. Wildcard slots and unlisted/closed headings are + skipped. +- **Lua deferred.** Ship json, yaml, and msgpack. A Lua + encoder can be added later behind the same `Format` + enum. +- **Custom bindings** ship in [plan + 167](167_custom-binding-overrides.md), layered on the + `keyFor` seam; out of scope here. +- **LSP / `query`-style selector** for extraction is out + of scope here. diff --git a/plan/167_custom-binding-overrides.md b/plan/167_custom-binding-overrides.md new file mode 100644 index 000000000..82fe3193d --- /dev/null +++ b/plan/167_custom-binding-overrides.md @@ -0,0 +1,67 @@ +--- +id: 167 +title: Custom binding overrides for mdsmith extract +status: "🔲" +model: opus +depends-on: [166] +summary: >- + Add an opt-in `bind:` key that overrides the default + schema-derived key in `mdsmith extract`, layered on + the `keyFor` seam from plan 166. +--- +# Custom binding overrides for mdsmith extract + +## Goal + +[Plan 166](166_schema-driven-data-extraction.md) derives +the extracted data tree from the schema hierarchy with no +annotations. This plan adds an opt-in `bind:` key that +renames or restructures a node when the default key is +wrong, without changing the default behavior. + +## Why this is a small change + +Plan 166 routes every key through one `keyFor(node)` +function. This plan only changes that function and adds +parsing. The walk, encoders, and CLI are untouched. + +- **`keyFor(node)`** returns the bind value when present, + else the default slug. `Bind` is a `*string` so an + unset key and an explicit empty one are distinct. +- A node with `bind: ""` (present, empty) is hoisted: its + children merge into the parent instead of nesting. +- Composition rule: two kinds binding one composed node + to different names is a compose-time error, reusing the + collision diagnostic from plan 166. + +## Tasks + +1. **Parse `bind:`.** Add `Bind *string` to `Scope` and + `ContentEntry` (nil = unset, non-nil = present, so + `bind: ""` is distinguishable); parse in + `parse_inline.go` and `parse_file.go`. Unit-test + round-trip including unset vs. explicit-empty. +2. **Override `keyFor`.** Return the bind value when + present; implement hoist for `bind: ""`. +3. **Validate binds.** Reject duplicate sibling binds and + unreachable binds via the schema diagnostic path. +4. **Compose binds.** Extend `schema.Compose()` so merged + headings union bound children; conflicting names are a + compose-time error. +5. **Fixtures and docs.** Add bind-override golden cases; + document `bind:` under + [schemas.md](../docs/guides/schemas.md). + +## Acceptance Criteria + +- [ ] `bind:` overrides the default key; output is + otherwise identical to plan 166. +- [ ] `bind: ""` hoists a node's children into its + parent. +- [ ] Duplicate or unreachable binds are rejected with + actionable diagnostics. +- [ ] Conflicting binds across composed kinds are a + compose-time error. +- [ ] All tests pass: `go test ./...` +- [ ] `go tool golangci-lint run` reports no issues +- [ ] `mdsmith check .` passes