Plans 165–167: portable export + schema-driven extraction - #292
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Plan 163 to the project roadmap, describing a schema-driven extraction feature (mdsmith extract) that would project validated Markdown (kinds + schemas) into structured data outputs (JSON/YAML/Lua/msgpack).
Changes:
- Added
plan/163_schema-driven-data-extraction.mddescribing the newbind:concept and the proposed extractor + CLI design. - Updated
PLAN.mdto include Plan 163 in the catalog table.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| plan/163_schema-driven-data-extraction.md | New plan document defining the bind: projection concept, extraction mapping rules, and implementation task breakdown. |
| PLAN.md | Catalog table now includes the new Plan 163 entry. |
jeduden
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
plan/163_schema-driven-data-extraction.md:136
- Task 4 refers to projecting
repeats: truescopes, butrepeatsis an old key name and is currently rejected in the schema parser; repeating is expressed viarepeat: { ... }. Please update the task text to use the current repeat-cardinality terminology so it matches the schema model the extractor will consume.
4. **Repeating scopes and placeholders.** Project
`repeats: true` scopes as arrays; each element retains
every captured `{field}` as a `name: value` field,
Introduces a `bind:` projection layer on schema scopes and content entries plus an `extract` subcommand that turns a kind-conformant Markdown file into JSON/YAML/Lua/msgpack. Sequenced on top of the schema-composition rework (plan 156) and content-schema work (plan 149). https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
… 164 - Default projection derives the data tree from the schema hierarchy; no annotations required. - Custom bindings move to follow-up plan 164, layered on a single keyFor() seam. - Address Copilot review: disambiguate the duplicate plan-156 id via explicit file link; name the concrete schema.MatchTree type returned by an extended schema.Validate. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
Addresses Copilot review: a plain string cannot represent "unset" vs `bind: ""` (hoist), so Bind is now *string. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
- Repeating-scope elements always retain each placeholder as a name:value field. - Root holds `frontmatter` plus projected sections beside it (no nesting under a single key). - Preamble projected under `preamble`; wildcard/unlisted skipped. - Defer Lua; ship json/yaml/msgpack. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
The schema parser rejects the legacy `repeats` key;
repeating cardinality is `repeat: {min, max}`.
https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
A source-to-source `export` subcommand that strips directive markers, keeps generated bodies, and inlines includes — distinct from schema extraction (163/164). https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
…eamble key Plan 156 entry-unification landed in main (PR #295): the standalone preamble: key is gone, replaced by heading: null. Project the no-heading section's content into the enclosing object instead of a preamble wrapper key. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
jeduden
force-pushed
the
claude/markdown-schema-conversion-socJI
branch
from
May 16, 2026 10:11
64eb45d to
e199eb2
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
plan/165_portable-markdown-export.md:101
- Acceptance criteria currently say “all directive markers removed” while also requiring “Nested same-type literal-content markers are preserved.” Since the preserved markers still look like
<?…?>lines, this is internally inconsistent. Consider tightening the wording to distinguish between (a) directive markers that the parser/engine recognizes as real directive start/end markers (which should be stripped) vs. (b) marker-like text treated as literal content (which should remain).
- [ ] `mdsmith export <file>` emits the file with all
directive markers removed and generated bodies
kept; `<?include?>` content is inlined.
- [ ] The source file is never modified.
- [ ] Stale directive bodies are regenerated before
stripping, so the output is never stale.
- [ ] Nested same-type literal-content markers are
preserved.
Rebasing onto main pulled in plan/163_public-markdown-library and plan/164_github-ui-releases-and-split-website, colliding with the new plans. Renumber schema-extraction 163->166 and custom-bindings 164->167 (export stays 165, no collision); update cross-references and depends-on. Plan 165: marker stripping is driven by gensection.FindMarkerPairs (MarkerPair.StartLine/EndLine), not lint.File.GeneratedRanges (which only records include/catalog body ranges). Tighten the acceptance criteria to distinguish engine-recognized markers from literal-content marker-like text. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
plan/167_custom-binding-overrides.md:35
- This bullet references “collision diagnostic from plan 163”, but the collision diagnostic being extended/relied on is part of the extraction work in Plan 166. Keeping plan IDs consistent here matters because plan 163 is a different feature area (public markdown library).
- Composition rule: two kinds binding one composed node
to different names is a compose-time error, reusing the
collision diagnostic from plan 163.
plan/167_custom-binding-overrides.md:59
- Acceptance criteria still say “otherwise identical to plan 163”, but the default-binding baseline for this plan is Plan 166. This is likely a renumbering leftover and should be corrected to avoid ambiguity during implementation/review.
- [ ] `bind:` overrides the default key; output is
otherwise identical to plan 163.
- [ ] `bind: ""` hoists a node's children into its
…s-on Renumbering leftovers: three "plan 163" references in plan 167 (keyFor seam origin, collision diagnostic, acceptance criterion) now point to the renumbered extraction plan 166. Drop ambiguous id 156 from plan 166's depends-on (two plan files share id 156); the composition dependency stays expressed as a filename link in the Sequencing section. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
jeduden
changed the title
Auto-regenerating directive bodies on export is surprising and masks drift. Default mode now fails on a stale body (exit non-zero, no output); --fix opts into in-memory regeneration; --no-check skips the check. The two flags are mutually exclusive. Tasks, acceptance criteria, and decisions updated. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
Tasks 1 and 4 disagreed on the return shape. Settle on Export(f, mode) ([]byte, []lint.Diagnostic), mirroring plan 166's Extract: a non-empty diagnostic slice means refusal (nil bytes); a stale body in Check mode appends a directive-positioned diagnostic; hard error is reserved for I/O only. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
…e set Export operates on an in-memory *lint.File, so it does no I/O and returns no error; reads/writes are the CLI layer's job (real error there). Spell out the success vs refusal contract: success = non-nil bytes + nil diagnostics; refusal = nil bytes + non-empty diagnostics. https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A small family of roadmap plans for getting data and portable Markdown out of mdsmith documents. (Plan IDs are 165–167 — 163/164 are used by other plans that landed on
main, so these were renumbered during a rebase.)Changes
plan/165_portable-markdown-export.md— amdsmith export <file>subcommand: a source-to-source transform that strips<?…?>directive markers, keeps generated bodies, and inlines<?include?>to produce portable, directive-free Markdown. Deliberately not a fourthextractformat — it needs no schema/kind and no conformance gate. Marker stripping is driven bygensection.FindMarkerPairs(MarkerPair.StartLine/EndLine) so only engine-recognized markers are removed and literal-content marker-like text survives.plan/166_schema-driven-data-extraction.md— core feature. A default binding layer derives a data tree directly from the schema hierarchy (rootfrontmatterobject plus projected sections beside it; keys from heading slugs; scopes withrepeat: {min, max}become arrays whose elements retain each{placeholder}as aname: valuefield). Aligned with the landed plan-156 entry-shape unification: there is nopreamble:key — aheading: nullno-heading section's content projects into its enclosing object.extract <kind> --format json|yaml|msgpack <file>is gated on a passing schema match and reuses the validated*schema.MatchTree. Lua deferred.plan/167_custom-binding-overrides.md— follow-up. Opt-inbind:key (*string, so unset vs. explicit-empty differ) overrides the default key or hoists children, layered on the singlekeyFor(node)seam from plan 166.PLAN.md— catalog regenerated; every plan ID is unique.Design points
jq/yq).Schemafrom kind-schema composition (PR Implement schema composition across multiple kinds (plan 156) #288), expressed as a filename link in the Sequencing section because two plan files shareid: 156.This PR is roadmap/docs only — no behavior change yet.
https://claude.ai/code/session_01Ar54BuJr8fFB9KzJGvLvYR