Extract Markdown parse/produce as public pkg/markdown library

[jeduden]

Completes plan #163 by extracting mdsmith's Markdown parsing and production capabilities into a public Go library at pkg/markdown.

Summary

This PR establishes pkg/markdown as mdsmith's canonical, importable Markdown surface. It moves parsing logic, processing instruction handling, and frontmatter utilities from internal/lint into the public package, making them available to external consumers while maintaining a single goldmark parser configuration across the codebase.

Key Changes

• New public package pkg/markdown: Exports the canonical goldmark parser and Markdown processing utilities

  • parser.go: NewParser() returns the shared parser instance with CommonMark + processing instruction support
  • parse.go: Parse() and Document type for parsing Markdown with frontmatter separation
  • pi.go & pi_parser.go: Processing instruction (<?...?>) AST node and parser
  • frontmatter.go: StripFrontMatter() utility for YAML frontmatter handling
  • produce.go: Edit and Splice() for Markdown mutation
  • doc.go: Package documentation and compatibility policy

• Refactored internal/lint: Now imports from pkg/markdown instead of duplicating logic

  • pi.go, pi_parser.go, frontmatter.go reduced to thin wrappers or removed
  • file.go and lint_coverage_test.go updated to use public APIs
  • pi_test.go moved to pkg/markdown/pi_test.go with expanded coverage

• Updated internal consumers: internal/release/syncdocs.go and internal/rules/markdownflavor/parser.go now import from pkg/markdown

• Documentation: Added docs/development/markdown-library.md describing the public API, compatibility policy, and usage

• Architecture updates: Updated architecture docs to reflect pkg/markdown as the single Markdown surface boundary

• Test infrastructure: Added golden-file test for syncdocs reconciliation and comprehensive unit tests for all public APIs

• Plan status: Marked plan Clean up completed and won't-fix plans #163 as complete (✅)

Implementation Details

• Parser is cached via sync.Once to ensure a single instance across the application
• Processing instructions are registered as a custom goldmark block parser
• Frontmatter stripping is idempotent and handles edge cases (empty files, missing delimiters)
• All public APIs include unit tests with 100% coverage expectations
• Backward compatibility maintained: internal code continues to work via public imports

https://claude.ai/code/session_01RBQSS4X7e6mfrkyXmDzGRX

[Copilot]

Pull request overview

This PR extracts mdsmith's Markdown parsing and producing logic from internal/lint into a new public Go package pkg/markdown. The new package owns the single canonical goldmark parser configuration (CommonMark + <?…?> processing-instruction block parser), the Parse/ParseContext entry points with frontmatter splitting, the Splice byte-exact producer, and the ProcessingInstruction AST node. internal/lint now re-exports these via type aliases and forwarders, internal/release/syncdocs.go migrates off its private goldmark.New() parser onto the shared one, and internal/rules/markdownflavor/parser.go switches to the public PI block-parser registration. A new golden corpus test pins reconcileDocForHugo output byte-for-byte across docs/** plus synthetic edge cases. Plan 163 is marked ✅.

Changes:

• New pkg/markdown package with Parse, ParseContext, NewParser, StripFrontMatter, CountLines, Splice/Edit, and ProcessingInstruction plus comprehensive unit tests, package docs, and a parser pool.
• internal/lint and internal/release/syncdocs.go refactored to consume pkg/markdown; lint keeps thin forwarders/type aliases for backward compatibility; PI grammar/frontmatter unit tests relocated next to the canonical implementation.
• New docs/development/markdown-library.md (compatibility policy, API surface) plus catalog/architecture/cross-system doc updates; new syncdocs_golden_test.go pins reconciliation output.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/markdown/parser.go	Canonical goldmark parser + pooled ParseContext.
pkg/markdown/parse.go	Parse + Document (frontmatter split + AST).
pkg/markdown/parse_test.go	Unit tests for Parse, ParseContext, Splice.
pkg/markdown/parser_test.go	Parser smoke + concurrent race-free pool test.
pkg/markdown/produce.go	Edit + Splice byte-exact span surgery.
pkg/markdown/pi.go	ProcessingInstruction AST node + KindProcessingInstruction.
pkg/markdown/pi_parser.go	Block parser for <?…?> PIs, with extractPINameBytes.
pkg/markdown/pi_test.go	Relocated PI grammar tests + parser/extract coverage.
pkg/markdown/frontmatter.go	StripFrontMatter + CountLines.
pkg/markdown/frontmatter_test.go	Frontmatter behavior incl. block-scalar fence regression.
pkg/markdown/doc.go	Package documentation and compatibility pointer.
internal/lint/pi.go	Reduced to type-alias forwarder over pkg/markdown.
internal/lint/pi_parser.go	Reduced to forwarders for NewPIBlockParser/PIBlockParserPrioritized.
internal/lint/pi_test.go	Removed PI grammar tests now living in pkg/markdown.
internal/lint/frontmatter.go	Forwarders to pkg/markdown for frontmatter helpers.
internal/lint/frontmatter_test.go	Removed forwarded-helper tests.
internal/lint/file.go	NewParser/parse path delegates to markdown.ParseContext.
internal/lint/lint_coverage_test.go	Removed tests now in pkg/markdown; left explanatory comments.
internal/rules/markdownflavor/parser.go	Uses markdown.PIBlockParserPrioritized instead of lint.*.
internal/rules/markdownflavor/parser_test.go	Updated import/comment to reference pkg/markdown.
internal/release/syncdocs.go	Migrated to markdown.ParseContext/Splice; matches PI block nodes.
internal/release/syncdocs_golden_test.go	New golden corpus test pinning reconcileDocForHugo output.
docs/development/markdown-library.md	New page: API surface + compatibility policy.
docs/development/architecture/{index,go,cross-system}.md	Reflect pkg/markdown as the single parse surface.
docs/development/index.md, CLAUDE.md, AGENTS.md, .github/copilot-instructions.md	Catalog/link updates for the new doc; minor wording trims.
plan/163_public-markdown-library.md, PLAN.md	Mark plan 163 ✅ and record decisions.

[Copilot]

Pull request overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 1 comment.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.98%. Comparing base (b15766b) to head (30df496).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Components	Coverage Δ
Go	95.95% <100.00%> (+0.02%)	⬆️
TypeScript	99.35% <ø> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated no new comments.