diff --git a/PLAN.md b/PLAN.md index df7b03fbe..21ce66ca3 100644 --- a/PLAN.md +++ b/PLAN.md @@ -14,39 +14,17 @@ footer: | ?> -| ID | Status | Title | -|-----|--------|-----------------------------------------------------------------------------------------------------------------| -| 50 | ✅ | [Redundancy / Duplication Detection](plan/50_redundancy-duplication-detection.md) | -| 51 | ✅ | [Section-Level Size Limits](plan/51_section-level-size-limits.md) | -| 52 | ✅ | [Archetype / Template Library for Agentic Patterns](plan/52_archetype-template-library.md) | -| 53 | ⛔ | [Conciseness Scoring](plan/53_conciseness-scoring.md) | -| 54 | ⛔ | [Conciseness Metrics Design and Implementation](plan/54_metrics-guide-tradeoffs.md) | -| 56 | ⛔ | [Spike Ollama for Weasel Detection](plan/56_spike-ollama-weasel-detection.md) | -| 58 | ⛔ | [Select and Package Fast Weasel Classifier (CPU Fallback)](plan/58_classifier-model-selection-and-embedding.md) | -| 61 | 🔳 | [Required Structure Rule Hardening](plan/61_required-structure-hardening.md) | -| 62 | ✅ | [Corpus Acquisition and Taxonomy](plan/62_corpus-acquisition.md) | -| 64 | ✅ | [Spike Pure-Go Embedded Weasel Classifier](plan/64_spike-go-native-linear-classifier.md) | -| 65 | 🔲 | [Spike WASM-Embedded Weasel Inference](plan/65_spike-wasm-embedded-inference.md) | -| 66 | ✅ | [Unified Conciseness Score](plan/66_unified-conciseness-score.md) | -| 68 | ⛔ | [Reorganize Documentation](plan/68_reorganize-docs.md) | -| 69 | ✅ | [Include enhancements: link adjustment and heading-level](plan/69_include-enhancements.md) | -| 73 | ✅ | [Unify template and processing directives](plan/73_unify-template-directives.md) | -| 74 | ✅ | [Directive guide](plan/74_directive-guide.md) | -| 75 | ✅ | [Single-brace placeholders everywhere](plan/75_single-brace-placeholders.md) | -| 76 | ✅ | [Rename misleading parameter names](plan/76_rename-misleading-params.md) | -| 77 | ✅ | [Template composition and cycle detection](plan/77_template-composition-and-cycles.md) | -| 78 | ✅ | [Query subcommand for front-matter filtering](plan/78_query-command.md) | -| 79 | ✅ | [Nested front-matter access](plan/79_nested-frontmatter-access.md) | -| 80 | ✅ | [Terminal recording in README](plan/80_terminal-recording-readme.md) | -| 81 | ✅ | [OOM mitigation: configurable file-size limit](plan/81_oom-file-size-limit.md) | -| 82 | ✅ | [YAML billion-laughs mitigation](plan/82_yaml-billion-laughs.md) | -| 83 | ✅ | [Security hardening batch](plan/83_security-hardening-batch.md) | -| 84 | 🔲 | [Symlink default-deny for file discovery](plan/84_symlink-default-deny.md) | -| 85 | 🔳 | [Increase test coverage to 95% by extracting shared rule helpers](plan/85_coverage-to-95-percent.md) | -| 86 | 🔳 | [Markdown flavor validation](plan/86_markdown-flavor-validation.md) | -| 87 | ✅ | [Flavor validation for GitHub Alerts](plan/87_markdown-flavor-github-alerts.md) | -| 88 | ✅ | [TOC directive migration aid](plan/88_toc-directive-migration.md) | -| 89 | 🔲 | [TOC generator directive and MDS035 auto-fix](plan/89_toc-generator-directive.md) | -| 90 | 🔲 | [Isolate corpus test git config from host signing](plan/90_corpus-test-git-config-isolation.md) | -| 91 | 🔲 | [MDS037 skips paragraphs inside generated sections](plan/91_mds037-skip-generated-sections.md) | +| ID | Status | Title | +|-----|--------|------------------------------------------------------------------------------------------------------| +| 52 | ✅ | [Archetype / Template Library for Agentic Patterns](plan/52_archetype-template-library.md) | +| 61 | 🔳 | [Required Structure Rule Hardening](plan/61_required-structure-hardening.md) | +| 65 | 🔲 | [Spike WASM-Embedded Weasel Inference](plan/65_spike-wasm-embedded-inference.md) | +| 78 | ✅ | [Query subcommand for front-matter filtering](plan/78_query-command.md) | +| 83 | ✅ | [Security hardening batch](plan/83_security-hardening-batch.md) | +| 84 | 🔲 | [Symlink default-deny for file discovery](plan/84_symlink-default-deny.md) | +| 85 | 🔳 | [Increase test coverage to 95% by extracting shared rule helpers](plan/85_coverage-to-95-percent.md) | +| 86 | 🔳 | [Markdown flavor validation](plan/86_markdown-flavor-validation.md) | +| 89 | 🔲 | [TOC generator directive and MDS035 auto-fix](plan/89_toc-generator-directive.md) | +| 90 | 🔲 | [Isolate corpus test git config from host signing](plan/90_corpus-test-git-config-isolation.md) | +| 91 | 🔲 | [MDS037 skips paragraphs inside generated sections](plan/91_mds037-skip-generated-sections.md) | diff --git a/docs/guides/metrics-tradeoffs.md b/docs/guides/metrics-tradeoffs.md index 9aba5c4f8..7ff39a475 100644 --- a/docs/guides/metrics-tradeoffs.md +++ b/docs/guides/metrics-tradeoffs.md @@ -6,7 +6,7 @@ summary: Trade-offs and threshold guidance for readability, structure, length, a ## Scope and disclaimer -This guide compares existing mdsmith rules that touch readability and length with token budget awareness and the proposed conciseness scoring (plan 53). Conciseness scoring is not implemented yet; any conciseness scores below are illustrative, not normative. +This guide compares existing mdsmith rules that touch readability and length with token budget awareness. Any metric scores and trade-offs below are illustrative and focus on the rules that are currently implemented. ## What the current rules measure @@ -20,13 +20,7 @@ This guide compares existing mdsmith rules that touch readability and length wit ## Planned metrics (not implemented) -| Metric | Goal | Status | -|-------------------------------------------------------------|--------------------------------------------|---------| -| [Conciseness Scoring](../../plan/53_conciseness-scoring.md) | Flag low information density in paragraphs | Planned | - -## What conciseness scoring is trying to measure - -Conciseness scoring (plan 53) focuses on information density rather than complexity or structure. It aims to flag paragraphs that are easy to read but say too little with too many words, which can waste tokens and create drift in agentic contexts. A plausible starting point is a heuristic that penalizes filler words, hedging language, and verbose phrases while rewarding content-bearing terms. +No additional metrics are planned at this time. ## What token budget awareness is trying to measure diff --git a/plan/50_redundancy-duplication-detection.md b/plan/50_redundancy-duplication-detection.md deleted file mode 100644 index 01cdab82f..000000000 --- a/plan/50_redundancy-duplication-detection.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -id: 50 -title: Redundancy / Duplication Detection -status: ✅ -summary: >- - MDS037 duplicated-content fingerprints paragraphs - by SHA-256 over lowercase, whitespace-normalized - text and scans the corpus (RootFS or the file's - directory) for matches above a configurable - min-chars threshold; opt-in by default. ---- -# Redundancy / Duplication Detection - -## Goal - -Flag substantial duplicated content across Markdown files -to reduce token waste and drift. - -## Tasks - -1. Chose paragraph fingerprints (SHA-256 over a - lowercase, whitespace-collapsed, trimmed form) - with a `min-chars` threshold defaulting to 200 - runes. -2. Implemented cross-file comparison via - [MDS037](../internal/rules/MDS037-duplicated-content/README.md); - scope follows `RootFS` when the project root is - known and falls back to the file's directory, and - `include`/`exclude` globs scope the walk. -3. Diagnostics point to the self line and the - other file's path and line - (`paragraph duplicated in {other}:{line}`). -4. README documents the *O(N²)* read pattern and - recommends an `exclude` entry for generated or - vendored directories. - -## Acceptance Criteria - -- [x] Rule identifies duplicated sections beyond - a configurable similarity threshold. -- [x] Findings include both source and duplicate locations. -- [x] Rule supports include/exclude patterns to limit scope. -- [x] All tests pass: `go test ./...` -- [x] `golangci-lint run` reports no issues diff --git a/plan/51_section-level-size-limits.md b/plan/51_section-level-size-limits.md deleted file mode 100644 index e2def595e..000000000 --- a/plan/51_section-level-size-limits.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -id: 51 -title: Section-Level Size Limits -status: ✅ ---- -# Section-Level Size Limits - -## Goal - -Enforce size limits on individual Markdown sections -to prevent oversized headings from hiding in otherwise compliant files. - -## Tasks - -1. [x] Define section boundary detection rules for heading levels. -2. [x] Add configuration for per-heading or per-pattern limits. -3. [x] Implement rule to count section length by lines. -4. [x] Document configuration and examples. - -## Acceptance Criteria - -- [x] Rule enforces size limits per heading level or heading pattern. -- [x] Findings include the heading title and measured size. -- [x] Limits can be configured independently of file-level limits. -- [x] All tests pass: `go test ./...` -- [x] `golangci-lint run` reports no issues - -## Notes - -Implemented as MDS036 `max-section-length` (named to mirror -`max-file-length`, since limits are measured in lines). A section -spans from its heading line up to (but not including) the next heading -line of any level, or end of file. Nested subsections are measured -independently of their parent, so the limit applies to direct content -under each heading. Lookup order for the applicable max: `per-heading` -regex (first match), `per-level`, `max`. Disabled by default; enable -by setting `max` (or any `per-level` / `per-heading` entry) to a -positive integer in `.mdsmith.yml`. diff --git a/plan/53_conciseness-scoring.md b/plan/53_conciseness-scoring.md deleted file mode 100644 index 70f75a9ac..000000000 --- a/plan/53_conciseness-scoring.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -id: 53 -title: Conciseness Scoring -status: "⛔" ---- -# Conciseness Scoring - -## Goal - -Measure information density and flag verbose Markdown content -that remains readable but unnecessarily long. - -Current state: heuristic prototype only. Keep rule disabled by default until -classifier-backed evaluation baselines are complete. - -## Tasks - -1. Define conciseness heuristics - (filler words, hedge phrases, low content-to-token ratios). -2. Implement scoring per paragraph and configurable thresholds. -3. Emit warnings with suggested targets and examples. -4. Document configuration and rationale. - -## Acceptance Criteria - -- [x] Rule flags paragraphs that exceed a configurable verbosity threshold. -- [x] Output includes the paragraph location and conciseness score. -- [x] Heuristics are configurable and documented. -- [x] All tests pass: `go test ./...` -- [x] `golangci-lint run` reports no issues -- [ ] Rule is validated and ready for default enablement diff --git a/plan/54_metrics-guide-tradeoffs.md b/plan/54_metrics-guide-tradeoffs.md deleted file mode 100644 index 5dca534b4..000000000 --- a/plan/54_metrics-guide-tradeoffs.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -id: 54 -title: Conciseness Metrics Design and Implementation -status: ⛔ ---- -# Conciseness Metrics Design and Implementation - -## Goal - -Design and implement conciseness scoring metrics -(heuristics, thresholds, and configuration) for mdsmith, -backed by tests and documentation. - -## Tasks - -1. Specify candidate conciseness metrics and choose a baseline heuristic - (filler/hedge ratios, content-to-token ratio, verbose phrase penalties). -2. Define tokenization and paragraph boundaries - that align with existing mdtext utilities. -3. Calibrate default thresholds using a representative doc set - and record false-positive risk. -4. Implement the conciseness rule with configurable thresholds, - word/phrase lists, and per-path overrides. -5. Add unit tests and fixtures covering false positives, - technical prose, and verbose-but-readable content. -6. Update rule docs and usage examples to explain configuration and trade-offs. - -## Acceptance Criteria - -- [ ] Conciseness metric is specified - with documented heuristics and default thresholds. -- [ ] Rule is implemented with configurable settings and per-path overrides. -- [ ] Tests cover representative readable, technical, and verbose cases. -- [ ] Documentation explains how to tune conciseness thresholds - and when to prefer other rules. diff --git a/plan/56_spike-ollama-weasel-detection.md b/plan/56_spike-ollama-weasel-detection.md deleted file mode 100644 index 7a36f36a5..000000000 --- a/plan/56_spike-ollama-weasel-detection.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -id: 56 -title: Spike Ollama for Weasel Detection -status: ⛔ ---- -# Spike Ollama for Weasel Detection - -## Goal - -Evaluate Ollama as a deterministic local inference backend -for weasel-language detection in mdsmith. - -## Tasks - -1. Build a reproducible Ollama spike setup suitable for local - development and CI-like environments. -2. Verify deterministic behavior using fixed sampling controls - (`temperature: 0`, fixed `seed`, stable prompt template). -3. Measure CPU latency, throughput, and memory across a small - markdown benchmark set. -4. Compare candidate lightweight models available in Ollama - for consistency and detection quality. -5. Define mdsmith integration contract: - invocation mode, timeout/retry policy, and strict fallback path. -6. Document operational constraints: model pull strategy, - artifact caching, and offline execution behavior. - -## Acceptance Criteria - -- [ ] Deterministic output is confirmed under fixed controls. -- [ ] CPU performance metrics are documented for benchmark files. -- [ ] Candidate model quality trade-offs are documented. -- [ ] Clear recommendation is produced for mdsmith adoption. diff --git a/plan/58_classifier-model-selection-and-embedding.md b/plan/58_classifier-model-selection-and-embedding.md deleted file mode 100644 index c38e018a8..000000000 --- a/plan/58_classifier-model-selection-and-embedding.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -id: 58 -title: Select and Package Fast Weasel Classifier (CPU Fallback) -status: ⛔ ---- -# Select and Package Fast Weasel Classifier (CPU Fallback) - -## Goal - -Pick a fast classifier for conciseness weasel-language detection. -Package its weights for offline use. Define a CPU fallback path. - -For this plan, "weasel-language" is operationalized as -`verbose-actionable` phrasing in conciseness evaluation. - -## Detection Contract - -Use one binary label with calibrated confidence: - -- `verbose-actionable`: wording is unnecessarily long -- `acceptable`: wording is concise enough for intent - -Classifier output contract: - -```json -{ - "label": "verbose-actionable", - "risk_score": 0.73, - "threshold": 0.60, - "model_id": "cue-linear-v1", - "mode": "auto", - "backend": "classifier", - "version": "2026-02-15" -} -``` - -Contract semantics: - -- `label` enum: `verbose-actionable` or `acceptable` -- `risk_score` range: `[0, 1]` where higher means more likely - `verbose-actionable` -- `mode` enum: `classifier`, `heuristic`, or `auto` -- `backend` enum: runtime backend actually used - (`classifier` or `heuristic`) -- decision rule: if `risk_score >= threshold` then label is - `verbose-actionable`; otherwise `acceptable` - -Threshold policy: - -- Optimize threshold on `dev` for `F0.5` (precision-weighted) -- Freeze threshold and report on `test` -- Keep one default threshold in config for deterministic behavior - -## Candidate Shortlist - -Model candidates are scoped to CPU-first, local/offline use: - -1. `cue-linear-lite-v1`: sparse linear model over cue counts -2. `cue-linear-v1`: sparse linear model over cues + density features -3. `hybrid-v1`: heuristic prefilter + linear classifier - -Selection criteria: - -- quality: precision, recall, `F1`, and `F0.5` on frozen `test` -- CPU latency: p50/p95 inference latency on local hardware -- artifact footprint: model size and binary size impact -- licensing: model weights and dependency compatibility with MIT repo - -## Evaluation Harness Design - -Harness inputs and assets: - -- rubric: `eval/conciseness/rubric.md` -- schema: `eval/conciseness/dataset.schema.cue` -- scorecard: `eval/conciseness/scorecard-template.md` -- dataset splits: `train/dev/test/holdout-outofdomain` JSONL - -Required benchmark outputs per candidate: - -- confusion matrix and precision/recall/`F1`/`F0.5` -- calibration summary (at least Brier score) -- diagnostics per KLOC at selected threshold -- p50/p95 CPU inference latency - -## Packaging Decision - -Preferred packaging strategy: - -1. Keep the selected weight artifact in-repo as versioned JSON -2. Embed the artifact with `go:embed` for offline default behavior -3. Store a SHA256 manifest for startup integrity checks -4. Allow an explicit external override path for local experiments -5. Reject mismatched checksum and fall back to heuristic mode - -Reproducibility requirements: - -- deterministic build pins model artifact path and checksum -- release notes record model version, checksum, and threshold -- checksum verification is covered by unit tests - -## CPU Fallback Behavior - -Runtime selection order: - -1. if classifier artifact loads and checksum matches, use classifier -2. else use current `MDS029` heuristic scoring path -3. if classifier inference exceeds timeout, degrade to heuristic - -Fallback must preserve rule behavior guarantees: - -- deterministic output for same input and config -- no runtime network dependency -- consistent diagnostic schema regardless of backend - -## Distribution Constraints - -- Keep embedded artifact footprint small enough to avoid major binary bloat -- Record per-platform binary size delta before enabling by default -- Keep third-party runtime dependencies minimal for portability -- Include model/weights license notes in rule docs and release notes - -## Integration Plan - -1. Add classifier interface and output schema in - `internal/rules/concisenessscoring/` -2. Add model loader with checksum verification and timeout controls -3. Add backend switch (`classifier`, `heuristic`, `auto`) in config -4. Add integration tests for classifier path and forced fallback path -5. Update MDS029 docs in - `internal/rules/MDS029-conciseness-scoring/` with thresholds, - packaging, and fallback semantics - -## Tasks - -1. Define detection contract: - labels, score threshold policy, and expected output schema. -2. Shortlist lightweight classifier models suitable for local use - (size, license, CPU speed, quality). -3. Build an evaluation harness over a labeled corpus and compare - precision, recall, F1, and latency on CPU. -4. Choose a model artifact packaging strategy: - embedded assets vs bundled files, checksum validation, - and update workflow. -5. Implement runtime selection and fallback behavior so detection - works in CPU-only environments without accelerator assumptions. -6. Document distribution constraints: - binary size impact, model weight footprint, and licensing notes. - -## Acceptance Criteria - -- [ ] One classifier model is selected with documented quality - and CPU performance metrics. -- [x] Model artifact packaging strategy is documented and reproducible. -- [ ] CPU fallback behavior is specified and validated. -- [x] Integration plan is ready for implementation in mdsmith. diff --git a/plan/62_corpus-acquisition.md b/plan/62_corpus-acquisition.md deleted file mode 100644 index ed5d61cd8..000000000 --- a/plan/62_corpus-acquisition.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -id: 62 -title: Corpus Acquisition and Taxonomy -status: ✅ ---- -# Corpus Acquisition and Taxonomy - -## Goal - -Create a repeatable workflow to collect and label Markdown files. -The corpus should cover agent docs, technical docs, and architecture -decision artifacts while tracking license and source metadata. - -## Tasks - -1. Define the corpus taxonomy and labeling rules. -2. Define source inclusion/exclusion policy - (license allowlist, repository quality, generated content filters). -3. Implement a collection pipeline that gathers Markdown files and - records provenance metadata - (repository, path, commit SHA, license, collection date). -4. Implement normalization and quality gates - (format normalization, minimum content thresholds, near-duplicate - detection, generated-file exclusion). -5. Implement a sampling strategy to balance categories and avoid - README-heavy skew. -6. Create a manual QA pass on a stratified sample and refine taxonomy - rules from observed confusion cases. -7. Freeze a versioned dataset manifest and deterministic - train/dev/test splits. -8. Document a periodic refresh workflow with drift reporting by - category. - -## Taxonomy Scope - -- Agent-control docs (`AGENTS.md`, `CLAUDE.md`, skills, task prompts) -- Tutorial -- How-to guide -- Reference -- Explanation/background -- Architecture decision record (ADR) -- Request for comments (RFC) -- Design proposal/tradeoff memo -- Runbook/playbook -- Incident postmortem -- Changelog/release notes/migration guide -- Project/process docs (`README`, `CONTRIBUTING`, `SECURITY`, governance) -- API/CLI/configuration/specification docs -- Troubleshooting/FAQ/onboarding/glossary - -## Acceptance Criteria - -- [x] Taxonomy includes category definitions, boundary rules, and at - least one positive/negative example per category. -- [x] Collection pipeline produces a manifest with source provenance and - license metadata for every file. -- [x] Dataset build excludes generated/low-signal content and reports - deduplication statistics. -- [x] Final corpus is category-balanced within defined target ranges and - reports per-category counts. -- [x] Manual QA on stratified samples reports precision/recall or - agreement metrics and drives at least one documented taxonomy - refinement. -- [x] Refresh process is documented and can publish a versioned corpus - update with drift summary. -- [x] All tests pass: `go test ./...` -- [x] `golangci-lint run` reports no issues diff --git a/plan/64_spike-go-native-linear-classifier.md b/plan/64_spike-go-native-linear-classifier.md deleted file mode 100644 index 2c352e30b..000000000 --- a/plan/64_spike-go-native-linear-classifier.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -id: 64 -title: Spike Pure-Go Embedded Weasel Classifier -status: ✅ ---- -# Spike Pure-Go Embedded Weasel Classifier - -## Goal - -Evaluate a fully embedded, pure-Go classifier path for weasel-language -(or `verbose-actionable`) detection with no runtime dynamic libraries. - -## Tasks - -1. Define a pure-Go model family to evaluate first - (for example sparse linear classifier over cue and n-gram features). -2. Build a minimal prototype inference package that runs with stdlib-only - runtime dependencies and deterministic scoring. -3. Define a weight packaging path that is fully embedded in the mdsmith - binary (for example `go:embed` plus checksum verification). -4. Measure CPU latency and memory on the same benchmark corpus used in - previous weasel spikes. -5. Measure binary-size impact versus current mdsmith and compare with the - yzma spike artifact footprint. -6. Define integration boundaries and fallback behavior for MDS029: - backend mode switch, timeout policy, and diagnostic stability. -7. Document maintenance workflow: training export format, versioning, - and safe model update procedure. - -## Results - -See `eval/conciseness/spikes/go-native-linear-classifier/README.md`. - -Highlights from the spike: - -- Prototype is fully pure-Go with stdlib-only runtime dependencies. -- Weights are embedded with `go:embed` and verified by pinned SHA256. -- Spike report includes classifier implementation details and raw captured - benchmark output in markdown. -- Cue lexicons were moved into the embedded artifact with validation gates - to keep list expansion maintainable and correctness-checked. -- Determinism was confirmed across in-process and process-restart runs - (`unique_hashes=1`). -- Latency and memory metrics were captured on the same six-sample corpus - used in the yzma spike. -- mdsmith binary delta for embedded artifact was measured at +480 bytes, - versus yzma's +0.5 MB binary delta plus external model/library artifacts. -- Recommendation: adopt this path as the CPU fallback candidate for plan 58, - pending full dataset quality validation. - -## Acceptance Criteria - -- [x] Prototype runs with no `YZMA_LIB` or external dynamic libraries. -- [x] Embedded weights load from binary-only assets. -- [x] Deterministic outputs are confirmed across repeat runs. -- [x] CPU latency and memory metrics are captured. -- [x] Binary-size delta is measured and documented. -- [x] Recommendation is made: adopt, defer, or reject this path. diff --git a/plan/66_unified-conciseness-score.md b/plan/66_unified-conciseness-score.md deleted file mode 100644 index 032080746..000000000 --- a/plan/66_unified-conciseness-score.md +++ /dev/null @@ -1,248 +0,0 @@ ---- -id: 66 -title: "Unified Conciseness Score" -status: ✅ ---- -# Plan 66: Unified Conciseness Score - -## Goal - -Produce a single `float64` conciseness score per -paragraph. Use a pure-Go linear classifier with 14 -features. Require zero external dependencies. - -## Motivation - -Six plans and seven PRs addressed conciseness with -diverging approaches. This plan consolidates them into -one roadmap that produces a single number per paragraph: -a `float64` in `[0, 1]` where `1.0` means maximally -concise. - -### Why this approach - -We select the enhanced pure-Go linear classifier because: - -1. **Infrastructure exists.** PR #33 has the classifier, - embedded weights, checksum verification, and benchmark - harness. 3.3 μs avg latency, +480 bytes binary, - deterministic across runs. -2. **Zero dependencies.** Pure Go, `CGO_ENABLED=0`, - single binary. No ONNX Runtime, no shared libraries, - no Ollama, no Python at runtime. -3. **100 % deterministic.** Same input always produces - the same score. Validated in PR #33 spike - (`unique_hashes=1`). -4. **Extensible.** Adding features to a linear model is - trivial. Each new feature is a pure-Go function that - returns a `float64`. Retraining weights is a single - Python script run offline. - -## Consolidated Plan Relationships - - - -### Superseded plans - -| Plan | Title | Disposition | -|------|--------------------------|------------------------| -| 53 | MDS029 conciseness score | Absorbed; close PR #21 | -| 54 | MDS029 conciseness rule | Absorbed; close PR #24 | -| 56 | Ollama spike | Won't continue; #34 | -| 58 | Classifier fallback | Partial absorb; #31 | - -### Completed foundation plans - -| Plan | Title | Disposition | -|------|--------------------------|-----------------------| -| 62 | Corpus acquisition | Absorbed; merge #35 | -| 64 | Pure-Go classifier spike | Foundation; merge #33 | - -### Merge order - -1. PR #33 (plan 64) — base classifier -2. PR #35 (plan 62) — corpus -3. This plan's PR — extended features, retrained weights, - MDS029 rule - -PRs #21, #24, #31, #34 are closed with a comment linking -to this plan. - -## Definition: The Conciseness Score - -```text -conciseness ∈ [0.0, 1.0] - -0.0 = maximally verbose (all filler, no content) -1.0 = maximally concise (every word carries meaning) -``` - -The score is the **sigmoid output** of a linear model -over paragraph-level features. The sigmoid maps to -`[0, 1]` and the model weights determine how each -feature contributes. - -The MDS029 rule fires when `conciseness < threshold` -(default `0.20`, configurable in `.mdsmith.yml`). - -```yaml -rules: - conciseness-scoring: - min-score: 0.20 # paragraphs below this are flagged -``` - -Diagnostic format: - -```text -README.md:14:1 MDS029 paragraph conciseness 0.38 … -``` - -## Features - -The unified scorer extracts these features from each -paragraph. All are pure Go, zero external dependencies. - -### Existing features (from PR #33) - -Implemented in the classifier package: - -- **filler_density** — filler words / total words -- **modal_density** — modal verbs / total words -- **vague_density** — vague words / total words -- **action_density** — action verbs / total words -- **hedge_density** — hedge phrases / total words -- **verbose_density** — verbose phrases / total words -- **stop_ratio** — stop words / total words - -### New features (this plan) - -| Feature | Signal | -|-------------------|--------------------------| -| compression_ratio | Redundancy via flate | -| type_token_ratio | Vocabulary repetition | -| nominal_density | Hidden verbs as nouns | -| sent_len_variance | Sentence length spread | -| func_word_ratio | Function word dilution | -| avg_word_length | Word length distribution | -| ly_adverb_density | Adverb overuse | - -Total: 15 features (8 existing + 7 new). - -## Implementation - -### Files to create or modify - -```text -internal/rules/concisenessscoring/ -├── classifier/ -│ ├── model.go # extend extractors -│ ├── model_test.go # extend tests -│ ├── features.go # NEW: 7 features -│ ├── features_test.go # NEW -│ └── data/ -│ └── cue-linear.json # NEW: weights -├── scorer.go # NEW: interface -├── scorer_test.go # NEW -├── rule.go # NEW: MDS029 rule -└── rule_test.go # NEW -internal/rules/MDS029-conciseness-scoring/ -└── README.md # update rule spec -``` - -### Weight retraining - -After adding the new features, retrain the model: - -1. Use the corpus from PR #35 (plan 62). -2. Extract all 15 features from each labeled paragraph. -3. Fit logistic regression (`sklearn.linear_model`). -4. Export weights and bias to `cue-linear.json`. -5. Generate SHA-256 checksum for `go:embed` verification. -6. Validate determinism: assert `unique_hashes=1`. - -The retraining script lives in `eval/conciseness/train/` -and runs offline when features or corpus change. - -## Tasks - -1. ~~Merge PR #33 (plan 64 base classifier)~~ (done) -2. ~~Merge PR #35 (plan 62 corpus)~~ (done) -3. ~~Add 7 new feature extractors in `features.go`~~ - (done) -4. ~~Add feature tests in `features_test.go`~~ (done) -5. ~~Retrain weights with 15 features, export v2 - JSON~~ (done) -6. ~~Implement `Scorer` interface in `scorer.go`~~ - (done) -7. ~~Implement MDS029 rule in - `internal/rules/concisenessscoring/rule.go`~~ - (done) -8. ~~Update MDS029 rule spec in - `internal/rules/MDS029-conciseness-scoring/`~~ - (done) -9. ~~Add config support for `min-score` threshold~~ - (done) -10. ~~Run determinism and benchmark validation~~ (done) -11. ~~Close superseded PRs #21, #24, #31, #34~~ (done) - -## Acceptance Criteria - -- [x] `mdsmith check` reports MDS029 diagnostics with - a conciseness score -- [x] Score is a `float64` in `[0, 1]`, printed to - 2 decimal places -- [x] Threshold configurable via `.mdsmith.yml` - `rules.conciseness-scoring.min-score` -- [x] All 15 features extracted in pure Go, - `CGO_ENABLED=0` -- [x] Deterministic: same paragraph produces same score - across runs and platforms -- [x] Binary size delta +32 KB (revised from < 2 KB; - 7 new feature functions + scorer + rule rewrite) -- [x] Latency ~46 μs per paragraph (< 100 μs p95) -- [x] `go test ./...` passes -- [x] `golangci-lint run` passes -- [x] `mdsmith check PLAN.md` passes -- [x] Superseded PRs (#21, #24, #31, #34) closed - -## Future: Transformer-Based Scoring - -The pure-Go ML ecosystem is maturing. Two projects -deserve re-evaluation in Q3 2026: - -- **GoMLX** (`gomlx/gomlx`) — pure-Go ML framework - with transformer support and SIMD acceleration. -- **Hugot** (`knights-analytics/hugot`) — runs - HuggingFace pipelines in pure Go. -- **gonnx** (`AdvancedClimateSystems/gonnx`) — pure-Go - ONNX runtime, ~8x slower but zero C deps. - -A fine-tuned small transformer could replace the linear -model for higher accuracy. Gate behind a build tag -(`-tags conciseness_ml`). The linear classifier remains -the default. - -See issue #111 for tracking. - -## Testing - -```bash -GOCACHE=/tmp/mdsmith-gocache go test ./... -GOCACHE=/tmp/mdsmith-gocache \ - GOLANGCI_LINT_CACHE=/tmp/mdsmith-golangci-cache \ - go tool golangci-lint run --allow-parallel-runners -GOCACHE=/tmp/mdsmith-gocache go run ./cmd/mdsmith check \ - PLAN.md plan/66_unified-conciseness-score.md -``` - -## References - -- PR #33: pure-Go classifier spike (plan 64) -- PR #35: corpus acquisition (plan 62) -- PR #24: MDS029 rule definition (plan 54) -- PR #31: classifier fallback interface (plan 58) -- ConCISE (2025, arxiv:2511.16846): reference-free - conciseness metric via compression ratios -- EMNLP 2022 TSAR: "Conciseness: An Overlooked Language - Task" (Stahlberg et al.) -- ACL 2023: compression-based text classification diff --git a/plan/68_reorganize-docs.md b/plan/68_reorganize-docs.md deleted file mode 100644 index d10c671e1..000000000 --- a/plan/68_reorganize-docs.md +++ /dev/null @@ -1,126 +0,0 @@ ---- -id: 68 -title: Reorganize Documentation -status: "⛔" ---- -# Reorganize Documentation - -## Goal - -Move reference content out of CLAUDE.md into dedicated -files. Make AGENTS.md use the include directive to keep -agent configs in sync. - -## Background - -CLAUDE.md mixes development workflow, CLI design spec, -background material, and agent-config housekeeping. -Moving each concern to its natural home makes CLAUDE.md -shorter. The include directive (``) lets AGENTS.md and -`.github/copilot-instructions.md` pull content from -shared files at fix-time. A single edit then spreads -to every file. - -## Tasks - -### Create target files - -1. Create `DEVELOPMENT.md` with sections moved from - CLAUDE.md (see below). -2. Create `docs/design/` directory. Move CLI Design - section to `docs/design/cli.md`. -3. Move `background/` to `docs/background/`. -4. Move `guides/` to `docs/guides/`. -5. Move `archetypes/` to `docs/design/archetypes/`. - -### Populate DEVELOPMENT.md - -6. Move "Build & Test Commands" from CLAUDE.md. -7. Move "Project Layout" from CLAUDE.md. -8. Move "Development Workflow" from CLAUDE.md. - Reword: "New features are test-driven" becomes - "Any change follows Red / Green TDD: write a - failing test (red), make it pass (green), commit". -9. Move "Code Style" from CLAUDE.md. -10. Move "PR Workflow" from CLAUDE.md. Update the - `gh api` example to use - `gh repo view --json nameWithOwner` for - dynamic owner/repo lookup and `--paginate`. - -### Update CLAUDE.md - -11. Add an include directive in CLAUDE.md that pulls - DEVELOPMENT.md so agents still see the development - info inline (`` ... ``). -12. Update "Merge Conflicts in PLAN.md and README.md" - to reference processing-instruction markers - (`` / ``) instead of old - HTML-comment markers (``). -13. Rewrite "Cross-Platform Agent Config" to state - that CLAUDE.md is the primary doc and mdsmith - keeps the others in sync via include directives. -14. Remove the "Config & Rules" section entirely. - -### Update AGENTS.md - -15. Replace the body of AGENTS.md with an include - directive (`` ... - ``) that pulls relevant sections so - it stays in sync automatically. - -### Update copilot-instructions.md - -16. Replace the body of - `.github/copilot-instructions.md` with an include - directive (same `` - approach). - -### Update README.md - -17. Update README.md to reference the new docs layout: - include DEVELOPMENT.md, and link to `docs/design/`, - `docs/guides/`, `docs/background/`, and `plan/`. - -### Fixups - -18. Update internal links in moved files and inbound - references from non-moved files so they resolve - from the new locations. -19. Update `.mdsmith.yml` overrides and ignore entries - that reference old paths (`background/`, `guides/`, - `archetypes/`). -20. Run `mdsmith fix .` to regenerate all include and - catalog sections. -21. Run `mdsmith check .` and fix any diagnostics. - -## Acceptance Criteria - -- [ ] CLAUDE.md includes DEVELOPMENT.md via an - `` directive -- [ ] CLAUDE.md no longer contains Build & Test, - Project Layout, Development Workflow, Code Style, - CLI Design, PR Workflow, or Config & Rules - as hand-maintained sections -- [ ] CLAUDE.md "Merge Conflicts" section references - `` processing-instruction syntax -- [ ] CLAUDE.md "Cross-Platform Agent Config" says - CLAUDE.md is the primary source and mdsmith - keeps others in sync -- [ ] AGENTS.md uses an `` - directive with a `file:` parameter -- [ ] `.github/copilot-instructions.md` uses an - `` directive -- [ ] `background/` moved to `docs/background/` -- [ ] `guides/` moved to `docs/guides/` -- [ ] `archetypes/` moved to `docs/design/archetypes/` -- [ ] CLI Design lives in `docs/design/cli.md` -- [ ] DEVELOPMENT.md exists with the moved sections -- [ ] README.md includes DEVELOPMENT.md via an - `` directive -- [ ] README.md links to `docs/design/`, - `docs/guides/`, `docs/background/`, and `plan/` -- [ ] All tests pass: `go test ./...` -- [ ] `golangci-lint run` reports no issues -- [ ] `mdsmith check .` reports zero diagnostics diff --git a/plan/69_include-enhancements.md b/plan/69_include-enhancements.md deleted file mode 100644 index b6d60321e..000000000 --- a/plan/69_include-enhancements.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -id: 69 -title: 'Include enhancements: link adjustment and heading-level' -status: "✅" ---- -# Include enhancements - -## Goal - -Add two features to the include directive (MDS021): -automatic link-path rewriting and a `heading-level` -parameter for heading-level adjustment. - -## Background - -Relative links break when a file is included into a -document in a different directory. For example, -`docs/guide.md` includes `DEVELOPMENT.md` (a copy or -symlink placed alongside the including file, since -include `file:` paths may not contain `..` traversal -segments). A link like `[layout](internal/rules/)` in -DEVELOPMENT.md resolves from the repo root, but when -included from `docs/guide.md` it points to -`docs/internal/rules/` instead. The include directive -must rewrite each relative link target so it resolves -from the including file. - -Heading levels also need adjustment. DEVELOPMENT.md -uses `##` headings. When included under `## Project` -in CLAUDE.md those headings appear as siblings, not -children. The `heading-level` parameter (set to `"absolute"`) -shifts included headings to nest under the parent. - -## Design - -### Link adjustment (always, automatic) - -After reading the file and stripping frontmatter, -scan every Markdown link and image. A link looks like -`[text](target)` and an image like `![alt](target)`. -For each relative target (not `/`, `#`, `http://`, -or `https://`): - -1. Get the included file's directory relative to the - FS root (e.g. `DEVELOPMENT.md` → `.`). -2. Get the including file's directory relative to the - FS root (e.g. `docs/guide.md` → `docs`). -3. Rewrite the target: - `newTarget = relpath(includingDir, - join(includedDir, target))`. - -Skip the transformation when both files share the -same directory. - -### `heading-level` parameter - -New optional parameter `heading-level` (values: -`"absolute"` or omitted). - -When `heading-level: "absolute"`: - -1. Find the heading level of the section that contains - the `` marker (the "parent level"). Use - 0 when the marker sits at the document root. -2. Find the minimum heading level in the included - content (the "source top level"). -3. Compute `shift = parentLevel - sourceTopLevel + 1` - so included top-level headings become children of - the parent. Skip when shift is zero. -4. Add `shift` to every ATX heading (`#` prefix) and - setext heading (underline). Cap at level 6. - -Example: include under `## Project` (level 2), source -has `## Build` (level 2) and `### Sub` (level 3). -`shift = 2 - 2 + 1 = 1`. Result: `### Build` (3), -`#### Sub` (4). - -## Tasks - -1. [x] Add a helper `adjustLinks(content, - includedFilePath, includingFilePath)` in - [`internal/rules/include/`](../internal/rules/include/) - that rewrites relative link/image targets -2. [x] Write unit tests for `adjustLinks`: same directory - (no-op), different directories, anchors and - absolute URLs left untouched, query strings - preserved -3. [x] Call `adjustLinks` in `generateIncludeContent` - after frontmatter stripping, before wrap -4. [x] Add a helper `adjustHeadings(content, parentLevel)` - that shifts ATX and setext heading levels -5. [x] Write unit tests for `adjustHeadings`: shift up, - shift down, cap at 6, no headings (no-op) -6. [x] Extend `validateIncludeDirective` to accept and - validate the `heading-level` parameter (only - `"absolute"` is valid) -7. [x] In `generateIncludeContent`, detect the parent - heading level from the marker position and call - `adjustHeadings` when `heading-level: "absolute"` -8. [x] Add test for parent-level detection (marker under - h2, under h3, at document root) -9. [x] Update the rule README at - [`MDS021-include/README.md`](../internal/rules/MDS021-include/README.md) - to document both features -10. [x] Update existing fixtures and tests if link - adjustment changes their expected output -11. [x] Run `go test ./...`, `go tool golangci-lint run`, - and `mdsmith check .` - -## Acceptance Criteria - -- [x] Relative links in included content are rewritten - so they resolve from the including file's - directory, not the source file's directory -- [x] Absolute URLs, anchor-only links (`#foo`), and - protocol links (`http://`, `https://`) are not - modified -- [x] `heading-level: "absolute"` shifts headings so - the included top-level headings appear one level - below the enclosing section -- [x] When `heading-level` is omitted, heading levels - stay unchanged -- [x] Heading level never exceeds 6 -- [x] Invalid `heading-level` values produce a diagnostic -- [x] Link adjustment is always applied (no parameter - needed) -- [x] All tests pass: `go test ./...` -- [x] `golangci-lint run` reports no issues -- [x] `mdsmith check .` reports zero diagnostics diff --git a/plan/73_unify-template-directives.md b/plan/73_unify-template-directives.md deleted file mode 100644 index ef125b072..000000000 --- a/plan/73_unify-template-directives.md +++ /dev/null @@ -1,264 +0,0 @@ ---- -id: 73 -title: Unify template and processing directives -status: "✅" -summary: >- - Research plan: blind trials on all 33 rules - identified six areas of user confusion. - Implementation split into plans 74-77. ---- -# Unify template and processing directives - -Research for -[#68](https://github.com/jeduden/mdsmith/issues/68) -and -[#70](https://github.com/jeduden/mdsmith/issues/70). - -Implementation plans: - -- [74](74_directive-guide.md) -- directive guide -- [75](75_single-brace-placeholders.md) -- - `{field}` syntax for required-structure -- [76](76_rename-misleading-params.md) -- - rename misleading parameter names -- [77](77_template-composition-and-cycles.md) -- - template composition, cycle detection, - `template` -> `schema` rename - -Follow-on: - -- [79](79_nested-frontmatter-access.md) -- - nested front-matter access (depends on 75) - -## Blind trial design - -Ten agents played "new developer." Each got a -two-sentence intro (max 50 words) and syntax -snippets. They guessed semantics and rated -confidence 1-5. Round 1 covered directives and -config (15 snippets). Round 2 covered the -remaining 25 rules. - -## Results: confidence scores - -| Snippet | Topic | Avg | Range | -|---------|-------------------------------|-----|-------| -| 1 | `` pair | 4.8 | 4-5 | -| 2 | `` pair | 4.8 | 4-5 | -| 3 | `` single marker | 4.0 | 2-5 | -| 4 | `` | 4.8 | 4-5 | -| 5 | `{{.id}}: {{.title}}` heading | 3.8 | 3-4 | -| 6 | catalog `row` table template | 4.4 | 4-5 | -| 7 | `line-length` config | 4.0 | 4-5 | -| 8 | config overrides | 4.8 | 4-5 | -| 9 | CUE front-matter schema | 4.4 | 4-5 | -| 10 | `{{.field}}` heading vs row | 4.0 | 3-5 | -| 11 | 4-space indented directive | 2.6 | 1-4 | -| 12 | nested directives in row | 2.0 | 1-3 | -| 13 | empty section at EOF | 4.6 | 3-5 | -| 14 | `paragraph-readability` | 4.0 | 3-5 | -| 15 | `token-budget` with ratio | 3.8 | 2-5 | - -Round 2 (MDS002-MDS033): all simple style rules -scored confidence 5. - -## Six areas of confusion - -1. `{{.field}}` means "insert" in catalog but - "match" in required-structure. All five - participants flagged this as the top source - of confusion. -> [plan 75](75_single-brace-placeholders.md) -2. 4-space indent silently kills directives. - No diagnostic. Confidence 2.6, 4/5 called - it a footgun. -> [plan 74](74_directive-guide.md) -3. Nested directives undefined. Confidence 2.0. - Nobody could predict what happens. - -> [plan 74](74_directive-guide.md) -4. Misleading parameter names: `ratio` (warning - threshold?), `max-words` (per paragraph?), - `max-column-width-variance` (statistical?). - -> [plan 76](76_rename-misleading-params.md) -5. Users cannot predict which rules auto-fix. - -> [plan 74](74_directive-guide.md) -6. `directory-structure: true` without `allowed` - is a silent no-op. - -> [plan 76](76_rename-misleading-params.md) - -## What works well - -- Self-describing names: `allow-empty-section` - (4.8), `overrides` (4.8). -- Marker pairs intuitive: catalog and include - both scored 4.8. -- CUE schema readable: `|` union and `?` - optional read naturally (4.4). -- PIs are hidden on GitHub renders (intended). -- `{field}` renders as visible literal text on - GitHub, making template placeholders readable - in unprocessed files. - -## Goal - -Give each directive a clear "if X then Y" rule. -Write one guide covering all rules with -examples. Fix misleading names. Switch -required-structure to `{field}` single braces. - -## Verification trial (post-change) - -Re-ran 10 snippets with the guide, `{field}` -syntax, renamed params, and fixability table. - -| Snippet | Topic | Before | After | -|---------|------------------------|--------|-------| -| 1 | `{field}` heading | 3.8 | 4.6 | -| 2 | `{{.f}}` vs `{f}` | 4.0 | 4.8 | -| 3 | 4-space indent | 2.6 | 5.0 | -| 4 | nesting | 2.0 | 4.8 | -| 5 | words-per-token | 3.8 | 4.6 | -| 6 | max-words-per-sentence | n/a | 5.0 | -| 7 | max-column-width-ratio | n/a | 4.8 | -| 8 | dir-structure no-op | n/a | 4.6 | -| 9 | line-length fixable? | ~2.5 | 5.0 | -| 10 | code-lang fixable? | ~2.5 | 5.0 | - -Key wins: - -- Indent footgun: 2.6 -> 5.0 (guide warning) -- Nesting: 2.0 -> 4.8 (guide statement) -- Fixability: ~2.5 -> 5.0 (fixability table) -- No "threshold" misreadings of renamed params -- `{field}` vs `{{.field}}` clearly distinct - -Remaining gaps: - -- `directory-structure: true` no-op still - confused 2/5. Plan 76 config warning needed. -- `` in a normal file is silently - ignored (5/5 flagged as confusing in - template-vs-normal trial). -- `` in a template does - not propagate to documents (5/5 noted the - misleading co-occurrence with `## ...`). -- Templates enforce headings and front matter - only, not directives (2/5 uncertain about - `` in a template). - -All three template-vs-normal gaps addressed -by plan 74 (guide section on templates). - -## Hugo-user trial (5 participants) - -Tested all 33 rules with agents primed as Hugo -users per -[#73](https://github.com/jeduden/mdsmith/issues/73). - -Hugo-specific traps (confidence drops): - -- `{{.title}}` vs Hugo's `{{ .Title }}`: - case-sensitive key lookup silently returns - empty string. 5/5 flagged. -- "Template" means validation schema, not - rendering. 5/5 confused by the word reuse. -- Generated content committed to git: inverts - Hugo's "never commit build output" model. - 5/5 called this disorienting. -- No nesting: Hugo shortcodes compose freely. - 5/5 expected nesting to work. -- `` YAML quoting rules are alien vs - Hugo's `{{< key="val" >}}`. 4/5 flagged. -- No template functions (`humanize`, etc.). - 3/5 reached for them. - -What worked: simple style rules (5.0), config -overrides (5.0), self-describing names (5.0). - -Confirmed prior findings: `max-words` misread -as per-paragraph (5/5), `max-column-width-variance` misread as -statistical (4/5). - -All Hugo-specific gaps addressed by plan 74 -(guide must include a "coming from Hugo" -section). - -## Final validation (9 personas, post-impl) - -3 normal devs, 3 Hugo users, 3 tech writers. -All given the shipped guide with `{field}` -syntax, renamed params, `schema:` key, -composition, and fixability table. - -| Snippet | Topic | Dev | Hugo | Writer | Avg | -|---------|------------------------|-----|------|--------|-----| -| S1 | catalog `{title}` | 5.0 | 5.0 | 5.0 | 5.0 | -| S2 | schema `{id}: {name}` | 4.0 | 4.0 | 5.0 | 4.3 | -| S3 | `{{.f}}` vs `{f}` | 5.0 | 5.0 | 5.0 | 5.0 | -| S4 | 4-space indent | 5.0 | 2.7 | 5.0 | 4.2 | -| S5 | nesting | 4.7 | 4.3 | 4.7 | 4.6 | -| S6 | words-per-token | 4.3 | 3.7 | 4.3 | 4.1 | -| S7 | max-words-per-sentence | 5.0 | 5.0 | 5.0 | 5.0 | -| S8 | column-width-ratio | 3.0 | 4.0 | 4.7 | 3.9 | -| S9 | `schema:` config | 3.7 | 4.0 | 4.3 | 4.0 | -| S10 | `` warning | 4.7 | 4.7 | 4.0 | 4.4 | -| S11 | fixability | 4.3 | 5.0 | 5.0 | 4.8 | -| S12 | schema `` | 4.0 | 4.0 | 3.7 | 3.9 | -| S13 | dir-structure warn | 4.0 | 4.3 | 4.0 | 4.1 | -| S14 | `{Title}` hint | 5.0 | 2.7 | 5.0 | 4.2 | - -Before/after on originally-confused areas: - -| Area | Before | After | Delta | -|-------------------|--------|----------|-------| -| `{field}` clarity | 4.0 | 5.0 (S3) | +1.0 | -| 4-space footgun | 2.6 | 4.2 | +1.6 | -| Nesting | 2.0 | 4.6 | +2.6 | -| Fixability | ~2.5 | 4.8 | +2.3 | -| `ratio` param | 3.8 | 4.1 (S6) | +0.3 | -| `max-words` param | ~3.0 | 5.0 (S7) | +2.0 | - -Remaining below 4.0: `max-column-width-ratio` -(3.9) and schema `` composition -(3.9). Both are new concepts with no prior -mental model. The guide explains them but users -need hands-on experience. - -Hugo users scored lower on S4 (indent, 2.7) -and S14 (case hint, 2.7) despite the guide. -S4: Hugo users unfamiliar with Markdown's -4-space code block rule. S14: the "did you -mean?" hint exists but Hugo muscle memory -overrides reading the guide. Both are -addressable by the "coming from Hugo" section -calling these out prominently. - -## Execution order - -Plans 75, 76, 77 are independent of each -other and can land in any order. Plan 74 (the -guide) depends on all three and must land last. - -```text -75 (single-brace) ──┐ -76 (param renames) ──┼──> 74 (guide) -77 (composition) ──┘ -``` - -## Issue coverage - -| Issue | Plans | -|-----------------------------------------------------|----------------| -| [#68](https://github.com/jeduden/mdsmith/issues/68) | 73, 74, 75, 77 | -| [#70](https://github.com/jeduden/mdsmith/issues/70) | 73, 74 | -| [#71](https://github.com/jeduden/mdsmith/issues/71) | 77 | -| [#73](https://github.com/jeduden/mdsmith/issues/73) | 73, 74, 76, 77 | - -## Tasks - -1. Plans 74, 75, 76, 77 written (done) - -## Acceptance Criteria - -- [x] Plans 74, 75, 76, 77 exist and pass lint -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues diff --git a/plan/74_directive-guide.md b/plan/74_directive-guide.md deleted file mode 100644 index 661676c9e..000000000 --- a/plan/74_directive-guide.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -id: 74 -title: Directive guide -status: "✅" -summary: >- - Central guide for all directives and rules - with examples, fixability table, placement - rules, and nesting behavior. ---- -# Directive guide - -Part of the user-model work from -[plan 73](73_unify-template-directives.md). -Sibling plans: - -- [75](75_single-brace-placeholders.md) -- - `{field}` syntax -- [76](76_rename-misleading-params.md) -- - param renames -- [77](77_template-composition-and-cycles.md) -- - composition, cycles, `schema` rename - -Addresses -[#68](https://github.com/jeduden/mdsmith/issues/68), -[#70](https://github.com/jeduden/mdsmith/issues/70). - -## Goal - -One guide a developer reads to understand every -directive and rule without consulting per-rule -READMEs. Must also serve Hugo users per -[#73](https://github.com/jeduden/mdsmith/issues/73). - -Depends on: plans 75, 76, 77 (guide documents -the final syntax, parameter names, and schema -composition). Write the guide last so it -reflects all changes: - -- Plan 75: `{field}` replaces `{{.field}}` in - both catalog rows and schema headings. One - syntax everywhere. Go templates removed from - user-facing surface. -- Plan 76: `ratio` -> `words-per-token`, - `max-words` -> `max-words-per-sentence`, - `max-column-width-variance` -> `max-column-width-ratio`; - "did you mean?" hint for case-mismatched - front-matter keys in catalog. -- Plan 77: config key `template` -> `schema`; - `` works in schema files for - composition; cycle detection with max depth - 10; `` in non-schema files emits - a warning. - -## Context - -Blind trials (plan 73) showed six gaps that -docs alone can close: - -- 4-space indent silently breaks directives - (confidence 2.6, no diagnostic emitted). -- Nested directives are undefined (confidence - 2.0, nobody could predict behavior). -- Users cannot predict which rules auto-fix - (fix confidence 2-3 points lower than check). -- `` in a normal file is silently - ignored (5/5 flagged as confusing -- looks - like it should work anywhere). -- `` in a template does - not propagate to documents using that - template (5/5 noted the misleading - co-occurrence with `## ...`). -- Templates only enforce headings and front - matter, not directives (2/5 uncertain - whether `` in a template requires - documents to also contain one). - -## Rendering note - -Processing instructions (``) are hidden -by GitHub's Markdown renderer (CommonMark -type-3 HTML blocks). Directives stay invisible -in rendered docs. Generated content between -markers is visible. - -## Tasks - -1. Create use-case guides split by topic: - - - `docs/guides/index.md`: guides index with - generated catalog of all guide pages - - `docs/guides/directives/generating-content.md`: - catalog and include use cases with examples, - placeholder syntax, nesting, and placement - rules - - `docs/guides/directives/enforcing-structure.md`: - schema, require, allow-empty-section use cases - with composition, schema-vs-normal-file, and - optional fields - - `docs/guides/directives/hugo-migration.md`: - standalone Hugo migration guide with - placeholder syntax and schema differences - - `rule-directory.md`: generated catalog of all - rules from rule READMEs (project root) - -2. Add a "see the directive guide" link from - each rule README that uses a directive - (MDS019, MDS020, MDS021, MDS030). -3. Replace embedded rules table in README.md with - a link to the rule directory. -4. Run `mdsmith fix` and `mdsmith check .` to - verify. - -## Acceptance Criteria - -- [x] `docs/guides/index.md` exists with generated - catalog of guides -- [x] Guide covers all four directives with - examples -- [x] Rule directory generated from rule READMEs - (replaces manual fixability table per review) -- [x] Guide documents 4-space indent footgun -- [x] Guide states nesting is not supported -- [x] Guide documents unified `{field}` syntax -- [x] Guide has schema-vs-normal-file section -- [x] Guide states `` is - schema-only -- [x] Guide states schema directives do not - propagate to documents -- [x] Guide documents schema composition via - `` -- [x] Guide documents renamed parameters -- [x] Guide has "coming from Hugo" section -- [x] Guide uses `schema` not `template` - throughout -- [x] Guide passes `mdsmith check docs/guides/` -- [x] Rule READMEs link to the guide -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues diff --git a/plan/75_single-brace-placeholders.md b/plan/75_single-brace-placeholders.md deleted file mode 100644 index c4b07505a..000000000 --- a/plan/75_single-brace-placeholders.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -id: 75 -title: Single-brace placeholders everywhere -status: "✅" -summary: >- - Replace {{.field}} with {field} in both - required-structure and catalog. One placeholder - syntax across the whole tool. ---- -# Single-brace placeholders everywhere - -Part of the user-model work from -[plan 73](73_unify-template-directives.md). -Addresses -[#68](https://github.com/jeduden/mdsmith/issues/68). - -Depends on: none (independent of other plans). -Update plan 74 guide after landing. - -## Goal - -One placeholder syntax: `{field}`. It works -the same way everywhere -- resolves a -front-matter key by name using CUE path syntax -([plan 79](79_nested-frontmatter-access.md) -adds nested access). - -- In catalog `row`/`header`/`footer`: inserts - the value from matched files' front matter. -- In schema headings: heading must contain the - value from the document's own front matter. - -`{{.field}}` (Go template syntax) is removed -from the user-facing surface entirely. - -## Context - -Blind trials (plan 73) showed `{{.field}}` was -the top confusion source. The original plan -separated the two syntaxes (`{field}` for -schemas, `{{.field}}` for catalog). But keeping -Go templates in catalog means two grammars. -Aligning to one syntax is simpler. - -Go `text/template` is overkill for catalog row -rendering: no user uses conditionals, loops, -or functions in row templates. Simple string -interpolation with `{field}` suffices. - -## Rendering note - -`{field}` renders as visible literal text on -GitHub. `# {id}: {name}` reads as a clear -pattern. Catalog row params in YAML also read -naturally: `row: "- [{title}]({filename})"`. -No backslash escaping or dot-prefix needed. - -## Design - -Replace Go `text/template` in catalog with -simple `{field}` interpolation. Same regex, -same resolver, same CUE path semantics as -required-structure. - -- `row: "- [{title}]({filename})"` -- catalog - row rendering. -- `# {id}: {name}` -- schema heading match. -- `{field}` resolves to empty string if the - key is missing (current behavior preserved). -- `{filename}` remains a built-in field in - catalog context (relative path). -- Other built-ins: `{title}`, `{summary}`, - etc. are front-matter lookups, not special. - -### What changes from Go templates - -| Feature | Go template | `{field}` | -|---------------|---------------------|------------------------| -| Syntax | `{{.field}}` | `{field}` | -| Nested access | `{{.a.b}}` | `{a.b}` (plan 79) | -| Quoted keys | `{{ index . "k" }}` | `{"my-key"}` (plan 79) | -| Conditionals | `{{ if .x }}` | Not supported | -| Loops | `{{ range }}` | Not supported | -| Functions | `{{ .x \| fn }}` | Not supported | -| Missing key | empty string | empty string | - -No catalog in the repo uses conditionals, -loops, or functions today. - -### Escaping literal braces - -Literal `{` is written as `{{`, literal `}` as -`}}`. Same convention as Python's `str.format`. -Example: `row: "{{literal}} {title}"` renders -as `{literal} My Title`. - -## Tasks - -1. Add a `{field}` interpolation engine in a - shared package (e.g. `internal/fieldinterp`): - - - Parse `{...}` placeholders from a string - - Resolve each placeholder against a - `map[string]any` using CUE path rules - - Return the interpolated string - - Handle `{{` as escaped literal `{` - -2. Update `catalog/generate.go`: - - - Replace `text/template` rendering with - `fieldinterp.Interpolate` - - Keep `{filename}` as a built-in injected - into the data map before interpolation - -3. Update `requiredstructure/rule.go`: - - - Replace `fieldPattern` regex with call to - the shared `fieldinterp` parser - - `resolveFields` uses the shared resolver - - Pattern matching builds regex from parsed - placeholders (same logic, shared parse) - -4. Update unit tests in both rules. -5. Update fixtures: - - - `internal/rules/MDS020-required-structure/` - - `internal/rules/MDS019-catalog/` - - Any fixture templates or catalog directives - using `{{.field}}` - -6. Migrate all schema files: - - - `plan/proto.md` - - `internal/rules/proto.md` - - `.claude/skills/proto.md` - -7. Migrate all catalog directives in the repo - (CLAUDE.md, README.md, rule READMEs) from - `{{.field}}` to `{field}`. -8. Update rule READMEs (MDS019, MDS020). -9. Update `docs/guides/directives/` guides (plan 74) - if it already exists. -10. Run `mdsmith check .` to verify. - -## Acceptance Criteria - -- [x] `{field}` is the only placeholder syntax - in both catalog and required-structure -- [x] `{{.field}}` is no longer recognized -- [x] All schema files use `{field}` -- [x] All catalog directives use `{field}` -- [x] Shared interpolation engine exists -- [x] Literal `{` is escaped as `{{` -- [x] MDS019 and MDS020 READMEs updated -- [x] All fixtures updated and passing -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues -- [x] `mdsmith check .` passes diff --git a/plan/76_rename-misleading-params.md b/plan/76_rename-misleading-params.md deleted file mode 100644 index 44d961a9c..000000000 --- a/plan/76_rename-misleading-params.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -id: 76 -title: Rename misleading parameter names -status: "✅" -summary: >- - Rename ratio to tokens-per-word, max-words - to max-words-per-sentence, max-column-width-variance - to max-column-width-ratio, and warn on - directory-structure no-op. ---- -# Rename misleading parameter names - -Part of the user-model work from -[plan 73](73_unify-template-directives.md). -Addresses -[#68](https://github.com/jeduden/mdsmith/issues/68) -(user model clarity) and -[#73](https://github.com/jeduden/mdsmith/issues/73) -(Hugo user "did you mean?" hint). - -Depends on: none (independent of other plans). -Update plan 74 guide after landing. - -## Goal - -Each config parameter name tells you what it -measures and what unit it uses. - -## Context - -Blind trials (plan 73) found three names that -mislead: - -- `ratio: 0.75` in token-budget: 2/5 read it - as a warning threshold. It is a words-to- - tokens multiplier. -- `max-words: 40` in paragraph-structure: reads - as per-paragraph limit. It is per-sentence. -- `max-column-width-variance: 60` in - table-readability: reads as statistical - variance. It is max/min ratio. - -Also found: `directory-structure: true` without -`allowed` is a silent no-op. - -## Design - -No deprecation. Rename in place, update all -config files and docs in a single PR. - -## Tasks - -1. Rename `ratio` to `tokens-per-word` in - MDS028 (token-budget): - - - Update `ApplySettings` and - `DefaultSettings` in - `internal/rules/tokenbudget/rule.go` - - Update `internal/rules/MDS028-token-budget/README.md` - - Update `.mdsmith.yml` - -2. Rename `max-words` to - `max-words-per-sentence` in MDS024 - (paragraph-structure): - - - Update - `internal/rules/paragraphstructure/rule.go` - - Update `internal/rules/MDS024-paragraph-structure/README.md` - - Update `.mdsmith.yml` - -3. Rename `max-column-width-variance` to - `max-column-width-ratio` in MDS026 - (table-readability): - - - Update - `internal/rules/tablereadability/rule.go` - - Update `internal/rules/MDS026-table-readability/README.md` - - Update `.mdsmith.yml` - -4. Add config warning for MDS033 - (directory-structure) when enabled without - `allowed`: - - - In `ApplySettings`, when enabled but - `allowed` is empty or absent, mark the rule - as configured so `Check` runs - - In `Check`, when configured with an empty - `allowed`, emit a config warning: - `directory-structure: rule enabled but no - "allowed" patterns configured` - - Update `internal/rules/MDS033-directory-structure/README.md` - -5. Add "did you mean?" diagnostic for - case-mismatched front-matter keys in catalog - (MDS019): - - - Extract referenced `{Field}` placeholder - names from the row template (or `{{.Field}}` - if plan 75 has not yet landed) - - For each name, check key presence in the - file's front-matter map (not empty-value) - - If exact key is missing but a case- - insensitive match exists, emit: - `catalog: field "Title" not found; - did you mean "title"?` - - Hugo users write `.Title`; this catches - muscle-memory errors without false- - positiving on intentionally empty values - -6. Update all overrides in `.mdsmith.yml` that - reference renamed keys. -7. Update `docs/guides/directives/` guides and - `docs/guides/metrics-tradeoffs.md` if they - reference old names. -8. Run `mdsmith check .` to verify. - -## Acceptance Criteria - -- [x] `ratio` renamed to `tokens-per-word` -- [x] `max-words` renamed to - `max-words-per-sentence` -- [x] `max-column-width-variance` renamed to - `max-column-width-ratio` -- [x] `directory-structure: true` without - `allowed` emits a config warning -- [x] Case-mismatched front-matter key in - catalog emits "did you mean?" hint -- [x] `.mdsmith.yml` uses new names throughout -- [x] All rule READMEs use new names -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues -- [x] `mdsmith check .` passes diff --git a/plan/77_template-composition-and-cycles.md b/plan/77_template-composition-and-cycles.md deleted file mode 100644 index d00f4427e..000000000 --- a/plan/77_template-composition-and-cycles.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -id: 77 -title: Template composition and cycle detection -status: "✅" -summary: >- - Allow include directives in schema files for - template composition, add cycle detection for - all include paths, rename template to schema. ---- -# Template composition and cycle detection - -Part of the user-model work from -[plan 73](73_unify-template-directives.md). -Addresses -[#71](https://github.com/jeduden/mdsmith/issues/71) -(cycle detection) and -[#73](https://github.com/jeduden/mdsmith/issues/73) -(Hugo comparison -- composition gap). -Also addresses -[#68](https://github.com/jeduden/mdsmith/issues/68) -(`schema` rename clarifies user model). - -Depends on: none (independent of other plans). -Update plan 74 guide after landing. - -## Goal - -Let schema files use `` to share -structure across templates. Add cycle detection -to all include resolution paths so composition -is safe. Rename the config key from `template` -to `schema` to stop Hugo users reading it as a -rendering template. - -## Context - -Hugo-user blind trials (plan 73) found: - -- 5/5 confused by the word "template" meaning - validation schema, not rendering. -- 5/5 expected composition (Hugo partials and - base templates compose freely). -- Issue #71 notes there is no cycle detection - for include directives today. - -Allowing `` in schema files solves -the composition gap but also introduces a new -cycle risk: schema A includes fragment B which -includes fragment C which includes A. Cycle -detection must land in the same PR. - -## Design - -### Schema composition via `` - -During `parseTemplate`, process `` -directives in the schema file. The included -fragment's headings are spliced into the schema -heading list at the include position. Front -matter from fragments is ignored (only the -root schema's front matter defines the CUE -constraints). `` in a fragment is -merged into the root schema's constraints. - -Example schema (`plan/proto.md`): - -```markdown ---- -id: 'int & >=1' -title: 'string & != ""' ---- - -# ? - -## Goal - -One-sentence summary. - - -``` - -Where `common/acceptance-criteria.md` has: - -```markdown -## Acceptance Criteria - -- [ ] All tests pass: `go test ./...` -``` - -The parsed schema would have headings: -`# ?`, `## Goal`, `## Acceptance Criteria`. - -Fragment files must be in the `ignore:` list -(they are not standalone documents). - -### Cycle detection - -Add a visited-file set to include resolution. -Applies to three paths: - -1. Normal-file `` (MDS021): track - included paths during `fix` and `check`. - If a file appears twice in the chain, emit - `cyclic include: A.md -> B.md -> A.md`. -2. Schema `` (new): track paths - during `parseTemplate`. Same diagnostic. -3. Catalog glob (MDS019): if a matched file - includes (via MDS021) the file containing - the catalog, the catalog body would contain - itself. Detect by checking whether the - catalog-owning file appears in any matched - file's include chain. - -Also add a max depth (default 10) as a safety -net. Depth exceeding the limit is an error even -without a detected cycle. - -### Config key rename: `template` -> `schema` - -Rename the config key in required-structure -settings. No deprecation; update all config -and docs in this PR. - -```yaml -# Before -required-structure: - template: plan/proto.md - -# After -required-structure: - schema: plan/proto.md -``` - -### Diagnostic for misplaced `` - -When `` appears in a file that is -not being parsed as a schema, emit: - -```text -MDS020 is only recognized in -schema files; this directive has no effect here -``` - -## Tasks - -1. ~~Add visited-file tracking to MDS021 include - resolution in `internal/rules/include/`.~~ - Done. -2. ~~Extend `parseTemplate` in - `internal/rules/requiredstructure/rule.go` - to process `` directives.~~ - Done. -3. ~~Add diagnostic for `` in - non-schema files.~~ Done. -4. ~~Rename `template` to `schema` in - required-structure config.~~ Done. -5. ~~Update fixtures.~~ Done. -6. ~~Update docs.~~ Done. -7. ~~Run `mdsmith check .` to verify.~~ Done. - -## Acceptance Criteria - -- [x] Schema files can use `` to - pull in heading fragments -- [x] Cycle detection works for normal-file - includes (direct and indirect) -- [x] Cycle detection works for schema-file - includes -- [x] Max include depth (10) is enforced -- [x] `` in a non-schema file emits - a warning -- [x] Config key is `schema`, not `template` -- [x] All `.mdsmith.yml` overrides updated -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues -- [x] `mdsmith check .` passes diff --git a/plan/79_nested-frontmatter-access.md b/plan/79_nested-frontmatter-access.md deleted file mode 100644 index 123eee33e..000000000 --- a/plan/79_nested-frontmatter-access.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -id: 79 -title: Nested front-matter access -status: "✅" -summary: >- - Use CUE path syntax in {field} placeholders - for nested and quoted front-matter access. ---- -# Nested front-matter access - -Follow-on to the user-model work in -[plan 73](73_unify-template-directives.md). -Extends -[plan 75](75_single-brace-placeholders.md) -(unified `{field}` syntax). - -Depends on: plan 75 (unified `{field}` must -land first so CUE paths extend the shared -interpolation engine). - -## Goal - -`{a.b}` resolves nested front matter. Works -in both catalog rows and schema headings (plan -75 unifies them). Quoted CUE labels handle -non-identifier keys: `{"my-key".sub}`. - -A document with: - -```yaml ---- -params: - subtitle: Overview -my-key: value ---- -``` - -matches `# {params.subtitle}` in a schema and -renders `{params.subtitle}` as `Overview` in -a catalog row. `{"my-key"}` resolves to -`value` in both contexts. - -## Context - -Plan 75 replaces Go `text/template` with a -shared `{field}` interpolation engine for both -catalog and required-structure. This plan -extends that engine with CUE path resolution -for nested maps and quoted keys. - -## Design - -Extend the `fieldinterp` engine (plan 75) to -parse CUE paths inside `{...}` placeholders. - -- `{a.b.c}` resolves nested maps: - `fm["a"].(map)["b"].(map)["c"]` -- `{"my-key".sub}` quotes non-identifier keys - (hyphens, dots, spaces), same as CUE -- `{"a.b"}` is one key with a literal dot, - distinct from `{a.b}` (two nested keys) -- If any step is not a map, emit a diagnostic: - `front-matter key "a.b" is not a map` - -### Why CUE paths - -The tool already uses CUE for front-matter -schema validation. One path grammar everywhere: - -| Context | Syntax | Example | -|---------------------|--------------|--------------------| -| Schema front matter | CUE expr | `'string & != ""'` | -| Schema heading | `{CUE path}` | `{params.title}` | -| Catalog row | `{CUE path}` | `{params.title}` | -| Quoted key | `{"..."}` | `{"my-key".sub}` | - -Same `{...}` syntax, same CUE path resolution, -in every context. No Go template syntax -anywhere in the user-facing surface. - -## Tasks - -1. Update front-matter handling in - `catalog/rule.go` (`readFrontMatter`) and - `requiredstructure/rule.go` - (`readDocFrontMatterRaw`/`stringifyFrontMatter`) - to preserve nested `map[string]any` values. -2. Extend the `fieldinterp` engine (plan 75) - with a `resolveCUEPath` function: - - - Parse CUE path segments (identifiers and - quoted labels) - - Walk nested `map[string]any` - - Return resolved string value or error - -3. Update the placeholder regex in `fieldinterp` - to capture CUE paths: identifiers, dots, - and quoted strings inside `{...}`. -4. Verify CUE schema derivation in - `requiredstructure/rule.go` handles nested - front matter; only adjust if gaps remain. -5. Add unit tests for nested and quoted access - in both catalog and required-structure. -6. Add fixtures with nested front matter. -7. Run `mdsmith check .` to verify. - -## Acceptance Criteria - -- [x] `{a.b}` resolves nested front-matter - in both catalog and schema headings -- [x] `{"my-key".sub}` resolves quoted - non-identifier key in both contexts -- [x] `{"a.b"}` resolves a single key with - a literal dot (CUE quoting) -- [x] Missing nested key emits a diagnostic -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues -- [x] `mdsmith check .` passes diff --git a/plan/80_terminal-recording-readme.md b/plan/80_terminal-recording-readme.md deleted file mode 100644 index 3bfca5e53..000000000 --- a/plan/80_terminal-recording-readme.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -id: 80 -title: "Terminal recording in README" -status: "✅" -summary: "Auto-generate a terminal demo GIF via GitHub Actions and embed it in README.md" ---- -# Terminal recording in README - -## Context - -The README describes mdsmith's features in text, but a -short terminal recording showing the tool in action is -more compelling. The recording must stay current — if -commands or output change, the GIF should update -automatically. - -## Goal - -Embed an auto-generated terminal demo GIF in the root -README, placed right after the intro paragraph. A -GitHub Actions workflow regenerates the recording on -every push to `main`, and PR CI verifies the recording -pipeline works without pushing artifacts. - -## Design - -### Recording tool - -Use [VHS](https://github.com/charmbracelet/vhs) from -Charm. VHS reads a declarative `.tape` file, drives a -headless terminal, and renders to GIF. It runs in CI -without a display server. Add VHS as a Go tool -dependency in `go.mod` so it is invoked via -`go tool vhs` — no separate install step needed. - -### Demo script (`demo.tape`) - -A VHS tape file at the repo root that cycles through -key mdsmith features: - -1. `./mdsmith init` in a temporary directory — shows - config generation without conflicting with the - repo's existing `.mdsmith.yml` -2. `./mdsmith check` on a sample file with lint - errors — shows diagnostic output with source context -3. `./mdsmith fix` on the same file — shows auto-fix -4. `./mdsmith check` again — clean pass, exit 0 -5. `./mdsmith help rule line-length` — shows built-in - rule docs -6. `./mdsmith help rule catalog` — shows catalog rule -7. `./mdsmith help rule directory-structure` — shows - directory-structure rule -8. `./mdsmith help rule required-structure` — shows - required-structure rule -9. `./mdsmith query 'status: "✅"' plan/` — shows - front-matter filtering -10. `./mdsmith metrics rank --by bytes --top 5 .` — - shows metrics - -Each step has a short pause so viewers can read the -output. The tape targets an 80x24 terminal at a -comfortable typing speed. - -### Sample fixture - -A small Markdown file `demo/sample.md` with intentional -lint issues (long line, trailing spaces, missing code -fence language). Kept out of normal lint runs via an -`ignore` entry in `.mdsmith.yml` so `./mdsmith check .` -in CI does not flag it. - -### README placement - -The GIF is embedded immediately after the first -paragraph (the one-liner description), before the -"Why mdsmith" section: - -```markdown -# 🔨 mdsmith - -A fast, auto-fixing Markdown linter ... - -![mdsmith demo](assets/demo.gif) - -## ✨ Why mdsmith -``` - -The `assets/` directory holds the generated GIF. It is -committed to the repo so the image renders on GitHub -without external hosting. - -### Workflows - -**Generate workflow** (`.github/workflows/demo.yml`): -runs on push to `main`. Steps: - -1. Checkout repo -2. Build mdsmith (`go build -o mdsmith ./cmd/mdsmith`) -3. Run `go tool vhs demo.tape` (VHS added as a tool - dependency in `go.mod`) -4. Configure git `user.name` / `user.email` for the - CI bot. If `assets/demo.gif` changed, commit with a - `[skip ci]` marker and push it back to `main`. Add - a loop guard (e.g. skip when `github.actor` is - `github-actions[bot]`) to avoid retriggering the - workflow. Request `permissions: contents: write` so - the `GITHUB_TOKEN` can push. - -This keeps the GIF in sync with the latest CLI output. - -**PR verification** (add a job to `.github/workflows/ci.yml`): -runs on pull requests. Steps: - -1. Checkout repo -2. Build mdsmith -3. Run `go tool vhs demo.tape` -4. Assert `assets/demo.gif` was produced and is a valid - GIF (check file header bytes `GIF89a` or `GIF87a`) -5. Assert file size is within a reasonable range - (> 10 KB, < 5 MB) to catch broken recordings -6. Analyze the GIF content: extract frames, verify - expected command output appears (e.g. grep rendered - text for key strings like `MDS001`, `./mdsmith check`, - `0 issues found`). Use a frame-to-text tool or compare - against a set of reference screenshots to catch - regressions where the GIF renders but shows wrong - or empty output - -The PR job does **not** commit — it only verifies the -pipeline succeeds and the output is sane. - -## Tasks - -1. Create `demo/sample.md` with intentional lint issues - for the demo -2. Write `demo.tape` VHS script that cycles through - init, check, fix, help-rule, query, and metrics - commands -3. Create `assets/` directory with a `.gitkeep` -4. Add the demo GIF embed to `README.md` after the - intro paragraph -5. Create `.github/workflows/demo.yml` that builds - mdsmith, runs VHS, and commits the updated GIF on - pushes to `main` -6. Add a `demo` job to `.github/workflows/ci.yml` that - runs VHS and validates the output GIF on PRs -7. Add `demo/` to the ignore list in `.mdsmith.yml` - (requires explicit user consent per CLAUDE.md) so - the intentionally broken sample file does not fail - `./mdsmith check .` -8. Test the full pipeline locally: run - `go tool vhs demo.tape`, verify the GIF renders - correctly - -## Acceptance Criteria - -- [x] `demo.tape` exists and defines a multi-step demo - covering init, check, fix, help-rule (line-length, - catalog, directory-structure, required-structure), - query, and metrics commands -- [x] `demo/sample.md` contains intentional lint errors - that produce visible diagnostics -- [x] `README.md` embeds `assets/demo.gif` between the - intro paragraph and the "Why mdsmith" section -- [x] `.github/workflows/demo.yml` regenerates the GIF - on push to `main` and commits it if changed -- [x] CI job in `.github/workflows/ci.yml` runs VHS on - PRs and asserts the GIF is valid (file exists, has a - correct GIF header, and falls within a reasonable file - size range) -- [x] `demo/` is excluded from mdsmith linting so the - sample file does not cause CI failures -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no issues diff --git a/plan/81_oom-file-size-limit.md b/plan/81_oom-file-size-limit.md deleted file mode 100644 index 55d39df59..000000000 --- a/plan/81_oom-file-size-limit.md +++ /dev/null @@ -1,168 +0,0 @@ ---- -id: 81 -title: 'OOM mitigation: configurable file-size limit' -status: "✅" -summary: >- - Guard every file-read path against OOM by enforcing - a configurable byte-size cap (default 2 MB). ---- -# OOM mitigation: configurable file-size limit - -## Goal - -Prevent out-of-memory crashes from large Markdown -files. Add a configurable byte-size limit enforced -before any file content is loaded into memory. - -## Background - -`os.ReadFile` and `io.ReadAll` load the entire input -into a byte slice with no size guard. A multi-GB `.md` -file (or stdin pipe) will OOM the process. The -`max-file-length` rule (MDS022) only emits a diagnostic -*after* the file is fully loaded — it does not prevent -the allocation. - -Every top-5 Markdown linter (Prettier, markdownlint, -Vale, remark-lint, textlint) has the same gap. remark's -docs advise callers to "cap input at 500 KB" but -nothing is enforced. - -## Design - -### Shared helper - -New file `internal/lint/limits.go`: - -```go -const DefaultMaxInputBytes int64 = 2 * 1024 * 1024 - -func ReadFileLimited(path string, max int64) ([]byte, error) -func ReadFSFileLimited(fsys fs.FS, name string, max int64) ([]byte, error) -``` - -Both use `Open` + `io.LimitReader(f, max+1)` + -`io.ReadAll` + post-read `len(data) > max` check. -The `+1` sentinel distinguishes "exactly at limit" -from "truncated". When `max <= 0`, no limit is applied -(unlimited mode). - -### Configuration - -Top-level key in `.mdsmith.yml` (not a rule setting — -this is infrastructure, not a lint rule): - -```yaml -max-input-size: 2MB -``` - -New `MaxInputSize` field on `config.Config`: - -```go -MaxInputSize string `yaml:"max-input-size"` -``` - -### Size-string parser - -New file `internal/config/size.go`: - -```go -func ParseSize(s string) (int64, error) -``` - -Accepted formats: `2MB`, `500KB`, `1GB`, bare integer -(bytes), `0` (unlimited). Case-insensitive. Uses -binary units (1 MB = 1,048,576 bytes). - -### CLI flag - -```text ---max-input-size -``` - -Added to both `check` and `fix` flag sets. CLI value -overrides the config value. Default: `"2MB"`. - -### Threading the limit - -Add `MaxInputBytes int64` field to `engine.Runner` and -`fix.Fixer`. Set from the parsed config + CLI override -in `cmd/mdsmith/main.go`. - -### Read sites to guard - -Primary entry points (replace `os.ReadFile` with -`lint.ReadFileLimited`): - -1. `internal/engine/runner.go:52` -2. `internal/fix/fix.go:84` -3. `cmd/mdsmith/main.go:577` (stdin via - `io.LimitReader`) - -Secondary read sites (replace with -`lint.ReadFileLimited` or `lint.ReadFSFileLimited`): - -4. `internal/rules/include/rule.go:194` -5. `internal/rules/catalog/rule.go:395,468` -6. `internal/rules/crossfilereferenceintegrity/rule.go:236,253` -7. `internal/rules/requiredstructure/rule.go:82,496` -8. `internal/metrics/rank.go:21` -9. `cmd/mdsmith/mergedriver.go:96,135` -10. `internal/config/load.go:16` - -Rules that read files need the limit threaded via -`lint.File` or a new field on the rule struct (set -during `ApplySettings` or via the runner). - -### Error behavior - -- `check`/`fix`: Emit error, skip file, continue. - Exit code 2. -- `stdin`: Print to stderr, exit 2. -- Message format: - `reading "huge.md": file too large (15728640 bytes, max 2097152)` - -## Tasks - -1. [x] Add `internal/lint/limits.go` with - `ReadFileLimited` and `ReadFSFileLimited` -2. [x] Add `internal/lint/limits_test.go` with tests for - normal, at-limit, over-limit, zero (unlimited), - and empty-file cases -3. [x] Add `internal/config/size.go` with `ParseSize` -4. [x] Add `internal/config/size_test.go` with tests for - `2MB`, `500KB`, `0`, bare integer, invalid input -5. [x] Add `MaxInputSize` field to `config.Config` -6. [x] Add `MaxInputBytes` field to `engine.Runner` and - `fix.Fixer` (and `lint.File` for rule threading) -7. [x] Add `--max-input-size` flag to `check` and `fix` - subcommands -8. [x] Replace `os.ReadFile` with `ReadFileLimited` at all - primary read sites (runner, fixer, stdin) -9. [x] Replace `os.ReadFile` / `fs.ReadFile` at all - secondary read sites (include, catalog, - cross-file-ref, required-structure, metrics, merge - driver, config) -10. [x] Thread `MaxInputBytes` to rules that read files - (via `lint.File` field) -11. [x] Add integration test: file exceeding limit produces - error diagnostic and exit code 2 -12. [x] Document `max-input-size` in `docs/reference/cli.md` - -## Acceptance Criteria - -- [x] `ReadFileLimited` returns error for files - exceeding the configured limit -- [x] `ReadFileLimited` succeeds for files at or below - the limit (no off-by-one) -- [x] `ReadFileLimited` with `max <= 0` applies no - limit (unlimited mode) -- [x] `ParseSize` handles `2MB`, `500KB`, `1GB`, bare - integers, and `0` -- [x] `.mdsmith.yml` `max-input-size` key is respected -- [x] `--max-input-size` CLI flag overrides config -- [x] `--max-input-size 0` disables the limit -- [x] All ~15 read sites use the limited helper -- [x] Error message includes actual size and limit -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no issues diff --git a/plan/82_yaml-billion-laughs.md b/plan/82_yaml-billion-laughs.md deleted file mode 100644 index 8d6808c03..000000000 --- a/plan/82_yaml-billion-laughs.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -id: 82 -title: 'YAML billion-laughs mitigation' -status: "✅" -summary: >- - Reject YAML anchor/alias syntax in user-supplied - content before unmarshalling to prevent exponential - memory expansion. ---- -# YAML billion-laughs mitigation - -## Goal - -Prevent OOM from YAML front matter or directive bodies -that use anchor/alias expansion. These create -exponential memory growth during `yaml.Unmarshal`. - -## Background - -`gopkg.in/yaml.v3` has no alias-expansion limit. -A 1 KB YAML with 8 levels of nested aliases can expand -to 10^8 strings. Byte-length caps on input do not -prevent this — the attack uses small input. - -Prettier and remark-lint use `eemeli/yaml` which -defaults to `maxAliasCount: 100`. Vale uses -`yaml.v2` v2.4.0 which back-ported alias-depth fixes. -markdownlint avoids the issue by not parsing YAML at -all. textlint (js-yaml v4) is vulnerable like mdsmith. - -Legitimate Markdown front matter virtually never uses -YAML anchors or aliases. - -## Design - -### Pre-scan approach - -Before any `yaml.Unmarshal` call on user-supplied -content, scan the raw bytes for YAML anchor (`&`) or -alias (`*`) characters. If found, return an error -diagnostic rather than proceeding to unmarshal. - -```go -func RejectYAMLAliases(data []byte) error { - if bytes.ContainsAny(data, "&*") { - return fmt.Errorf( - "YAML anchors/aliases are not permitted") - } - return nil -} -``` - -### Refinement: context-aware scan - -A bare `&` or `*` can appear in YAML string values -(e.g., `title: "Q&A"`). To reduce false positives, -only reject when `&` or `*` appears in a YAML -structural position: - -- `&` followed by a YAML identifier (anchor - definition): pattern `&\w` -- `*` followed by a YAML identifier (alias reference): - pattern `\*\w` at the start of a value - -Use a simple regex or byte scan, not a full YAML -parser. - -### Call sites to guard - -All `yaml.Unmarshal` sites processing user-supplied -content (13 total): - -1. `internal/archetype/gensection/parse.go:189` - (directive YAML body) -2. `internal/rules/catalog/rule.go:415` - (per-file front matter) -3. `internal/rules/requiredstructure/rule.go:220,233,556,924` - (schema front matter, require directives) -4. `cmd/mdsmith/main.go:352` - (`query` subcommand front matter) -5. `internal/config/load.go:22,35` - (config file — lower risk, operator-controlled) -6. `internal/corpus/config.go:35,82` - (corpus config — internal tooling) - -Sites 1–4 are high priority (user-supplied `.md` -content). Sites 5–6 are lower priority (operator- -controlled config files) but should be guarded for -defense in depth. - -### Alternative considered - -Use `goccy/go-yaml` with `yaml.WithMaxAliasesNum(100)` -to mirror `eemeli/yaml`. This is a larger dependency -change. Defer unless the pre-scan produces false -positives. - -## Tasks - -1. [x] Add `internal/lint/yamlsafe.go` with - `RejectYAMLAliases(data []byte) error` -2. [x] Add `internal/lint/yamlsafe_test.go` with tests - for clean YAML, anchor YAML, alias YAML, `Q&A` - in string values (false positive check) -3. [x] Guard all 11 `yaml.Unmarshal` call sites with a - `RejectYAMLAliases` check before unmarshalling -4. [x] Add integration test: `.md` file with YAML anchor - front matter produces a clear error diagnostic - -## Acceptance Criteria - -- [x] YAML with `&anchor` / `*alias` is rejected - before unmarshalling -- [x] Legitimate front matter with `&` in string - values (e.g., `"Q&A"`) is accepted -- [x] All 11 `yaml.Unmarshal` sites are guarded -- [x] Error message clearly states anchors/aliases - are not permitted -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no issues diff --git a/plan/87_markdown-flavor-github-alerts.md b/plan/87_markdown-flavor-github-alerts.md deleted file mode 100644 index caa9419e5..000000000 --- a/plan/87_markdown-flavor-github-alerts.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -id: 87 -title: Flavor validation for GitHub Alerts -status: "✅" -summary: >- - Extend MDS034 to detect GitHub Alerts syntax - (`> [!NOTE]` blockquote prefix) as a GFM-only - feature with auto-fix that demotes the alert to - a plain blockquote when the target flavor does - not support it. ---- -# Flavor validation for GitHub Alerts - -Extends [plan 86](86_markdown-flavor-validation.md) -(MDS034, flavor validation). Add one feature — -GitHub Alerts — to the MDS034 feature enum. - -Depends on: plan 86 lands first (provides the -dual parser, feature enum, fix pipeline). - -## Goal - -MDS034 flags `> [!NOTE]`-style alert blockquotes -when the target flavor is `commonmark` or -`goldmark`. `gfm` accepts them. Auto-fix demotes -the alert marker so the blockquote still renders -on non-GFM renderers. - -## Context - -GitHub added Alerts to GFM in December 2023 -(see the `github.blog` changelog entry for -`new-syntax-for-alerts-on-github`). Five tokens -are recognized: `[!NOTE]`, `[!TIP]`, -`[!IMPORTANT]`, `[!WARNING]`, `[!CAUTION]`. -Obsidian callouts use the same prefix and accept -extra tokens, but only these five are standard -GFM. - -On CommonMark / goldmark-default, the marker -renders as literal text inside a blockquote: - -```markdown -> [!NOTE] -> Something to remember. -``` - -becomes a blockquote whose first line is the -literal string `[!NOTE]`. The author intended a -styled callout; the reader sees unstyled text -with the marker token visible inside the -blockquote. The failure is visible, not silent, -but the author's intent is still lost. - -### Why not a generic container rule - -The research spike evaluated four other -container syntaxes (Pandoc `:::` fenced divs, -MyST `:::{note}`, markdown-it-container, MkDocs -`!!! note`). None are mutually compatible and no -linter in the comparison covers them. GitHub -Alerts are the only variant with a standardized -spec, broad renderer support, and a clear -failure mode — so this plan covers them alone. - -## Design - -### Detection - -GitHub Alerts need no new goldmark extension. -The syntax is a plain Blockquote. Its first -paragraph text must match -`^\[!(NOTE|TIP|IMPORTANT|WARNING|CAUTION)\]\s*$` -(case-sensitive per GFM). - -Detection is an AST walk over `ast.Blockquote` -nodes on the dual parser's tree. The same walk -pattern the other 12 features use. - -### Configuration - -No new settings. GitHub Alerts join the existing -feature enum in -`internal/rules/markdownflavor/features.go` as -feature 13 (`GitHubAlerts`). Flavor support: - -| Flavor | GitHub Alerts | -|------------|---------------| -| commonmark | unsupported | -| gfm | supported | -| goldmark | unsupported | - -### Auto-fix - -Remove the `[!TOKEN]` marker line, keeping the -rest of the blockquote intact: - -```markdown -> [!NOTE] > Something to -> Something to → > remember. -> remember. -``` - -If the alert marker is the only line in the -blockquote, remove the whole blockquote. The -marker line has no meaningful content once the -token is gone. - -### Error message - -`github alerts are not supported by {flavor}` - -Severity: `warning`, matching the other MDS034 -features. - -## Tasks - -1. [x] Add `GitHubAlerts` to the feature enum in - `internal/rules/markdownflavor/features.go` -2. [x] Add flavor support table entry: supported in - `gfm`, unsupported in `commonmark` and - `goldmark` -3. [x] Implement an AST detector that walks - `ast.Blockquote` nodes and matches the five - GFM tokens on the first paragraph child -4. [x] Implement the fix: strip the marker line; - drop the blockquote if empty afterward -5. [x] Add unit tests: each of the five tokens, - lower-case tokens (should not match), mixed - content after the marker, marker as the only - line -6. [x] Add bad/fixed fixtures under - `internal/rules/MDS034-markdown-flavor/` -7. [x] Update the MDS034 README to list GitHub - Alerts as the 13th feature - -## Acceptance Criteria - -- [x] `flavor: commonmark` flags all five alert - tokens -- [x] `flavor: goldmark` flags all five alert - tokens -- [x] `flavor: gfm` accepts all five tokens -- [x] `mdsmith fix` removes the marker line, - preserves remaining blockquote content -- [x] `mdsmith fix` removes the whole blockquote - when the marker was its only line -- [x] Lower-case or unknown tokens (e.g. - `[!note]`, `[!INFO]`) produce no - diagnostic — they are ordinary blockquote - text -- [x] Nested blockquotes are checked recursively -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues diff --git a/plan/88_toc-directive-migration.md b/plan/88_toc-directive-migration.md deleted file mode 100644 index 7509d47e1..000000000 --- a/plan/88_toc-directive-migration.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -id: 88 -title: TOC directive migration aid -status: "✅" -summary: >- - New rule MDS035 that flags renderer-specific - table-of-contents directives (`[TOC]`, - `[[_TOC_]]`, `[[toc]]`, `${toc}`) which render - as literal text on CommonMark / goldmark - instead of expanding into a TOC. The - diagnostic points authors at mdsmith's - `` directive for the file-index use - case; heading-level TOCs have no direct - mdsmith equivalent. ---- -# TOC directive migration aid - -## Goal - -Catch renderer-specific TOC directives that do -not expand into a TOC on CommonMark or -goldmark. The diagnostic tells authors which -use case has a mdsmith equivalent and which -does not. - -## Context - -Four TOC directive variants appear in the wild: - -- `[TOC]` — Python-Markdown, MultiMarkdown, - Pandoc (with `--toc`) -- `[[_TOC_]]` — GitLab Flavored Markdown, - Azure DevOps -- `[[toc]]` — markdown-it-toc-done-right, - VitePress -- `${toc}` — some VitePress configurations - -None are standardized TOC directives in -CommonMark, GFM, or goldmark. `[TOC]` is still -valid CommonMark syntax — a shortcut reference -link — so it only renders as literal text when -no matching `[TOC]: ` definition is -present; with a definition, it renders as a -link. The other three tokens are not valid -link-reference shorthand and always render as -literal text. On those renderers no token in -this set expands into a generated TOC. - -Concrete render outputs when no link reference -is present: - -- `[TOC]` renders as the literal string `[TOC]` - (goldmark emits a "no matching link - reference" fallback, which is verbatim text) -- `[[_TOC_]]` renders as `[[_TOC_]]` inside a - paragraph -- `[[toc]]` renders as `[[toc]]` inside a - paragraph -- `${toc}` renders as `${toc}` inside a - paragraph - -The author intended a generated table of -contents; the reader sees the directive token -instead. This is a visible failure, not a -silent one, but it is still a failure worth -catching at lint time. - -### Heading TOC vs file index - -The flagged directives and -[``][catalog] solve different -problems: - -| Directive | Generates | Input | -|----------------|-------------------------------------------|--------------------| -| `[TOC]` et al. | Table of **headings in the current file** | Current doc | -| `` | Table of **other files** matching a glob | Glob + frontmatter | - -[catalog]: ../internal/rules/MDS019-catalog/README.md - -`` is the right replacement only -when a directive is used on an index page to -list sibling or child documents (e.g. a wiki -homepage with `[[_TOC_]]` listing all pages in -the space). For in-document heading TOCs — the -more common case — mdsmith has no built-in -generator; the author must either drop the -directive or maintain a manual list. - -### Why this rule, not MDS034 - -MDS034 ([plan 86](86_markdown-flavor-validation.md)) -validates syntax support against a declared -flavor. TOC directives are not "flavor features" -— they are per-renderer conventions with no -canonical spec and no fix path that applies to -every call site. A dedicated opt-in rule with a -diagnostic tailored to the use-case distinction -above is a better fit than folding them into -MDS034's fix pipeline. - -### Scope - -Flag only the four directives above. Do not try -to auto-generate a `` block — the -right glob and front-matter fields depend on the -project and are not knowable from the TOC call -site. The diagnostic is informational and names -both the file-index case (points to MDS019) and -the heading-TOC case (no equivalent). - -## Design - -### Detection - -Line-level regex on the raw source, scoped to -paragraph nodes (skip code blocks, HTML blocks, -and inline code spans). Four patterns: - -- `^\[TOC\]\s*$` -- `^\[\[_TOC_\]\]\s*$` -- `^\[\[toc\]\]\s*$` -- `^\$\{toc\}\s*$` - -Goldmark parses `[TOC]` as a link reference node -and `[[_TOC_]]` / `[[toc]]` as text inside a -paragraph. AST detection would require -per-variant walkers; raw-line regex is simpler -and avoids false positives by restricting the -match to paragraph-only regions. - -#### Link reference exception for `[TOC]` - -`[TOC]` is syntactically a valid CommonMark -shortcut reference link. If the document -contains a matching link reference definition -(`[TOC]: `), `[TOC]` resolves to a -legitimate link and must not be flagged. - -Before emitting a diagnostic for the `[TOC]` -pattern, consult the goldmark parser context's -link reference map for a definition with the -label `TOC` (case-insensitive, per the -[CommonMark matching rules][cm-refs]). If one -exists, suppress the diagnostic. - -[cm-refs]: https://spec.commonmark.org/0.31.2/#matches - -The other three patterns do not have this -ambiguity: `[[_TOC_]]`, `[[toc]]`, and `${toc}` -do not form valid link references in CommonMark -and always render as literal text in a -paragraph. No exception handling is needed for -them. - -### Configuration - -Rule `toc-directive`, category `meta`, disabled -by default (opt-in) — consistent with MDS034's -opt-in posture. No settings. - -### Error message - -Templated on the matched directive token, so -each variant produces its own message. Given a -matched token `{directive}`: - -```text -unsupported TOC directive `{directive}`; mdsmith has no heading TOC equivalent; use `` for file indexes (MDS019) -``` - -Example messages the rule would emit for each -of the four patterns: - -```text -unsupported TOC directive `[TOC]`; mdsmith has no heading TOC equivalent; use `` for file indexes (MDS019) -unsupported TOC directive `[[_TOC_]]`; mdsmith has no heading TOC equivalent; use `` for file indexes (MDS019) -unsupported TOC directive `[[toc]]`; mdsmith has no heading TOC equivalent; use `` for file indexes (MDS019) -unsupported TOC directive `${toc}`; mdsmith has no heading TOC equivalent; use `` for file indexes (MDS019) -``` - -The leading word is lowercase. No trailing -punctuation, per [CLAUDE.md](../CLAUDE.md). -Both the matched token and the replacement -`` are backticked so they read as -quoted directives, not prose. - -Severity: `warning`. - -### No auto-fix - -The rule is detection-only. Whether the right -replacement is ``, a manually -maintained list, or deletion depends on intent -that is not recoverable from the directive -alone. - -## Tasks - -1. [x] Create `internal/rules/MDS035-toc-directive/` - with `README.md`, and the implementation in - `internal/rules/tocdirective/rule.go` -2. [x] Implement paragraph-scoped line scanning - for the four directive patterns -3. [x] For the `[TOC]` pattern, consult the - goldmark parser context's link reference - definition map; suppress the diagnostic when - a label `TOC` (case-insensitive) is defined -4. [x] Implement `rule.Defaultable` with - `EnabledByDefault` returning `false` -5. [x] Register as MDS035 in category `meta` -6. [x] Add good/bad fixtures with front-matter - specifying the expected diagnostics, including - a good fixture that has `[TOC]: https://x` - as a reference definition alongside a `[TOC]` - line -7. [x] Document the rule in the renderer - portability section in - [docs/background/markdown-linters.md](../docs/background/markdown-linters.md) - -## Acceptance Criteria - -- [x] `[TOC]` on its own line produces a - diagnostic that names both the heading-TOC - gap and the `` file-index - alternative -- [x] `[[_TOC_]]` on its own line produces the - same diagnostic -- [x] `[[toc]]` on its own line produces the - same diagnostic -- [x] `${toc}` on its own line produces the - same diagnostic -- [x] `[TOC]` inside a fenced code block - produces no diagnostic -- [x] `[TOC]` inside an inline code span - produces no diagnostic -- [x] `[TOC]` used as legitimate link text - (with a matching `[TOC]: url` definition) - produces no diagnostic -- [x] Rule is disabled by default (opt-in) -- [x] No auto-fix is applied -- [x] All tests pass: `go test ./...` -- [x] `go tool golangci-lint run` reports no - issues diff --git a/plan/91_mds037-skip-generated-sections.md b/plan/91_mds037-skip-generated-sections.md index a377b44fe..de86e4291 100644 --- a/plan/91_mds037-skip-generated-sections.md +++ b/plan/91_mds037-skip-generated-sections.md @@ -6,12 +6,12 @@ summary: >- Paragraphs inside `` and `` directive bodies are copies of content owned by another file; MDS037 must not count them as - cross-file duplicates. Extends plan 50. + cross-file duplicates. --- # MDS037 skips paragraphs inside generated sections -Extends [plan 50](50_redundancy-duplication-detection.md) -(MDS037 duplicated-content). +Builds on redundancy detection work to skip paragraphs +inside generated sections (MDS037 duplicated-content). ## Goal