Merge PR #709: Add opt-in reflow auto-fix to line-length (MDS001)

Author: merge-queue-bot

docs/background/markdown-linters.md

@@ -665,7 +665,7 @@ lack determinism.
 | ------------------ | --------------------- | ------------------------ | ------------ | --------------- | ---- | --------------------- | ----------------- |
 | Autofix CLI        | `fix`                 | `--write`                | `--fix`      | `--fix` / `fmt` | no   | `format`              | yes (AST rewrite) |
 | Table alignment    | [MDS025][mds025]      | yes                      | no           | MD055/56/58     | no   | yes                   | via plugin        |
-| Prose wrapping     | no                    | [`proseWrap`][prosewrap] | no           | no              | no   | no                    | no                |
+| Prose wrapping     | opt-in (reflow)       | [`proseWrap`][prosewrap] | no           | no              | no   | no                    | no                |
 | Embedded code fmt  | no                    | JS/TS/CSS/JSON           | no           | no              | no   | delegates to external | no                |
 | Multi-pass fix     | yes                   | single pass              | single pass  | single pass     | no   | single pass           | single pass       |
 | Generated sections | catalog, include, toc | no                       | no           | no              | no   | no                    | no                |
@@ -678,9 +678,11 @@ line per paragraph), or `preserve` (leave as-is, the
 default). remark-lint has no prose-wrap setting, but it
 serializes through its AST when fixing, so paragraphs
 can be incidentally rewrapped to match its stringify
-defaults. The others — mdsmith, markdownlint, rumdl,
-mado, panache — diagnose long lines but preserve the
-existing breaks.
+defaults. mdsmith adds an opt-in reflow fix
+(`line-length.reflow`). It rewraps prose to the width
+you set, and leaves breaks alone otherwise.
+markdownlint, rumdl, mado, and panache flag long
+lines but keep the existing breaks.
 
 Prettier is the strongest pure formatter. rumdl and
 panache bring native autofix to the Rust side; mado is

docs/features/auto-fix.md

@@ -21,6 +21,13 @@ rule, re-parses, and repeats until the document stops changing or
 it has run ten passes. Stabilization means one fix never undoes
 another.
 
+Paragraph reflow is the one fix that stays off by default. Set
+`rules.line-length.reflow: true` to let `mdsmith fix` rewrap
+over-long prose paragraphs to the `max` width. The wrap is
+abbreviation-aware — it never breaks `e.g.` or initials such as
+`J. R. R. Tolkien` across lines — and it leaves headings, lists,
+tables, code, and generated sections untouched.
+
 `mdsmith check` runs the same rules without writing. It is the
 read-only sibling for CI, returning a non-zero exit code when any
 rule fails so a pipeline can block the merge.

docs/guides/coexist-with-prettier.md

@@ -83,9 +83,16 @@ in `.mdsmith.yml`). Both indent with two spaces
 (`tabWidth` and `rules.list-indent.spaces`).
 Prettier's default `proseWrap: "preserve"` leaves
 existing line breaks alone. mdsmith's `line-length`
-rule reports long lines but does not rewrap them.
-At defaults, neither tool reflows paragraphs. Long
-lines are yours to wrap by hand.
+rule reports long lines but does not rewrap them
+unless you opt in. At defaults, neither tool
+reflows paragraphs. Long lines are yours to wrap by
+hand.
+
+Set `rules.line-length.reflow: true` to let
+`mdsmith fix` rewrap prose paragraphs to `max`. Keep
+Prettier on its default `proseWrap: "preserve"` so
+the two tools do not fight over the same line
+breaks.
 
 If you set `proseWrap: "always"`, Prettier rewraps
 paragraphs to `printWidth`. Keep `printWidth` no

internal/punkt/abbrtoken_test.go

@@ -0,0 +1,50 @@
+package punkt
+
+import "testing"
+
+func TestIsAbbrevToken(t *testing.T) {
+	s := NewEnglish().Storage
+	cases := []struct {
+		tok  string
+		want bool
+	}{
+		{"J.", true},        // single-letter initial
+		{"e.g.", true},      // dotted abbreviation pattern
+		{"i.e.", true},      // dotted abbreviation pattern
+		{"U.S.A.", true},    // dotted abbreviation pattern
+		{"Dr.", true},       // trained AbbrevTypes (dr)
+		{"vs.", true},       // trained AbbrevTypes (vs)
+		{"Mr.", true},       // trained AbbrevTypes (mr)
+		{"e.g.,", true},     // trailing comma ignored
+		{"Dr.;", true},      // trailing semicolon ignored
+		{"cat.", false},     // ordinary word ending a sentence
+		{"Go.", false},      // ordinary word ending a sentence
+		{"plain", false},    // no trailing period
+		{"e.g", false},      // no trailing period, not in AbbrevTypes
+		{",", false},        // trims to empty
+		{"", false},         // empty
+		{"unknown.", false}, // period-final but untrained
+	}
+	for _, c := range cases {
+		if got := s.IsAbbrevToken(c.tok); got != c.want {
+			t.Errorf("IsAbbrevToken(%q) = %v, want %v", c.tok, got, c.want)
+		}
+	}
+}
+
+// TestIsAbbrevToken_SyntheticStorage drives the AbbrevTypes branch with
+// a hermetic Storage so the trained-asset cases above are not the only
+// coverage of the lowercase/drop-period lookup.
+func TestIsAbbrevToken_SyntheticStorage(t *testing.T) {
+	s := NewStorage()
+	s.AbbrevTypes.Add("approx")
+	if !s.IsAbbrevToken("approx.") {
+		t.Errorf("IsAbbrevToken(approx.) should be true for a seeded type")
+	}
+	if !s.IsAbbrevToken("Approx.") {
+		t.Errorf("IsAbbrevToken(Approx.) should lowercase before lookup")
+	}
+	if s.IsAbbrevToken("approxx.") {
+		t.Errorf("IsAbbrevToken(approxx.) should be false; not a seeded type")
+	}
+}

internal/punkt/storage.go

@@ -2,6 +2,8 @@ package punkt
 
 import (
 	"encoding/json"
+	"strings"
+	"unicode/utf8"
 )
 
 // SetString is a string-keyed set matching upstream's JSON shape:
@@ -69,6 +71,38 @@ func (s *Storage) IsAbbr(tokens ...string) bool {
 	return false
 }
 
+// abbrTokenCutset is the trailing clause punctuation IsAbbrevToken
+// ignores, so "e.g.," and "cf.;" are judged like "e.g." and "cf.".
+const abbrTokenCutset = ",;:"
+
+// IsAbbrevToken reports whether tok — a single whitespace-delimited
+// word such as "Dr.", "e.g.", or "J." — is an abbreviation under this
+// trained model, without running the full sentence tokenizer. It
+// recognises three shapes the pipeline's type-annotation pass treats as
+// abbreviations: a single-letter initial ("J."), a dotted abbreviation
+// pattern ("e.g.", "U.S.A."), and a member of the trained AbbrevTypes
+// set ("dr", "vs", "no"). Trailing clause punctuation is ignored, and
+// the AbbrevTypes lookup drops the final period and lowercases the word,
+// matching how typeAnnotation keys the set.
+//
+// It exists for callers — line-length reflow is the first — that need
+// the model's per-token abbreviation judgement to decide a wrap, not a
+// sentence boundary.
+func (s *Storage) IsAbbrevToken(tok string) bool {
+	t := strings.TrimRight(tok, abbrTokenCutset)
+	if t == "" {
+		return false
+	}
+	if isInitial(t) || MatchAbbrPattern(t) {
+		return true
+	}
+	if !HasPeriodFinal(t) {
+		return false
+	}
+	_, sz := utf8.DecodeLastRuneInString(t)
+	return s.IsAbbr(strings.ToLower(t[:len(t)-sz]))
+}
+
 // addOrthoContext sets the ortho flag for typ. Used by training
 // loaders; kept here only because tests in this package construct
 // Storage values and seed OrthoContext directly.

internal/rules/MDS001-line-length/README.md

@@ -42,6 +42,8 @@ Line exceeds maximum length.
 | `code-block-max` | int  | --                                  | Max length for code block lines; inherits `max` when unset |
 | `stern`          | bool | false                               | Only flag long lines that contain a space past the limit   |
 | `exclude`        | list | `["code-blocks", "tables", "urls"]` | Categories to exclude from checking                        |
+| `reflow`         | bool | false                               | Auto-fix over-long prose paragraphs by rewrapping to `max` |
+| `abbreviations`  | list | `[]`                                | Extra abbreviations the reflow fixer keeps unbroken        |
 
 Valid `exclude` values:
 
@@ -76,6 +78,44 @@ is excluded via `exclude: [code-blocks]` is still skipped regardless of stern.
 Stern uses the active max for each line type, so it respects `heading-max` and
 `code-block-max` when set.
 
+### Reflow (auto-fix)
+
+By default the rule reports long lines but does not rewrite them: `mdsmith fix`
+leaves prose untouched. Set `reflow: true` to opt into the auto-fix, which
+rewraps over-long top-level prose paragraphs so every line fits within `max`.
+
+Reflow is deliberately conservative. It rewraps only paragraphs that sit
+directly in the document body and contain a flagged long line. It skips
+headings, list items, block quotes, tables, generated sections, paragraphs that
+carry a Markdown hard line break, and paragraphs that contain inline raw HTML.
+Inline code spans are preserved verbatim, and a single word wider than `max` (a
+long URL or link) keeps its own over-long line rather than being broken.
+
+Wrapping is abbreviation-aware. It never ends a wrapped line on an abbreviation,
+and it never splits a run of initials. So `e.g.` stays with the word it
+introduces, and `J. R. R. Tolkien` is never broken across lines. Detection
+reuses mdsmith's trained abbreviation model — the same one the readability rules
+use to split sentences. It recognises honorifics (`Dr.`, `Mr.`), reference forms
+(`vs.`, `No.`), initials (`J.`), and dotted forms (`e.g.`, `i.e.`, `U.S.A.`). The
+`abbreviations` setting adds project-specific entries the model does not know
+(`etc.`, `approx.`), and append-merges across config layers so a kind can extend
+the inherited list without restating it.
+
+The payoff is clearest at the wrap boundary. A naive word wrap of this line at 80
+columns splits the spaced acronym across the break:
+
+```text
+Historians traced the founding and the early constitutional debates of the U. S.
+A. with real care.
+```
+
+Reflow breaks before the acronym instead, keeping `U. S. A.` whole on one line:
+
+```text
+Historians traced the founding and the early constitutional debates of the
+U. S. A. with real care.
+```
+
 ## Config
 
 Enable (default):
@@ -128,6 +168,18 @@ rules:
     exclude: []
 ```
 
+Custom (enable the reflow auto-fix and add project abbreviations):
+
+```yaml
+rules:
+  line-length:
+    max: 80
+    reflow: true
+    abbreviations:
+      - etc.
+      - approx.
+```
+
 Custom (skip only code blocks and URLs; check tables):
 
 ```yaml
@@ -269,7 +321,7 @@ This line inside a code block is over 80 characters but within the code-block-ma
 - **Name**: `line-length`
 - **Status**: ready
 - **Default**: enabled, max: 80
-- **Fixable**: no
+- **Fixable**: yes (opt-in via `reflow`)
 - **Implementation**:
   [source](./)
 - **Category**: line

internal/rules/MDS001-line-length/bad/reflow-abbreviations.md

@@ -0,0 +1,11 @@
+---
+settings:
+  reflow: true
+diagnostics:
+  - line: 3
+    column: 81
+    message: "line too long (99 > 80)"
+---
+# Initials
+
+Historians traced the founding and the early constitutional debates of the U. S. A. with real care.

internal/rules/MDS001-line-length/bad/reflow.md

@@ -0,0 +1,11 @@
+---
+settings:
+  reflow: true
+diagnostics:
+  - line: 3
+    column: 81
+    message: "line too long (135 > 80)"
+---
+# Reflow
+
+The tool wraps each long line of prose. It makes the text fit the width you set. It does not change the way the page looks to a reader.

internal/rules/MDS001-line-length/fixed/reflow-abbreviations.md

@@ -0,0 +1,4 @@
+# Initials
+
+Historians traced the founding and the early constitutional debates of the
+U. S. A. with real care.

internal/rules/MDS001-line-length/fixed/reflow.md

@@ -0,0 +1,4 @@
+# Reflow
+
+The tool wraps each long line of prose. It makes the text fit the width you set.
+It does not change the way the page looks to a reader.