Add configurable file-size limit to prevent OOM

[jeduden]

Summary

Implements a configurable file-size limit across all file-read paths in mdsmith to prevent out-of-memory (OOM) errors when processing unexpectedly large files. The default limit is 2 MB, configurable via .mdsmith.yml or the --max-input-size CLI flag.

Key Changes

• New size-limit helpers (internal/lint/limits.go):

  • ReadFileLimited() and ReadFSFileLimited() enforce byte-size caps on file reads
  • Zero or negative limits disable the cap (unlimited mode)
  • Error messages include actual file size and configured limit

• Size parsing (internal/config/size.go):

  • ParseSize() parses human-readable sizes: 2MB, 500KB, 1GB, bare integers, or 0
  • Case-insensitive, uses binary units (1 MB = 1,048,576 bytes)

• Configuration:

  • Added MaxInputSize field to config.Config
  • Added MaxInputBytes field to engine.Runner, fix.Fixer, and lint.File
  • Config file reads capped at 1 MB to prevent accidental OOM from config itself

• CLI integration:

  • New --max-input-size flag on check and fix subcommands
  • CLI flag overrides config file setting
  • Default: 2MB

• Comprehensive coverage:

  • Replaced ~15 os.ReadFile() and fs.ReadFile() calls with limited variants
  • Affected: runner, fixer, stdin, includes, catalog, cross-file references, required-structure, metrics, merge driver, and config loading
  • Rules access limit via lint.File.MaxInputBytes field

• Testing:

  • Unit tests for ReadFileLimited, ReadFSFileLimited, and ParseSize
  • E2E tests: file exceeding limit produces error diagnostic and exit code 2
  • Config override test: CLI flag overrides config setting

Implementation Details

• Files exceeding the limit are rejected with exit code 2 and a clear error message
• The +1 sentinel in readLimited() distinguishes "exactly at limit" from "truncated"
• When available, actual file size is reported in error messages (via Stat())
• Unlimited mode (max <= 0) bypasses all limit checks for performance
• Config merge logic refactored to use new copyConfig() helper for consistency

https://claude.ai/code/session_01HHssHCn4zfbMv1praKoxGJ

[codecov]

Codecov Report

❌ Patch coverage is 85.41033% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.35%. Comparing base (1bde75e) to head (ece99e8).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/mdsmith/main.go	86.88%	10 Missing and 6 partials ⚠️
cmd/mdsmith/mergedriver.go	54.54%	10 Missing and 5 partials ⚠️
internal/config/load.go	80.95%	2 Missing and 2 partials ⚠️
internal/lint/limits.go	86.66%	2 Missing and 2 partials ⚠️
internal/rules/catalog/rule.go	84.61%	3 Missing and 1 partial ⚠️
internal/config/merge.go	91.30%	1 Missing and 1 partial ⚠️
internal/rules/crossfilereferenceintegrity/rule.go	84.61%	2 Missing ⚠️
internal/rules/requiredstructure/rule.go	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
+ Coverage   86.31%   86.35%   +0.04%     
==========================================
  Files          92       94       +2     
  Lines        9950    10149     +199     
==========================================
+ Hits         8588     8764     +176     
- Misses        889      903      +14     
- Partials      473      482       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a configurable maximum input size (default 2 MiB) to harden mdsmith against OOM by enforcing byte caps across file-read paths, with configuration via .mdsmith.yml and --max-input-size.

Changes:

• Introduces limited read helpers (ReadFileLimited, ReadFSFileLimited) and threads MaxInputBytes through runner/fixer/rules.
• Adds human-readable size parsing (ParseSize) and wires CLI/config merging + docs updates.
• Expands test coverage with unit tests for helpers/parsing and e2e tests for limit behavior and CLI override.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
plan/81_oom-file-size-limit.md	Marks plan complete and checks off tasks/acceptance criteria.
PLAN.md	Updates plan index status for item 81 to ✅.
internal/rules/requiredstructure/rule.go	Applies size-limited reads to schemas and schema includes; threads max bytes into parsing flow.
internal/rules/requiredstructure/rule_test.go	Updates tests for new parseSchema(..., maxBytes) signature.
internal/rules/include/rule.go	Applies size-limited reads to includes and cycle scanning.
internal/rules/crossfilereferenceintegrity/rule.go	Applies size-limited reads when resolving targets for cross-file anchor checks.
internal/rules/catalog/rule.go	Applies size-limited reads for front matter extraction and include-cycle scanning.
internal/rules/catalog/rule_test.go	Updates tests for new readFrontMatter(..., maxBytes) signature.
internal/metrics/rank.go	Applies size-limited reads in metrics collection and threads max bytes into API.
internal/metrics/metrics_coverage_test.go	Updates tests for new Collect(..., maxBytes) signature.
internal/lint/limits.go	Adds default cap constant and limited-read helpers with size error reporting.
internal/lint/limits_test.go	Adds unit tests for limited-read helpers (normal/at/over/unlimited/etc).
internal/lint/file.go	Adds MaxInputBytes field to thread limit into rules.
internal/fix/fix.go	Uses limited reads for fix input and propagates max bytes into lint.File.
internal/engine/runner.go	Uses limited reads for check input and propagates max bytes into lint.File.
internal/config/size.go	Adds ParseSize for human-readable size strings.
internal/config/size_test.go	Adds unit tests for ParseSize valid/invalid cases.
internal/config/merge.go	Merges MaxInputSize and refactors default-copy logic via helper.
internal/config/load.go	Caps config file reads to 1 MiB to avoid config-induced OOM.
internal/config/config.go	Adds max-input-size config field.
docs/reference/cli.md	Documents --max-input-size flag behavior and defaults.
cmd/mdsmith/metrics.go	Uses limited metrics reads with default cap.
cmd/mdsmith/mergedriver.go	Uses limited reads in merge-driver paths.
cmd/mdsmith/main.go	Adds --max-input-size flag to check/fix, stdin limiting, and config/CLI resolution logic.
cmd/mdsmith/e2e_test.go	Adds e2e tests for oversized files, unlimited mode, and CLI override vs config.

Comments suppressed due to low confidence (2)

internal/rules/include/rule.go:198

• This diagnostic says the include file was "not found" for any read error, but ReadFSFileLimited can also fail with "file too large". This makes the error misleading. Consider changing the message to something like "cannot read include file" or branching on the error type so oversized files are reported accurately.

	data, err := lint.ReadFSFileLimited(readFS, readPath, f.MaxInputBytes)
	if err != nil {
		return "", []lint.Diagnostic{makeDiag(filePath, line,
			fmt.Sprintf("include file %q not found: %v", file, err))}
	}

internal/rules/requiredstructure/rule.go:507

• The wrapper error text always reports "not found" on read failure, but ReadFileLimited can also return "file too large". That makes oversized schema include failures look like missing files. Consider changing this to a neutral "cannot read" message or handling the size-limit error separately so the user gets the correct cause.

	fragData, err := lint.ReadFileLimited(includedPath, maxBytes)
	if err != nil {
		return nil, "", fmt.Errorf(
			"schema include file %q not found: %w", includedPath, err)
	}

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

[jeduden]

🟢 Merge Queue — picked up

This PR is in the queue and will be batched with other queue-labelled PRs.

Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run.

[jeduden]

🔵 Merge Queue — CI running

Merged into batch branch merge-queue/batch-130-1776615112. View CI run.

Next: No action needed — you'll be notified when CI completes.

[jeduden]

✅ Merge Queue — merged

This PR landed on main via commit ccd272a. CI run that validated the merge.

Next: Done — nothing more to do here.