MDS037: Skip paragraphs inside generated sections - #166
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #166 +/- ##
==========================================
+ Coverage 88.03% 88.04% +0.01%
==========================================
Files 110 110
Lines 14106 14148 +42
==========================================
+ Hits 12418 12457 +39
- Misses 1228 1230 +2
- Partials 460 461 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Implements skipping of paragraph fingerprinting/checking inside <?include?> and <?catalog?> generated-section bodies for MDS037 (duplicated-content), to prevent false positives caused by included or catalog-rendered content being duplicated across files.
Changes:
- Added generated-section range detection (
generatedRanges) and offset lookup helper (inGeneratedRange) in MDS037. - Updated paragraph extraction to skip paragraphs whose first line starts inside a generated-section body (applies to both checked file and corpus indexing).
- Added tests and updated MDS037 documentation; marked plan item 91 complete.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| plan/91_mds037-skip-generated-sections.md | Marks plan as complete and checks off tasks/acceptance criteria. |
| internal/rules/duplicatedcontent/rule.go | Adds generated-section range detection + skips paragraphs inside generated bodies during extraction. |
| internal/rules/duplicatedcontent/rule_test.go | Adds unit tests covering include/catalog skipping, corpus indexing skip, and basic range detection. |
| internal/rules/MDS037-duplicated-content/README.md | Documents the new “Generated sections” behavior and rationale. |
| PLAN.md | Updates plan status catalog entry for item 91 to ✅. |
Walk top-level AST nodes once per file to collect [start, stop) byte ranges for <?include?> and <?catalog?> directive bodies. extractParagraphs checks each paragraph's first-line byte offset against those ranges and skips any that fall inside one. The same extraction runs for corpus files during index building, so a host file's generated copy is never indexed. Closes plan 91. https://claude.ai/code/session_018GsQpm14NmJL1XfxvLnGY1
Track nesting depth per directive so that an inner <?include?> inside an outer <?include?> body does not prematurely close the outer byte range. The outer range now correctly spans all content up to the outer closing marker, keeping paragraphs after the inner pair inside the generated section. Adds TestGeneratedRanges_NestedSameNamePair and TestCheck_SkipsNestedIncludeGeneratedSection as regression tests. https://claude.ai/code/session_018GsQpm14NmJL1XfxvLnGY1
cb825b2 to
8161d44
Compare
jeduden
added
queue
|
🟢 Merge Queue — picked up This PR is in the queue and will be batched with other Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run. |
|
🔵 Merge Queue — CI running Merged into batch branch Next: No action needed — you'll be notified when CI completes. |
jeduden
removed
the
queue:active
|
✅ Merge Queue — merged This PR landed on Next: Done — nothing more to do here. |
Summary
Implement automatic skipping of paragraphs inside
<?include?>and<?catalog?>generated sections for the MDS037 duplicated content rule. This prevents false positives when content is included or cataloged from other files.Key Changes
generatedRanges()function: Walks the AST to identify byte ranges covering the bodies of<?include?>and<?catalog?>directives by tracking opening and closing processing instruction markers.inGeneratedRange()helper: Checks whether a given byte offset falls within any generated section range.extractParagraphs(): Now skips paragraphs whose first line starts within a generated section range. This applies both when checking a file and when indexing corpus files.TestCheck_SkipsIncludeGeneratedSection: Verifies paragraphs in<?include?>bodies are not flaggedTestCheck_SkipsCatalogGeneratedSection: Verifies paragraphs in<?catalog?>bodies are not flaggedTestCheck_DuplicateOutsideGeneratedSectionStillFires: Ensures real duplicates outside generated sections are still detectedTestCheck_CorpusSkipsIncludeGeneratedSection: Ensures corpus files don't index their generated contentTestGeneratedRanges_*: Unit tests for the range detection logicImplementation Details
https://claude.ai/code/session_018GsQpm14NmJL1XfxvLnGY1