Skip to content

Commit 7a0d191

Browse files
pablofdezrclaudeatomiks
authored
fix(core): stop chars ranges leaking across patterns and shared tokens (#276)
* fix(core): stop chars ranges leaking across patterns and shared tokens Two defects made ranged char highlights (`/foo/2`, `/foo/#a`) affect the wrong characters, per #169. 1. Range/pattern misalignment (index.ts). `charsList` always received an entry, but `charsListNumbers` (the parallel ranges array) was only pushed when a numeric range was present. An id-only annotation like `/foo/#a` therefore left the arrays misaligned, so a later pattern inherited the previous pattern's range. Always push a ranges entry (empty array when there is no range) to keep the arrays index-aligned. 2. Ignored occurrences over-consumed their token (chars/*). When a range excluded an occurrence, `splitElement` returned the whole node unsplit, so the entire containing token was marked visited. Any other pattern living in the same token on that line (e.g. `Length` inside `getStringLength` when `/get/1` is range-ignored on later lines) could then never match. Let ignored occurrences split like any other so only the matched part is marked visited; the now-unused `ignoreChars` plumbing is removed from `getElementsToHighlight`/`splitElement`. Result: `/get/1 /Length/` now highlights `Length` on every line while `get` stays on its first occurrence, and `/get/#a /Length/2` highlights `get` everywhere while `Length` is limited to its second occurrence. Snapshots: `highlightedMultipleCharsRange` gains the two occurrences that were previously dropped; the other three updated snapshots are unchanged in what they highlight (identical <mark> sets) and only gain cosmetic span splits on ignored, unhighlighted tokens. Added a `charsRangeInheritance` fixture covering both scenarios. This addresses the range-inheritance part of #169. Highlighting a pattern that is a substring of another highlighted pattern (`/getStringLength/ /get/`) is a separate nested-highlight feature and is out of scope here. Refs #169 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: add patch changeset * fix(core): keep split fragments from spelling synthetic matches Splitting an occurrence a range excludes leaves the token in pieces around a node the scanner has to skip. Two things then went wrong. The remaining text was recomputed by dropping the excluded node, which joined its neighbours: 'bal' + 'on;' reads as 'balon;' and contains an occurrence of 'lo' the code never had. That phantom either advanced the range counter past a real later occurrence, or led the matcher across the gap and returned non-adjacent elements, which wrapHighlightedChars then spliced as if they were consecutive -- deleting and duplicating text. Keep a boundary in place of an excluded node so fragments cannot spell a match across it, count an occurrence only once it resolves to elements, abandon a partial run as soon as it stops being contiguous, and read the positions back from the tree before splicing. * test(core): pin the unranged variant of the fragment-joining bug `/con/` on `coconnection` renders as `coconneconon;` on master: the highlighted `con` is dropped when the remaining text is recomputed, `co` and `nection;` join into a phantom second occurrence, and wrapping it splices siblings that were never matched. * fix(core): preserve chars highlights across token trees * fix(core): preserve markup around ranged highlights * fix(core): batch character highlight mutations * fix(core): preserve whole transformer boundaries --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: atomiks <cc.glows@gmail.com>
1 parent 012acb6 commit 7a0d191

11 files changed

Lines changed: 1399 additions & 434 deletions

‎.changeset/lucky-clocks-thank.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"rehype-pretty-code": patch
3+
---
4+
5+
fix: stop character-highlight ranges leaking across patterns, so an id-only annotation like `/foo/#a` no longer makes a later pattern inherit the previous pattern's range, and a range-ignored occurrence no longer consumes the whole token it lives in (#169)
6+
7+
Range-excluded occurrences are counted without changing the token tree, so unhighlighted markup remains intact.
8+
9+
Partial highlights contained within a transformer-generated element preserve that surrounding semantic element instead of cloning it.

‎packages/core/src/chars/charsHighlighter.ts‎

Lines changed: 199 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,190 @@
11
import type { Element } from 'hast';
2-
import type { CharsHighlighterOptions, CharsElement } from '../types';
3-
import { getElementsToHighlight } from './getElementsToHighlight';
4-
import { wrapHighlightedChars } from './wrapHighlightedChars';
52
import { toString as hastToString } from 'hast-util-to-string';
6-
import { isElement } from '../utils';
3+
import type { CharsHighlighterOptions, CharsElement } from '../types';
4+
import {
5+
getContentLength,
6+
getElementsToHighlight,
7+
} from './getElementsToHighlight';
8+
import {
9+
wrapHighlightedChars,
10+
wrapHighlightedCharsInRange,
11+
} from './wrapHighlightedChars';
12+
13+
interface SelectedMatch {
14+
chars: string;
15+
start: number;
16+
end: number;
17+
element?: CharsElement;
18+
}
19+
20+
interface MatchComponent {
21+
firstIndex: number;
22+
lastIndex: number;
23+
start: number;
24+
matches: Array<SelectedMatch>;
25+
}
26+
27+
function isAvailable(occupied: Uint8Array, start: number, end: number) {
28+
for (let index = start; index < end; index++) {
29+
if (occupied[index] === 1) {
30+
return false;
31+
}
32+
}
33+
34+
return true;
35+
}
36+
37+
function findMatches(
38+
source: string,
39+
chars: string,
40+
charsIndex: number,
41+
occupied: Uint8Array,
42+
options: CharsHighlighterOptions,
43+
): Array<SelectedMatch> {
44+
const selectedMatches: Array<SelectedMatch> = [];
45+
const currentRange = options.ranges[charsIndex] || [];
46+
const counterId = `${chars}-${charsIndex}`;
47+
let searchStart = 0;
48+
49+
while (searchStart <= source.length - chars.length) {
50+
const matchStart = source.indexOf(chars, searchStart);
51+
if (matchStart === -1) {
52+
break;
53+
}
54+
55+
const matchEnd = matchStart + chars.length;
56+
if (!isAvailable(occupied, matchStart, matchEnd)) {
57+
searchStart = matchStart + 1;
58+
continue;
59+
}
60+
61+
const occurrence = (options.counterMap.get(counterId) || 0) + 1;
62+
options.counterMap.set(counterId, occurrence);
63+
occupied.fill(1, matchStart, matchEnd);
64+
65+
if (currentRange.length === 0 || currentRange.includes(occurrence)) {
66+
selectedMatches.push({ chars, start: matchStart, end: matchEnd });
67+
}
68+
69+
searchStart = matchEnd;
70+
}
71+
72+
return selectedMatches;
73+
}
74+
75+
function createMatchComponents(
76+
element: Element,
77+
selectedMatches: Array<SelectedMatch>,
78+
) {
79+
const ends: Array<number> = [];
80+
let offset = 0;
81+
82+
for (const child of element.children) {
83+
offset += getContentLength(child);
84+
ends.push(offset);
85+
}
86+
87+
const components: Array<MatchComponent> = [];
88+
const matchesByPosition = [...selectedMatches].sort(
89+
(first, second) => first.start - second.start,
90+
);
91+
let firstIndex = 0;
92+
let lastIndex = 0;
93+
94+
for (const match of matchesByPosition) {
95+
while (
96+
firstIndex < element.children.length &&
97+
ends[firstIndex] <= match.start
98+
) {
99+
firstIndex++;
100+
}
101+
102+
lastIndex = Math.max(lastIndex, firstIndex);
103+
while (lastIndex < element.children.length && ends[lastIndex] < match.end) {
104+
lastIndex++;
105+
}
106+
107+
const previous = components.at(-1);
108+
if (previous && firstIndex <= previous.lastIndex) {
109+
previous.lastIndex = Math.max(previous.lastIndex, lastIndex);
110+
previous.matches.push(match);
111+
} else {
112+
components.push({
113+
firstIndex,
114+
lastIndex,
115+
start: firstIndex === 0 ? 0 : ends[firstIndex - 1],
116+
matches: [match],
117+
});
118+
}
119+
}
120+
121+
return components;
122+
}
123+
124+
function applyMatches(
125+
element: Element,
126+
selectedMatches: Array<SelectedMatch>,
127+
options: CharsHighlighterOptions,
128+
) {
129+
const components = createMatchComponents(element, selectedMatches);
130+
131+
for (
132+
let componentIndex = components.length - 1;
133+
componentIndex >= 0;
134+
componentIndex--
135+
) {
136+
const component = components[componentIndex];
137+
const container: Element = {
138+
type: 'element',
139+
tagName: 'span',
140+
properties: {},
141+
children: element.children.slice(
142+
component.firstIndex,
143+
component.lastIndex + 1,
144+
),
145+
};
146+
147+
for (let index = component.matches.length - 1; index >= 0; index--) {
148+
const match = component.matches[index];
149+
const target = getElementsToHighlight(
150+
container,
151+
match.start - component.start,
152+
match.end - component.start,
153+
);
154+
if (!target) {
155+
continue;
156+
}
157+
158+
if (target.type === 'elements') {
159+
match.element = wrapHighlightedChars(
160+
target.parent,
161+
target.elements,
162+
match.chars,
163+
options,
164+
);
165+
} else {
166+
match.element = wrapHighlightedCharsInRange(
167+
target.parent,
168+
target.childIndex,
169+
target.start,
170+
target.end,
171+
match.chars,
172+
options,
173+
);
174+
}
175+
}
176+
177+
element.children.splice(
178+
component.firstIndex,
179+
component.lastIndex - component.firstIndex + 1,
180+
...container.children,
181+
);
182+
}
183+
}
7184

8185
/**
9-
* Loops through the child nodes and finds the nodes that make up the chars.
10-
* If the chars cross node boundaries, those nodes are wrapped with
11-
* <span data-highlighted-chars-mark>, and that node is passed to
12-
* onVisitHighlightedChars.
13-
*
14-
* If a node partially matches the chars, its content is replaced with the
15-
* matched part, and the left and/or right parts are cloned to sibling nodes.
186+
* Finds each requested string in the line, then applies the selected matches
187+
* without materializing range-excluded occurrences.
16188
*/
17189
export function charsHighlighter(
18190
element: Element,
@@ -23,81 +195,25 @@ export function charsHighlighter(
23195
id: string | undefined,
24196
) => void,
25197
) {
26-
const { ranges = [] } = options;
27-
const textContent = hastToString(element);
28-
29-
// biome-ignore lint/complexity/noExcessiveCognitiveComplexity: char matching is inherently branchy
30-
charsList.forEach((chars, index) => {
31-
if (chars && textContent?.includes(chars)) {
32-
let textContent = hastToString(element);
33-
let startIndex = 0;
34-
35-
while (textContent.includes(chars)) {
36-
// Snapshot the remaining text so we can bail out if an iteration
37-
// fails to make progress. The recomputed `textContent` excludes
38-
// already-highlighted nodes, so a productive iteration always makes
39-
// it strictly shorter; if it does not shrink, no occurrence was
40-
// consumed and continuing would loop forever.
41-
const previousTextContent = textContent;
42-
const currentCharsRange = ranges[index] || [];
43-
const id = `${chars}-${index}`;
44-
45-
options.counterMap.set(id, (options.counterMap.get(id) || 0) + 1);
46-
47-
const ignoreChars =
48-
currentCharsRange.length > 0 &&
49-
!currentCharsRange.includes(options.counterMap.get(id) ?? -1);
50-
51-
const elementsToWrap = getElementsToHighlight(
52-
element,
53-
chars,
54-
startIndex,
55-
ignoreChars,
56-
);
57-
58-
// maybe throw / notify due to failure here
59-
if (elementsToWrap.length === 0) break;
198+
const source = hastToString(element);
199+
const occupied = new Uint8Array(source.length);
200+
const selectedMatches: Array<SelectedMatch> = [];
60201

61-
wrapHighlightedChars(
62-
element,
63-
elementsToWrap,
64-
options,
65-
ignoreChars,
66-
onVisitHighlightedChars,
67-
);
68-
69-
// re-start from the 'last' node (the chars or part of them may exist
70-
// multiple times in the same node)
71-
// account for possible extra nodes added from split with - 2
72-
startIndex = Math.max(
73-
elementsToWrap[elementsToWrap.length - 1].index - 2,
74-
0,
75-
);
76-
77-
textContent = element.children
78-
.map((childNode) => {
79-
const props = isElement(childNode) ? childNode.properties : {};
80-
if (
81-
props &&
82-
!Object.hasOwn(props, 'rehype-pretty-code-visited') &&
83-
!Object.hasOwn(props, 'data-highlighted-chars-mark')
84-
) {
85-
return hastToString(childNode);
86-
}
87-
})
88-
.join('');
89-
90-
// Safety guard: if the remaining text did not shrink, this iteration
91-
// consumed nothing, so stop instead of spinning forever.
92-
if (textContent.length >= previousTextContent.length) break;
93-
}
202+
charsList.forEach((chars, charsIndex) => {
203+
if (!chars) {
204+
return;
94205
}
206+
207+
selectedMatches.push(
208+
...findMatches(source, chars, charsIndex, occupied, options),
209+
);
95210
});
96211

97-
element.children.forEach((childNode) => {
98-
if (!isElement(childNode)) return;
99-
if (Object.hasOwn(childNode.properties, 'rehype-pretty-code-visited')) {
100-
childNode.properties['rehype-pretty-code-visited'] = undefined;
212+
applyMatches(element, selectedMatches, options);
213+
214+
for (const match of selectedMatches) {
215+
if (match.element) {
216+
onVisitHighlightedChars?.(match.element, options.idsMap.get(match.chars));
101217
}
102-
});
218+
}
103219
}

0 commit comments

Comments
 (0)