Skip to content

Commit 57627b2

Browse files
youyongsongysyouclaude
authored
fix(translate): stop a sentence's punctuation from being read as part of a URL (#360)
`translation-url-residue` matched `\S+` after a scheme — everything up to the next space. A scheme with nothing after it therefore became a URL, so the English prose `(https://)` yielded `https://)`; and Chinese does not write spaces, so on the translation side the match ran past the address and took the rest of the sentence with it, `(https://)和路径` yielding `https://)和路径`. The two sides could not agree no matter how faithful the translation was. Measured: `connectors-operator` release-1.14 was red on every docs build for 22 hours — 9 runs — on one line reading `includes the correct protocol (https://) and path`. Nothing had been lost. The rule's own note said a check that misfires costs "one finding to read"; it does not. This rule reports against the whole page, so the assembled document could not attribute the finding to a segment and failed outright, no repair round, blast radius the whole page. - a URL now ends at the first character a URL cannot contain (printable ASCII), not at the next space, so both languages agree where it ends. The letters are spelled out rather than left to an `i` flag on purpose: under `iu` the engine case-folds first, and `s`/`k` fold to `ſ`/`K`, which `\P{ASCII}` would then swallow — caught by the one test with an `s` mid-URL; - trailing punctuation comes off the way GFM's autolink extension takes it off, and a closing bracket only when the URL did not open it; - a scheme with no host is not a URL. And the backstop: `translation-url-residue` is now checked per segment during acceptance too, as `translation-heading-sequence`, `translation-component-multiset` and `no-unparsed-emphasis` already are. A whole-document finding names a defect but not a place, so it had no segment to be sent back to and could only fail the document; checked per segment it arrives with the segment attached and is simply retranslated. Eight of the ten `translation-*` rules still have no per-segment mirror and all report against the document root, so any of them firing at assembly time fails the page the same way. Left as it is, and written down. Claude-Session: https://claude.ai/code/session_01XrrxNMokh3mx56riYUDnbQ Co-authored-by: ysyou <xiaodongzhang2026@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 40a0852 commit 57627b2

5 files changed

Lines changed: 339 additions & 10 deletions

File tree

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
"@alauda/doom": patch
3+
---
4+
5+
Stop a sentence's own punctuation from being read as part of a URL.
6+
7+
`translation-url-residue` matched `\S+` after a scheme — everything up to the
8+
next space. Two things follow from that, and both were wrong.
9+
10+
A scheme with nothing after it became a URL. `(https://)` yielded `https://)`,
11+
so a line of English prose naming a protocol was treated as an address the
12+
translation had to reproduce byte for byte. And Chinese does not write spaces,
13+
so on the translation side the match ran on past the address and took the rest
14+
of the sentence with it: `(https://)和路径` yielded `https://)和路径`, which
15+
could not equal anything the English side produced no matter how faithful the
16+
translation was.
17+
18+
Measured on a real build: `connectors-operator` release-1.14 was red on every
19+
docs build for 22 hours — 9 runs — on one line that reads `includes the correct
20+
protocol (https://) and path`. Nothing had been lost; the Chinese said the same
21+
thing with full-width brackets. The rule's own note said a check that misfires
22+
"costs one finding to read", and that turned out to be the wrong price: this
23+
rule reports against the whole page, so the assembled document could not
24+
attribute the finding to any segment and failed outright, with no repair round
25+
and the whole page as the blast radius.
26+
27+
Three changes, from cause to backstop:
28+
29+
- a URL now ends at the first character a URL cannot contain — printable ASCII
30+
is what a URL is made of — instead of at the next space, so the two languages
31+
agree about where it ends. An internationalised host written out in prose
32+
(`https://例え.jp`) is no longer detected, which is the right trade: a
33+
detector that cannot agree with its own translation is not a detector;
34+
- trailing punctuation comes off the end the way GFM's autolink extension takes
35+
it off, and a closing bracket comes off only when the URL did not open it —
36+
so `ftp://h/Foo_(disambiguation)` keeps its parenthesis and `(see ftp://h/x)`
37+
does not;
38+
- a scheme with no host is not a URL. An address that cannot be fetched cannot
39+
be a link a translation has to carry through.
40+
41+
And the backstop, which is the part that outlives this particular misfire:
42+
`translation-url-residue` is now also checked per segment, during acceptance,
43+
the way `translation-heading-sequence`, `translation-component-multiset` and
44+
`no-unparsed-emphasis` already are. A whole-document finding names a defect but
45+
not a place, and the pipeline rightly refuses to route one by line number — so
46+
until now a finding from this rule had no segment to be sent back to and could
47+
only fail the document. Checked per segment it arrives with the segment
48+
attached, gets three attempts and a repair agent, and the whole-page rule goes
49+
back to meaning what its note says it means: if it fires after every segment
50+
passed the same comparison, assembly did it.
51+
52+
Worth writing down for whoever meets this next: eight of the ten
53+
`translation-*` rules still have no per-segment mirror, and every one of them
54+
reports against the document root. Any of them that fires at assembly time
55+
fails the page the same way this one did.

‎packages/doom/src/cli/translate-segment-check.ts‎

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@ import {
88
import {
99
collectComponents,
1010
collectHeadingDepths,
11+
diffMultiset,
1112
} from '../remark-lint/translation-parity/shared.ts'
13+
import { collectProseUrls } from '../remark-lint/translation-parity/url-residue.ts'
1214

1315
import type { TranslationFinding } from './translate-checker.ts'
1416
import type { Judge } from './translate-judge.ts'
@@ -229,11 +231,13 @@ const collectTrailingAnchors = (tree: Root) => {
229231
/**
230232
* The skeleton is the source's, not the model's.
231233
*
232-
* Headings and components are structure rather than prose: translating a page
233-
* never changes a heading's level and never adds or removes a component. Both
234-
* are already checked over the whole document by `translation-heading-sequence`
235-
* and `translation-component-multiset` — this runs the *same* collectors, one
236-
* segment at a time.
234+
* Headings, components and URLs written in prose are structure rather than
235+
* prose: translating a page never changes a heading's level, never adds or
236+
* removes a component, and never drops a URL that markdown left as plain text.
237+
* All three are already checked over the whole document by
238+
* `translation-heading-sequence`, `translation-component-multiset` and
239+
* `translation-url-residue` — this runs the *same* collectors, one segment at
240+
* a time.
237241
*
238242
* That is not a duplicate of those rules, it is where they become actionable.
239243
* Every pairwise rule reports against the document as a whole, with no line
@@ -292,6 +296,29 @@ const structureFindings = (
292296
})
293297
}
294298

299+
// URLs markdown did not turn into links.
300+
//
301+
// The whole-document rule that owns this comparison,
302+
// `translation-url-residue`, is the one that proved the note above the hard
303+
// way: it fired on an assembled page, had no segment to be routed to, and
304+
// failed the document with no repair round — on a false positive, and on
305+
// every build for a day. Checked here it arrives with a segment attached, so
306+
// a genuine loss is retranslated instead of being fatal, and the whole-page
307+
// rule goes back to meaning what the note says it means: assembly did it.
308+
const expectedUrls = collectProseUrls(source)
309+
const actualUrls = collectProseUrls(tree)
310+
const { missing: missingUrls } = diffMultiset(expectedUrls, actualUrls)
311+
if (missingUrls.length > 0) {
312+
findings.push({
313+
rule: 'doom-translate:segment-url-residue',
314+
reason: `This segment's translation dropped ${missingUrls.length} URL(s) that the source writes out in prose: ${missingUrls
315+
.map((url) => `\`${url}\``)
316+
.join(
317+
', ',
318+
)}. They are plain text rather than links, so nothing carries them through for you — reproduce each one exactly as the source spells it.`,
319+
})
320+
}
321+
295322
// Only meaningful while the headings still correspond one to one; when they
296323
// do not, the sequence finding above already says so and pairing by index
297324
// would invent a second, misleading one.

‎packages/doom/src/remark-lint/translation-parity/url-residue.ts‎

Lines changed: 114 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,117 @@ import { currentPair, diffMultiset } from './shared.ts'
2222
* between prose and inline code. Those are identifiers, not links, and a guard
2323
* that cries wolf about them is worse than no guard at all.
2424
*
25-
* Honest caveat: measured exposure on this corpus is zero — all seven bare URLs
26-
* carry `https://`, which makes them link nodes, protected structurally. This
27-
* is a guard for a case nothing else covers, not a demonstrated detector.
25+
* Honest caveat, kept because it turned out to matter: when this shipped,
26+
* measured exposure on the corpus was zero — all seven bare URLs carried
27+
* `https://`, which makes them link nodes, protected structurally. It was a
28+
* guard for a case nothing else covered, not a demonstrated detector. Its
29+
* first meeting with real prose was a false positive that took a build down
30+
* for a day; what that cost, and what it changed, is written up on
31+
* {@link BARE_URL} and {@link HAS_HOST}.
2832
*/
29-
const BARE_URL = /\b(?:[a-z][a-z0-9+.-]*:\/\/|www\.)\S+/gi
33+
34+
/**
35+
* A URL runs to the first character a URL cannot contain.
36+
*
37+
* This used to be `\S+`, "everything up to the next space" — and Chinese does
38+
* not write spaces. On the translation side the match therefore swallowed the
39+
* rest of the sentence: `(https://)和路径` yielded `https://)和路径`, which
40+
* could not equal anything the English side produced no matter how faithful
41+
* the translation was. `[!-~]` is printable ASCII, which is what a URL is made
42+
* of, so a match now ends at the first full-width bracket or Han character.
43+
*
44+
* `[^\s\P{ASCII}]` reads as a double negative because that is the only way to
45+
* say it without an opaque code-point range: not whitespace, and not outside
46+
* ASCII.
47+
*
48+
* The letters are spelled out rather than left to an `i` flag, and that is not
49+
* style. Under `iu` the engine case-folds before testing the class, and two
50+
* ASCII letters fold to non-ASCII ones — `s` to `ſ` (U+017F), `k` to `K`
51+
* (U+212A) — so `\P{ASCII}` would swallow them and the negated class would
52+
* *reject* `s` and `k`. Caught by the one test with an `s` mid-URL:
53+
* `ftp://h/Foo_(disambiguation)` matched as far as `ftp://h/Foo_(di`.
54+
*
55+
* That misfire is not hypothetical and it was not cheap. `connectors-operator`
56+
* release-1.14 was red on every run for 22 hours over one line of prose —
57+
* `includes the correct protocol (https://) and path` — where nothing had been
58+
* lost at all: the translation said the same thing with full-width brackets.
59+
* And because this rule reports against the whole page, the assembled document
60+
* could not attribute the finding to any segment and failed outright without a
61+
* repair round. A misfire here does not cost "one finding to read".
62+
*
63+
* The price of the narrowing, stated plainly: an internationalised host
64+
* written out in prose (`https://例え.jp`) is no longer detected. That is the
65+
* right trade — the sources are English technical documentation, and a
66+
* detector that cannot agree with its own translation is not a detector.
67+
*
68+
* The sibling rule made the opposite choice, and deliberately:
69+
* `translation-link-isomorphism` *reports* the full-width stop an autolink
70+
* absorbed rather than normalising it away, because there the punctuation ends
71+
* up inside a real `href` and the link genuinely breaks. Prose has no `href`
72+
* to break — `(ftp://h/x)` reads exactly as intended — so here the two
73+
* spellings are made comparable instead of being reported.
74+
*/
75+
const BARE_URL = /\b(?:[a-zA-Z][a-zA-Z0-9+.-]*:\/\/|[wW]{3}\.)[^\s\P{ASCII}]+/gu
76+
77+
/**
78+
* Punctuation that ends the sentence rather than the URL.
79+
*
80+
* `(see ftp://h/x)` and `fetch ftp://h/x.tar.` both put a character in the
81+
* match that the writer did not mean as part of the address, and the
82+
* translation — which repunctuates, because that is what translating prose
83+
* does — will not have it. Trimmed the way GFM's autolink extension trims it,
84+
* which is also the way a reader would.
85+
*
86+
* Only ASCII forms are listed: {@link BARE_URL} cannot match anything else, so
87+
* a full-width bracket here would be a branch that can never be taken.
88+
*/
89+
const TRAILING_PUNCTUATION = /[!"'*,.:;?_~]+$/u
90+
91+
/**
92+
* Closing brackets, and the opener each needs in order to belong to the URL.
93+
*
94+
* So `ftp://h/Foo_(disambiguation)` keeps its parenthesis — the URL opened it
95+
* — and `(see ftp://h/x)` does not.
96+
*/
97+
const BRACKETS = new Map([
98+
[')', '('],
99+
[']', '['],
100+
['}', '{'],
101+
])
102+
103+
const occurrences = (value: string, character: string) =>
104+
value.split(character).length - 1
105+
106+
const trimSentence = (url: string) => {
107+
let value = url
108+
for (;;) {
109+
const before = value
110+
value = value.replace(TRAILING_PUNCTUATION, '')
111+
const closing = value.at(-1)
112+
const opening = closing == null ? undefined : BRACKETS.get(closing)
113+
if (
114+
closing != null &&
115+
opening != null &&
116+
occurrences(value, closing) > occurrences(value, opening)
117+
) {
118+
value = value.slice(0, -1)
119+
}
120+
if (value === before) {
121+
return value
122+
}
123+
}
124+
}
125+
126+
/**
127+
* A scheme with nothing after it is not a URL.
128+
*
129+
* `https://` on its own is prose — a sentence naming the protocol, as in "the
130+
* address must include the protocol (https://)". Requiring a host is what
131+
* tells the two apart, and it gives up nothing this rule was protecting: an
132+
* address with no host cannot be fetched, so it cannot be a link a translation
133+
* has to carry through.
134+
*/
135+
const HAS_HOST = /(?::\/\/|^[wW]{3}\.)[^\s\P{ASCII}]/u
30136

31137
/**
32138
* Text inside a link is not prose for this purpose.
@@ -54,7 +160,10 @@ export const collectProseUrls = (tree: Root) => {
54160
return
55161
}
56162
for (const [match] of node.value.matchAll(BARE_URL)) {
57-
found.push(match)
163+
const url = trimSentence(match)
164+
if (HAS_HOST.test(url)) {
165+
found.push(url)
166+
}
58167
}
59168
})
60169
return found
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
import { describe, expect, test } from '@rstest/core'
2+
3+
import { escapeMarkdownHeadingIds } from '#cli/helpers.ts'
4+
import { maskAst } from '#cli/translate-mask.ts'
5+
import { checkSegment } from '#cli/translate-segment-check.ts'
6+
import { planSegments } from '#cli/translate-segment.ts'
7+
import { mdxProcessor } from '#plugins/index.ts'
8+
9+
/**
10+
* The comparisons a segment is accepted on, and why they are made here as well
11+
* as over the whole page.
12+
*
13+
* A pairwise rule that fires at assembly time names a defect but not a place:
14+
* it reports against the document, and the pipeline refuses — correctly — to
15+
* route a document-wide finding by line number. With no segment to send back,
16+
* the document simply fails, no repair round, blast radius the whole page.
17+
*
18+
* Checked per segment the same defect arrives with the segment attached and is
19+
* merely retranslated. `translation-url-residue` is the rule that proved this
20+
* the expensive way: it had no mirror here, fired once on an assembled page,
21+
* and took `connectors-operator` release-1.14 down for 22 hours.
22+
*/
23+
24+
/** The one segment of a small document, with its content masked as the pipeline masks it. */
25+
const segmentContaining = (source: string, needle: string) => {
26+
const tree = mdxProcessor.parse(escapeMarkdownHeadingIds(source))
27+
maskAst(tree)
28+
const plan = planSegments({ tree, processor: mdxProcessor, floor: 0 })
29+
const segment = plan.segments.find((candidate) =>
30+
candidate.text.includes(needle),
31+
)
32+
expect(segment).toBeDefined()
33+
return segment!
34+
}
35+
36+
/** No judge: these are claims about the free layers, not about meaning. */
37+
const check = (source: string, needle: string, translation: string) =>
38+
checkSegment({
39+
segment: segmentContaining(source, needle),
40+
translation,
41+
processor: mdxProcessor,
42+
sourceLanguage: 'English',
43+
targetLanguage: 'Chinese',
44+
})
45+
46+
const SOURCE = `---
47+
title: Fetching
48+
---
49+
50+
# Fetching
51+
52+
Fetch the archive from ftp://h/x.tar before you start.
53+
`
54+
55+
describe('a URL written in prose, checked where it can be retranslated', () => {
56+
test('a segment that dropped one is faulted, with the segment attached', async () => {
57+
const findings = await check(
58+
SOURCE,
59+
'x.tar',
60+
['# 获取', '', '开始之前先获取归档。'].join('\n'),
61+
)
62+
63+
expect(findings.map((finding) => finding.rule)).toEqual([
64+
'doom-translate:segment-url-residue',
65+
])
66+
expect(findings[0].reason).toContain('ftp://h/x.tar')
67+
})
68+
69+
test('a translation that punctuates itself in Chinese is left alone', async () => {
70+
// The false positive: full-width brackets, no spaces, same URL. Nothing is
71+
// lost here and nothing should be reported.
72+
const findings = await check(
73+
SOURCE,
74+
'x.tar',
75+
// The escaped colon is how the segment itself spells it: the stringifier
76+
// escapes `ftp:` so `remark-directive` cannot read it as a directive, so
77+
// that is what the model is shown and what it writes back.
78+
['# 获取', '', '开始之前,请先从(ftp\\://h/x.tar)获取归档。'].join(
79+
'\n',
80+
),
81+
)
82+
83+
expect(findings).toEqual([])
84+
})
85+
})

‎packages/doom/test/remark-lint/translation-parity.spec.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,59 @@ describe('the two sides of a comparison are parsed the same way', () => {
243243
})
244244
})
245245

246+
/**
247+
* Where a URL stops, on both sides of a translation.
248+
*
249+
* Every case here comes from one incident: `connectors-operator` release-1.14
250+
* was red on every docs build for 22 hours because a line of English prose
251+
* naming a protocol — `includes the correct protocol (https://) and path` —
252+
* was read as a URL, and the Chinese translation, which brackets it the way
253+
* Chinese typography requires, was told it had lost one. Nothing had been
254+
* lost, and the finding could not be routed to a segment, so the document
255+
* failed with no repair round.
256+
*/
257+
describe('a URL, and the sentence it sits in', () => {
258+
test('a scheme with no host is prose, not a URL', () => {
259+
expect(
260+
collectProseUrls(
261+
syntaxProcessor.mdx.parse(
262+
'Verify that the address includes the correct protocol (https://) and path',
263+
),
264+
),
265+
).toEqual([])
266+
})
267+
268+
test('the two languages agree about where the URL ends', () => {
269+
const english = syntaxProcessor.mdx.parse(
270+
'See (ftp://h/x.tar) for the archive.',
271+
)
272+
// No spaces, which is why "everything up to the next space" could never
273+
// work here: it took the rest of the sentence with it.
274+
const chinese = syntaxProcessor.mdx.parse(
275+
'\u53c2\u89c1\uff08ftp://h/x.tar\uff09\u83b7\u53d6\u5f52\u6863\u3002',
276+
)
277+
278+
expect(collectProseUrls(english)).toEqual(['ftp://h/x.tar'])
279+
expect(collectProseUrls(chinese)).toEqual(collectProseUrls(english))
280+
})
281+
282+
test('a bracket the URL opened itself is part of it', () => {
283+
expect(
284+
collectProseUrls(
285+
syntaxProcessor.mdx.parse('see ftp://h/Foo_(disambiguation) for more'),
286+
),
287+
).toEqual(['ftp://h/Foo_(disambiguation)'])
288+
})
289+
290+
test('a full stop that ends the sentence is not part of the path', () => {
291+
expect(
292+
collectProseUrls(
293+
syntaxProcessor.mdx.parse('fetch it from ftp://h/x.tar.'),
294+
),
295+
).toEqual(['ftp://h/x.tar'])
296+
})
297+
})
298+
246299
describe('punctuation a bare URL swallowed is named as such', () => {
247300
test('a non-ASCII suffix is reported as absorbed, with the url intact', () => {
248301
const { absorbed, lost, unmatched } = partitionAbsorbed(

0 commit comments

Comments
 (0)