Commit b284571
Also strip unpaired surrogates in the OAI sanitizer
The previous commit replaced escapeXml10 with a regex to stop the double
escaping, but escapeXml10 was an AggregateTranslator that ALSO ran
UnicodeUnpairedSurrogateRemover, and the regex reproduced only the control-char
part. Consequence: a lone surrogate in metadata (truncated 4-byte character,
mojibake corpus) makes the StAX writer throw "Broken surrogate pair" inside
XOAI.index(), and because that is caught per item the record is silently dropped
from the OAI index. Measured before this fix: 2048 code units broke indexing,
against 0 for vanilla 7.6.7.
Verified over the whole BMP: 0 XML-1.0-illegal code units leak, 0 legal ones are
removed, valid surrogate pairs survive, unpaired ones do not, and XML metacharacters
still pass through unescaped.
Known and intentional difference from escapeXml10: C0/DEL controls U+007F-U+009F are
left as-is instead of being turned into &#NNN; entities. They are legal XML 1.0
characters (only XML 1.1 requires escaping them), so well-formedness is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent c6a1101 commit b284571
1 file changed
Lines changed: 23 additions & 9 deletions
Lines changed: 23 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
243 | 243 | | |
244 | 244 | | |
245 | 245 | | |
246 | | - | |
247 | | - | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
248 | 255 | | |
249 | | - | |
250 | | - | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
251 | 260 | | |
252 | 261 | | |
253 | 262 | | |
254 | 263 | | |
255 | 264 | | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
261 | 275 | | |
262 | 276 | | |
263 | 277 | | |
| |||
0 commit comments