Skip to content

Commit b284571

Browse files
milanmajchrakclaude
andcommitted
Also strip unpaired surrogates in the OAI sanitizer
The previous commit replaced escapeXml10 with a regex to stop the double escaping, but escapeXml10 was an AggregateTranslator that ALSO ran UnicodeUnpairedSurrogateRemover, and the regex reproduced only the control-char part. Consequence: a lone surrogate in metadata (truncated 4-byte character, mojibake corpus) makes the StAX writer throw "Broken surrogate pair" inside XOAI.index(), and because that is caught per item the record is silently dropped from the OAI index. Measured before this fix: 2048 code units broke indexing, against 0 for vanilla 7.6.7. Verified over the whole BMP: 0 XML-1.0-illegal code units leak, 0 legal ones are removed, valid surrogate pairs survive, unpaired ones do not, and XML metacharacters still pass through unescaped. Known and intentional difference from escapeXml10: C0/DEL controls U+007F-U+009F are left as-is instead of being turned into &#NNN; entities. They are legal XML 1.0 characters (only XML 1.1 requires escaping them), so well-formedness is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent c6a1101 commit b284571

1 file changed

Lines changed: 23 additions & 9 deletions

File tree

dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -243,21 +243,35 @@ private static Element createBundlesElement(Context context, Item item, AtomicBo
243243
}
244244

245245
/**
246-
* Matches the characters that XML 1.0 forbids outright: C0 controls other than tab, LF and CR,
247-
* plus the two non-characters U+FFFE and U+FFFF. See https://www.w3.org/TR/xml/#charsets
246+
* Matches everything XML 1.0 forbids outright, in three alternations:
247+
* <ol>
248+
* <li>C0 controls other than tab, LF and CR, plus the non-characters U+FFFE and U+FFFF;</li>
249+
* <li>a high surrogate not followed by a low surrogate;</li>
250+
* <li>a low surrogate not preceded by a high surrogate.</li>
251+
* </ol>
252+
* See https://www.w3.org/TR/xml/#charsets. Unpaired surrogates matter in practice: a truncated
253+
* 4-byte character in ingested metadata makes the StAX writer throw "Broken surrogate pair", and
254+
* because XOAI.index() catches that per item the record is silently dropped from the OAI index.
248255
*/
249-
private static final Pattern INVALID_XML10_CHARS =
250-
Pattern.compile("[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F\\uFFFE\\uFFFF]");
256+
private static final Pattern INVALID_XML10_CHARS = Pattern.compile(
257+
"[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F\\uFFFE\\uFFFF]"
258+
+ "|[\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])"
259+
+ "|(?<![\\uD800-\\uDBFF])[\\uDC00-\\uDFFF]");
251260

252261
/**
253262
* Sanitizes a string to remove characters that are invalid in XML 1.0.
254263
* <P>
255264
* NOTE: this deliberately REMOVES illegal characters rather than escaping the string. The value
256-
* returned here is handed to the XOAI serializer, which performs XML escaping itself, so escaping
257-
* here as well would double-escape every value containing &amp;, &lt;, &gt;, " or ' — a harvester
258-
* would then read the literal text "&amp;lt;" instead of a "&lt;" character. That silently corrupts
259-
* every OAI format built on the xoai document, including the cmdi and olac formats CLARIN/LINDAT
260-
* is aggregated through.
265+
* returned here is handed to the XOAI serializer, which performs XML escaping itself (every text
266+
* event goes through {@code XMLStreamWriter.writeCharacters}), so escaping here as well would
267+
* double-escape every value containing &amp;, &lt;, &gt;, " or ' — a harvester would then read the
268+
* literal text "&amp;lt;" instead of a "&lt;" character. That silently corrupts every OAI format
269+
* built on the xoai document, including the cmdi and olac formats CLARIN/LINDAT is aggregated
270+
* through.
271+
* <P>
272+
* The removal set must stay equivalent to what {@code StringEscapeUtils.escapeXml10} removed —
273+
* notably including unpaired surrogates — otherwise items carrying them fail to serialize and
274+
* drop out of the OAI index entirely.
261275
* @param value The string to sanitize.
262276
* @return A sanitized string, or null if the input was null.
263277
*/

0 commit comments

Comments
 (0)