fix(entity): guard candidate extraction against ReDoS on base64/minif… - #2082
fix(entity): guard candidate extraction against ReDoS on base64/minif…#2082bkr499 wants to merge 1 commit into
Conversation
…ied content
The i18n candidate_pattern's nested ambiguous quantifiers
`(?:[A-Z][a-z]+|[A-Z]{2,})+` partition long whitespace-free mixed-case
runs (base64 blobs, minified code) in exponentially many ways, causing
catastrophic backtracking that pins `mempalace mine` at 100% CPU
indefinitely.
Add `_strip_long_tokens()` to collapse `\S{30,}` runs to a single space
before the candidate regexes run, applied at both entry points:
`extract_candidates()` and `palace._candidate_entity_words()` (the
per-drawer mining hot path). The guard pattern is linear and cannot
itself backtrack. Natural-language extraction is unaffected — proper
nouns are far shorter than 30 chars.
Reproduced with `"Xa" + "B"*42 + "1"`; adds regression tests with
thread-watchdog timeouts so a regression fails fast instead of hanging.
Fixes MemPalace#2063.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port of upstream PR MemPalace#2082 (issue MemPalace#2063). The CamelCase candidate pattern's nested ambiguous quantifier ((?:[A-Z][a-z]+|[A-Z]{2,})+) can partition a long whitespace-free mixed-case run in exponentially many ways; base64 blobs and minified code pin re.findall at 100% CPU for hours and hang the mine. Confirmed on this fork's exact pattern: a 48-char adversarial token ("Xa" + 45 uppercase + digit) never returns. extract_candidates now collapses whitespace-free runs of 30+ chars to a space before any candidate regex runs. Proper nouns are far shorter (the simple-name pattern itself caps at 20 chars), so natural-language extraction is unaffected. The upstream palace.py twin fix has no counterpart here — this fork's per-drawer mining path does not run entity candidate extraction. Co-authored-by: bkr499 <bkr499@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Heads up on the guard's scope. The On current develop: import re
from mempalace.entity_detector import extract_candidates
text = "张三:方案没问题。张三,明天继续。张三。已经搞定了。团队同意了这个方向。" # 36 chars, no whitespace at all
extract_candidates(text, languages=("zh-CN",)) # {'张三': 3}
extract_candidates(re.sub(r"\S{30,}", " ", text), languages=("zh-CN",)) # {}Three other shapes behave the same (a longer variant, 「」 quotes, a numbered list): The zh tests will not catch this. Both positive ones still pass under the guard ( Scoping the class to ASCII fixes this without weakening the guard. The exploding pattern is Disclosure: I have #2065 open against the same issue taking that ASCII-scoped route, so read this as a data point rather than a neutral review. |
…ied content
The i18n candidate_pattern's nested ambiguous quantifiers
(?:[A-Z][a-z]+|[A-Z]{2,})+partition long whitespace-free mixed-case runs (base64 blobs, minified code) in exponentially many ways, causing catastrophic backtracking that pinsmempalace mineat 100% CPU indefinitely.Add
_strip_long_tokens()to collapse\S{30,}runs to a single space before the candidate regexes run, applied at both entry points:extract_candidates()andpalace._candidate_entity_words()(the per-drawer mining hot path). The guard pattern is linear and cannot itself backtrack. Natural-language extraction is unaffected — proper nouns are far shorter than 30 chars.Reproduced with
"Xa" + "B"*42 + "1"; adds regression tests with thread-watchdog timeouts so a regression fails fast instead of hanging.Fixes #2063.
What does this PR do?
How to test
Checklist
python -m pytest tests/ -v)ruff check .)