Skip to content

fix(entity): guard candidate extraction against ReDoS on base64/minif… - #2082

Open
bkr499 wants to merge 1 commit into
MemPalace:developfrom
bkr499:fix/2063-entity-regex-redos
Open

fix(entity): guard candidate extraction against ReDoS on base64/minif…#2082
bkr499 wants to merge 1 commit into
MemPalace:developfrom
bkr499:fix/2063-entity-regex-redos

Conversation

@bkr499

@bkr499 bkr499 commented Jul 27, 2026

Copy link
Copy Markdown

…ied content

The i18n candidate_pattern's nested ambiguous quantifiers (?:[A-Z][a-z]+|[A-Z]{2,})+ partition long whitespace-free mixed-case runs (base64 blobs, minified code) in exponentially many ways, causing catastrophic backtracking that pins mempalace mine at 100% CPU indefinitely.

Add _strip_long_tokens() to collapse \S{30,} runs to a single space before the candidate regexes run, applied at both entry points: extract_candidates() and palace._candidate_entity_words() (the per-drawer mining hot path). The guard pattern is linear and cannot itself backtrack. Natural-language extraction is unaffected — proper nouns are far shorter than 30 chars.

Reproduced with "Xa" + "B"*42 + "1"; adds regression tests with thread-watchdog timeouts so a regression fails fast instead of hanging.

Fixes #2063.

What does this PR do?

How to test

Checklist

  • Tests pass (python -m pytest tests/ -v)
  • No hardcoded paths
  • Linter passes (ruff check .)

…ied content

The i18n candidate_pattern's nested ambiguous quantifiers
`(?:[A-Z][a-z]+|[A-Z]{2,})+` partition long whitespace-free mixed-case
runs (base64 blobs, minified code) in exponentially many ways, causing
catastrophic backtracking that pins `mempalace mine` at 100% CPU
indefinitely.

Add `_strip_long_tokens()` to collapse `\S{30,}` runs to a single space
before the candidate regexes run, applied at both entry points:
`extract_candidates()` and `palace._candidate_entity_words()` (the
per-drawer mining hot path). The guard pattern is linear and cannot
itself backtrack. Natural-language extraction is unaffected — proper
nouns are far shorter than 30 chars.

Reproduced with `"Xa" + "B"*42 + "1"`; adds regression tests with
thread-watchdog timeouts so a regression fails fast instead of hanging.

Fixes MemPalace#2063.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gnusam pushed a commit to gnusam/mempalace-pgsql that referenced this pull request Jul 29, 2026
Port of upstream PR MemPalace#2082 (issue MemPalace#2063). The CamelCase candidate
pattern's nested ambiguous quantifier ((?:[A-Z][a-z]+|[A-Z]{2,})+) can
partition a long whitespace-free mixed-case run in exponentially many
ways; base64 blobs and minified code pin re.findall at 100% CPU for
hours and hang the mine. Confirmed on this fork's exact pattern: a
48-char adversarial token ("Xa" + 45 uppercase + digit) never returns.

extract_candidates now collapses whitespace-free runs of 30+ chars to a
space before any candidate regex runs. Proper nouns are far shorter
(the simple-name pattern itself caps at 20 chars), so natural-language
extraction is unaffected. The upstream palace.py twin fix has no
counterpart here — this fork's per-drawer mining path does not run
entity candidate extraction.

Co-authored-by: bkr499 <bkr499@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mvalentsev

Copy link
Copy Markdown
Contributor

Heads up on the guard's scope. The \S{30,} sketch comes from the issue itself, but \S matches 65389 of the 65408 codepoints above U+007E, so it also matches Chinese prose: CJK carries no ASCII whitespace, so a whole paragraph is a single run and gets collapsed before any candidate pattern runs.

On current develop:

import re
from mempalace.entity_detector import extract_candidates

text = "张三:方案没问题。张三,明天继续。张三。已经搞定了。团队同意了这个方向。"  # 36 chars, no whitespace at all
extract_candidates(text, languages=("zh-CN",))                           # {'张三': 3}
extract_candidates(re.sub(r"\S{30,}", " ", text), languages=("zh-CN",))  # {}

Three other shapes behave the same (a longer variant, 「」 quotes, a numbered list): {'张三': 3} before, {} after. Nothing recovers it downstream, since get_entity_patterns(("zh-CN",)) resolves 1 candidate pattern and 0 multi-word patterns, and the known-systems prepass reads the same collapsed text.

The zh tests will not catch this. Both positive ones still pass under the guard ({'朱宜振': 4} and {'张三': 4}) because their longest whitespace-free runs are 7 and 6 characters, under the threshold. The only input that reaches 30, test_zh_tw_known_limitation_inline_name_no_boundary, asserts the name is absent, so it passes too.

Scoping the class to ASCII fixes this without weakening the guard. The exploding pattern is \b([A-Z][a-z]+(?:[A-Z][a-z]+|[A-Z]{2,})+|[A-Z][a-z]{1,19})\b, whose classes are ASCII-only, so catastrophic backtracking needs an ASCII run by construction. On the 4920-char blob from the issue both \S{30,} and [!-~]{24,} give the same result, and [!-~] matches nothing above U+007E.

Disclosure: I have #2065 open against the same issue taking that ASCII-scoped route, so read this as a data point rather than a neutral review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Entity extraction regex backtracks catastrophically on base64/minified content — mines pinned for days

2 participants