Support ranges with negative endpoints in search parsing

[roed-math]

Search inputs like nilpotency=-1..3, nilpotency=-1-3 or cm=-4..-1 used to flash "not a valid input" because LIST_RE rejected signed range endpoints, and the splitters located the range dash with arg.index("-", 1), which picks the wrong dash in inputs like --4 (what prep_ranges produces from ..-4). This PR extends LIST_RE/LIST_FLOAT_RE to allow signed endpoints and introduces RANGE_DASH_RE to identify the separator dash (first dash preceded by a digit or decimal point, or a leading dash directly before a minus sign), using it consistently in parse_range, parse_range2, parse_range3, parse_range_float and parse_ints_to_list. A single leading - still denotes a negative number (-5 means −5), matching existing parse_range2 semantics. Along the way this fixes open-left ranges with negative endpoints (..-4), -4..-1 in parse_ints_to_list (used e.g. by ECNF CM-discriminant search), and scientific-notation floats (1e-5, 1e-5-2e-4), which previously errored due to an inverted exponent sign in find_prec. Adds lmfdb/utils/test_search_parsing.py with focused unit tests; galois_groups, maass_forms and number_fields suites pass, and the issue's URLs were verified through the test client. Addresses LMFDB#3825.

🤖 Generated with Claude Code

[roed-math]

Addressed the external review's P1 finding (CI test-file census): commit 21245d2 bumps the census 43→44 in python-package.yml and registers lmfdb/utils/test_search_parsing.py in both the proddb and devmirror shard entries of matrix_includes.json (same shard as test_utils.py), so the new tests actually run. Verified the exact census shell command passes at 44 and the new test file passes (9 tests).

Coordination note: PR #24 adds another test file with the same census fix — whichever merges second must bump 44→45 (deliberate same-line conflict).

Drive-by observation, out of scope: pre-existing test_workshoplinks.py is counted by the census but assigned to no shard, so it never runs in CI.

[roed314]

LGTM

[roed-math]

Second review pass requested two P1 changes; both are addressed, and the branch is now merged with main.

1. Exponential backtracking in LIST_RE / LIST_FLOAT_RE

The signed alternatives I added were ambiguous for every negative singleton: -1 matched both the singleton branch and the branch with an omitted lower endpoint. Rejecting a comma-separated list of n negative singletons therefore took time exponential in n ("-1,"*20 + "x" → 0.32 s, *22 → 1.3 s here; LIST_FLOAT_RE was worse). These patterns run on raw search input in the request thread, so that is a request-thread DoS.

Both are rewritten with a factored item grammar in which every comma-separated item has exactly one parse:

RANGE_SEP_STR      = r"(?:\.\.|-)"
UNSIGNED_INT_STR   = r"\d+"
SIGNED_INT_STR     = r"-?\d+"
INT_ITEM_STR       = r"(?:-?\d+(?:(?:\.\.|-)(?:-?\d+)?)?|--\d+|\.\.-?\d+)"
LIST_RE            = r"^INT_ITEM(?:,INT_ITEM)*$"
UNSIGNED_FLOAT_STR = r"(?:(?:\d+(?:[.]\d*)?|[.]\d+)(?:e[-+]?\d+)?|\d+/\d+)"
FLOAT_STR          = r"-?" + UNSIGNED_FLOAT_STR      # same language as before
FLOAT_ITEM_STR     = r"(?:FLOAT(?:(?:\.\.|-)(?:FLOAT)?)?|--UNSIGNED_FLOAT|\.\.FLOAT)"

Rejecting 5000 items now takes 0.45 ms (LIST_RE) / 0.99 ms (LIST_FLOAT_RE), and a fuzz over 22 repeated-unit shapes shows no blowup anywhere. A bare - is no longer accepted as an item, as asked. I also checked the file's other list patterns (LIST_POSINT_RE, LIST_RAT_RE, QQ_LIST_RE, SIGNED_LIST_RE, MULTISET_RE, the BRACKETED_*) for the same defect: all linear.

The new test_lists_reject_malformed_input_promptly runs each match in a subprocess with a 30 s timeout, so a return of exponential behaviour fails the test instead of hanging the run. Confirmed it times out on the old patterns and passes instantly on the new ones.

2. ..10 silently becoming -10

Rather than reject, this now works as written. prep_ranges folded every .. into - before the parser ran; it now folds .. only when a lower endpoint precedes it (RANGE_DOTS_RE = (?<=\d)\.\.), and LIST_RE/LIST_FLOAT_RE accept a leading .., which parse_range2/parse_range_float already knew how to handle:

parse_ints({"c": "..10"}, query, "c")     # {"c": {"$lte": 10}}
parse_ints({"c": "-5,..10"}, query, "c")  # {"$or": [{"c": -5}, {"c": {"$lte": 10}}]}
parse_floats({"c": "..4.5"}, query, "c")  # {"c": {"$lte": 4.5...}}

The tests for these go through the decorated public parsers, not just the helpers. ..-4 now stays ..-4 rather than being rewritten to --4 (both forms are still accepted), and the echo of every previously-working input is unchanged, since -4..-1 still folds to -4--1.

Parsers whose own regex does not accept .. — parse_posints, parse_signed_ints, parse_element_of/integer_options — now reject ..10 with the usual invalid-input flash instead of silently searching for -10 (checked they flash rather than 500). Extending open-left support to those is a natural follow-up; I left it out of this PR.

Merge with main, and the census

The branch picked up a merge of main while I was working. Merged it in (no conflicts), but note a silent one: main added lmfdb/tests/test_connection_reset.py and set the census to 44, which is the same number this branch had set for test_search_parsing.py, so git kept 44 without a conflict while the tree actually holds 45 test files. Bumped to 45 in a separate commit; both new test files are assigned to 2 shard entries each in matrix_includes.json. (lmfdb/tests/test_workshoplinks.py is still counted but unsharded — pre-existing, untouched.)

Verified

• lmfdb/utils/test_search_parsing.py → 12 passed (was 9), on both the pre-merge and post-merge trees.

• galois_groups, ecnf, maass_forms, number_fields suites → 79 passed on the pre-merge tree; the post-merge re-run was still going when I wrote this, so treat CI as the authority there.

• pyflakes and ruff check --preview --select=E722 clean on both changed files (pylint isn't installed here).

• Test client, inspecting the query actually sent to postgres rather than just the status code:

  input	query
  /GaloisGroup/?nilpotency=-1..3	{'nilpotency': {'$gte': -1, '$lte': 3}}
  /GaloisGroup/?nilpotency=..-1	{'nilpotency': {'$lte': -1}}
  /GaloisGroup/?nilpotency=..3	{'nilpotency': {'$lte': 3}} (was a search for −3)
  /GaloisGroup/?nilpotency=-1,..3	{'$or': [{'nilpotency': -1}, {'nilpotency': {'$lte': 3}}]}
  /EllipticCurve/?cm_disc=-4..-3	{'cm': {'$in': [-4, -3]}}
  /ModularForm/GL2/Q/Maass/?spectral_parameter=..9.6	{'spectral_parameter': {'$lte': 9.6...}}

  Controls n=2..8, conductor=11-30, discriminant=-3-5 unchanged; - and 3-- still flash errors.

• Differential check over ~11,400 random and exhaustive strings: against the previous head, LIST_RE gains only ..-forms and loses only items containing a bare -; LIST_FLOAT_RE gains only ..-forms and loses nothing.

[roed-math]

Confirmed, and fixed in 4ff6314fa. "1...2," * 22 + "x" took 1.81 s at c95b259e, doubling per item — the review is right that my "exactly one parse per item" claim was false, and my earlier timing check used hand-picked units that never included this shape.

The fix

Took the first suggested option. prep_ranges already normalizes an internal .. to a dash before either pattern runs, so .. never needs to separate two endpoints; only the leading .. of an omitted lower endpoint reaches them. Dropped the alternative from both item grammars (and the now-pointless RANGE_SEP_STR), keeping the dedicated leading-.. branch:

INT_ITEM_STR   = r"(?:-?\d+(?:-(?:-?\d+)?)?|--\d+|\.\.-?\d+)"
FLOAT_ITEM_STR = r"(?:FLOAT(?:-(?:FLOAT)?)?|--UNSIGNED_FLOAT|\.\.FLOAT)"

I applied it to LIST_RE too. The integer grammar was never ambiguous — it has no decimal points — but there is no reason for it to accept a separator that preprocessing has already removed, and keeping the two grammars parallel makes the invariant easier to state.

The raw patterns therefore now reject 1..5, 2..10, -4..-1 and 5.., which they only ever saw post-prep_ranges anyway. New test_dotted_ranges_still_work_through_the_parsers pins that both spellings still reach the intended query through the public parsers, as asked:

assert not LIST_RE.match("1..5")
parse_ints({"c": "1..5"}, query, "c")    # {"c": {"$gte": 1, "$lte": 5}}
parse_ints({"c": "..5"}, query, "c")     # {"c": {"$lte": 5}}
parse_floats({"c": "1..5"}, ...)         # $gte ≈ 1, $lte ≈ 5
parse_floats({"c": "..5"}, ...)          # $lte only

Better method, since hand-picked units were the actual failure

I replaced the ad-hoc timing spot-check with an exhaustive sweep: enumerate every string up to length 6 over 01.-/e (plus a length-4 pass over 01.-,/e29), keep the ones the pattern accepts as a single item, and time rejecting (item + ",") * 26 + "x". An item with two parses explodes; a unique-parse item stays flat.

• On c95b259e it flags exactly 0...0, 0...1, 1...0, 1...1, at ~29 s each — so the detector does catch this class.
• On 4ff6314fa: 3866 accepted float items and 706 int items tested, 0 flagged, worst case 36 µs.
• Run over the file's other list patterns (LIST_POSINT_RE, LIST_RAT_RE, QQ_LIST_RE, SIGNED_LIST_RE, MULTISET_RE, BRACKETED_*): all clean. That claim was hand-picked units last time; now it is exhaustive.
• Plus a structured sweep for anything too long for brute force — every {"", "-", "--", "..", "..-"} × {"1", ".2", "1.", "1/2", "1e-5", "1.2e-3", …} × {"", "-", "..", "-1", "--1", "-.2", "-1..2", …} combination, items up to 14 characters: 301 accepted, 0 flagged.

test_lists_reject_malformed_input_promptly now covers 9 (pattern, input) pairs — -1, --1, -1--1, 1...2 and 1.-.2 against both patterns — rather than only the negative-singleton class.

Verified

• lmfdb/utils/test_search_parsing.py → 13 passed.
• galois_groups, ecnf, maass_forms, number_fields → 68 passed.
• pyflakes + ruff check --preview --select=E722 clean.
• Test client re-run: every query dict is byte-identical to the previous commit — nilpotency=-1..3 / ..3 / -1,..3, n=2..8, cm_disc=-4..-3, spectral_parameter=-1..9.6 and ..9.6, the controls, and - / 3-- still flashing errors. Narrowing the patterns changed nothing a user can reach.

CI for c95b259e had got as far as green setup jobs with the shards still running when this went up; 4ff6314fa supersedes it.

[roed314]

GPT signed off.

[roed-math]

Superseded by LMFDB#7141, opened upstream from this same branch. Closing here; review continues upstream.