Parsing: Normalize Typographic (Smart) Quotes in Query Strings - #999
Draft
jbylund wants to merge 3 commits into
Draft
Parsing: Normalize Typographic (Smart) Quotes in Query Strings#999jbylund wants to merge 3 commits into
jbylund wants to merge 3 commits into
Conversation
Every word processor and phone keyboard turns a typed apostrophe into U+2019 and typed double quotes into U+201C/U+201D, so a pasted query carries them constantly. This parser read them as ordinary letters, which made a query for Gaea's Blessing (curly apostrophe) a search for a name containing that curly apostrophe: no rows, no error, no clue. fold_typographic_quotes folds exactly four characters -- U+2018/U+2019 to the ASCII apostrophe, U+201C/U+201D to the ASCII double quote -- and leaves everything else quotation-shaped alone. Measured against api.scryfall.com by putting each candidate around a phrase and asking whether it searched as one term: the guillemets, low-9 quotes, primes, fullwidth forms, CJK brackets, ornate quotes, backtick, acute and U+02BC all stay literal and match nothing, so widening the table would silently answer more than Scryfall does. It is a character substitution over the whole query, not a rule about quoted regions -- a curly apostrophe inside straight double quotes folds too, which is the only reason `name:"Gaea's Blessing"` (with a curly apostrophe) finds the card at all. Lives in spans.py, the leaf module already shared by both parsers and the typeahead balancer for exactly this kind of character-level rule, and applies before anything reads a character as a delimiter: at the top of parse_query, parse_search_query, and balance_partial_query. The balancer and the lexer have to agree about what a quote is, or a typed opening curly quote balances to nothing and then fails to lex as an unclosed `name:'` once parse_query folds it. Both parsers take the change, so test_parser_parity keeps them from diverging. Closes #970.
github-actions
Bot
added
parser
The spans.py comment and the test module docstring both asserted "every word processor and phone keyboard" curly-quotes typed apostrophes -- an overclaim nobody had actually checked, and a 24-line comment for a 4-line function regardless. Cut to the mechanism (which four characters fold, and that the table is measured against api.scryfall.com) and dropped the universal claim about input devices.
Both the spans.py comment and the test docstring wrote out the example as name:'Gaea's Blessing' with a straight ASCII apostrophe, then labeled it "(curly apostrophe)" -- true of the bug, not of the text on the line. Switched to the <U+2019> placeholder already used elsewhere in this file so the example doesn't contradict its own caption.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #970: curly quotes (U+2018/U+2019 single, U+201C/U+201D double) were read as ordinary letters, so
name:'Gaea's Blessing'typed with a curly apostrophe silently matched nothing — no rows, no error.fold_typographic_quotes()folds just those four characters to their ASCII equivalents, measured against what api.scryfall.com itself treats as a quote, and runs before parsing at the top ofparse_query,parse_search_query, andbalance_partial_queryso both parsers and the typeahead balancer agree on what counts as a quote.Original approach and measurement carried by #926 (commit
77e8eb640b, authored by @daveycodez), whose sibling colour-name half already landed as #989 — this PR pulls just the quote-folding piece, landing it inspans.pyrather thanhand_parser.pysince that shared module didn't exist yet when the original commit was written.Not in scope here: the JS-side live typeahead balancer (
app.js'sscanSpans) still only recognizes ASCII quotes; filed separately.Test plan
python -m pytest api/parsing/tests/test_typographic_quotes.py -q— 32 passedpython -m pytest api/parsing/tests/ -q— 2332 passedmake test-unit— 3270 passedpython -m ruff check/ruff format --checkon changed files — clean