[python3_13] The parser cannot distinguish whitespace in f-strings

[andrewstein]

In the examples of https://docs.python.org/3.13/reference/lexical_analysis.html#formatted-string-literals there is an example:

>>> foo = "bar"
>>> f"{ foo = }" # preserves whitespace
>>> " foo = 'bar'"

I have confirmed this behavior on the command line with cpython:

Python 3.10.15 | packaged by conda-forge | (main, Oct 16 2024, 01:24:20) [Clang 17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> foo = 'bar'
>>> print(f'{     foo     =      }')
     foo     =      'bar'

And the results are similar in python 3.13.9.

However, by the time the f-string f'{ foo = }' hits the parser, it cannot be distinguished from f'{foo=}', f'{foo = }', f'{ foo= }', etc.

Quoting from the spec:

When the equal sign '=' is provided, the output will have the expression text, the '=' and the evaluated value. Spaces after the opening brace '{', within the expression and after the '=' are all retained in the output.

[RobEin]

Is the problem related to the ANTLR4 Python 3.13.2 parser?
I tested the examples with the Java port, but I didn’t find any issues.
The lexer sends the whitespaces to the hidden channel in the correct places and in the correct amount.

foo = "bar"
print(f'{ foo= }')

[@0,0:0='utf-8',<ENCODING>,channel=HIDDEN,0:-1]
[@1,0:2='foo',<NAME>,1:0]
[@2,3:3=' ',<WS>,channel=HIDDEN,1:3]
[@3,4:4='=',<'='>,1:4]
[@4,5:5=' ',<WS>,channel=HIDDEN,1:5]
[@5,6:10='"bar"',<STRING>,1:6]
[@6,11:12='\r\n',<NEWLINE>,1:11]
[@7,13:17='print',<NAME>,2:0]
[@8,18:18='(',<'('>,2:5]
[@9,19:20='f'',<FSTRING_START>,2:6]
[@10,21:21='{',<LBRACE>,2:8]
[@11,22:22=' ',<WS>,channel=HIDDEN,2:9]
[@12,23:25='foo',<NAME>,2:10]
[@13,26:26='=',<'='>,2:13]
[@14,27:27=' ',<WS>,channel=HIDDEN,2:14]
[@15,28:28='}',<RBRACE>,2:15]
[@16,29:29=''',<FSTRING_END>,2:16]
[@17,30:30=')',<')'>,2:17]
[@18,31:32='\r\n',<NEWLINE>,2:18]
[@19,33:32='<EOF>',<EOF>,3:0]

After the opening curly brace (TokenIndex @11) there is a hidden space, and likewise before the closing curly brace (TokenIndex @14) as well.
I also tested it in the upcoming Python 3.14 ANTLR4 parser, and I didn’t find this issue there either.

[RobEin]

And if You want to make a distinction in the parser, You can achieve this with a semantic predicate or a parser superClass, monitoring the hidden WS tokens.

[andrewstein]

Yes, I am referring to the parser, specifically the parser in python/python3_13. Doing a simple "toString()" on the ParseTree produced by the parser returns

(fstring f' (fstring_middle (fstring_replacement_field { (annotated_rhs (star_expressions (star_expression (expression (disjunction (conjunction (inversion (comparison (bitwise_or (bitwise_xor (bitwise_and (shift_expr (sum (term (factor (power (await_primary (primary (atom (name (name_except_underscore foo))))))))))))))))))))) = })) ')

for all the examples above.