Newline escapes are removed

[jdufresne]

input:

from textwrap import dedent

def f():
    arg = "text"
    return dedent(
        """\
        some {}
        lorem ipsum
        """.format(arg)
    )

expected:

from textwrap import dedent

def f():
    arg = "text"
    return dedent(
        f"""\
        some {arg}
        lorem ipsum
        """
    )

actual:

from textwrap import dedent

def f():
    arg = "text"
    return dedent(
        f"""        some {arg}
        lorem ipsum
        """
    )

The newline escape was intentionally introduced to make the dedent block more readable.

[ikamensh]

Hi, thanks for reporting the bug. Similarly to #55 the problem lays in the symbols being gone from the parsed expression (escaping the newline has the specific purpose of removing this newline as seen by the interpreter).

I imagine a solution would be to find the diff on the parsed version of the code, and then apply this diff to the original section of the code. From fast research, this library could do this: https://pypi.org/project/diff-match-patch/. Yet this means very deep reworking of the library internals to adopt this new method of deriving new code. I can't provide any ETA on this.

[akx]

I think doing this robustly would require something like https://github.com/Instagram/LibCST instead of just working with the AST.

[ikamensh]

tracking the problem with dedicated test test/integration/test_issue83.py

[ikamensh]

Observation: openai codex can't solve it.

[ikamensh]

The issue occurs because newline escapes () inside multiline strings are dropped when parsing to AST and then unparsing. The earlier fix only handled cases where a line consisted solely of a backslash, so a string such as:

def f():
arg = "text"
return dedent(
"""

some {}
hohö
lorem ipsum
""".format(arg)
)
still loses the escapes after “some {}” and “hohö” when converted to an f‑string.

Attempted strategy
I attempted to implement a more general “preserve_escaped_newlines” helper that:

Uses diff-match-patch to diff the original snippet against the transformed one.

Detects deleted \n sequences that follow a backslash and re‑inserts them at the corresponding locations in the new code, preserving indentation of the following line.

During experimentation, I created a prototype that computed diffs and tried to reinsert removed \n sequences with their original indentation. However, the diff algorithm often grouped the backslash with other characters (e.g., the removed “}” from the .format() call), so reliably detecting the newline escapes proved difficult. The resulting output either duplicated text or still missed certain escaped lines.

Given the time constraints and the intricacies of accurately mapping positions after AST transformations, the new approach did not produce a stable solution. Further work would require a more robust mapping from original source locations to the transformed code—potentially rewriting the conversion logic to operate on tokens instead of relying solely on the AST.

No code changes were committed because the attempted implementation did not function correctly.