Can't handle rst

[boosh]

Issue

The equals signs in .rst files mess up aider's search/replace tokens, so it's unable to apply edits, e.g.:

<<<<<<< SEARCH
=======
=======
New heading
=======
This is some new text
>>>>>>> REPLACE

Version and model info

Aider: v0.57.1

[akaihola]

Also, this seems to happen even if the number of equal signs in the heading underline doesn't match the Aider search/replace separator (see this comment to #1817 for details).

[akaihola]

@boosh, I made it simpler to change the divider strings in #1817. I guess the simplest fix for the problem with reStructuredText would be to extend the middle divider from just ======= to e.g. ======= <<SEARCH >>REPLACE, but I don't know how much it would increase the frequency of badly formatted code edit responses from LLMs.

I'm also wondering whether Anthropic's advice to Use XML tags to structure your prompts would also apply to requested response formats. Would Claude also give higher quality responses using pseudo XML tags, similarly to supposed higher quality "understanding" of prompts structured with those?

[akaihola]

By the way, I believe the change in 7fa1620 ("feat: Allow flexible matching of 5-9 characters in SEARCH/REPLACE block prefixes" on Sep 20) made reStructuredText handling worse:

-HEAD = "<<<<<<< SEARCH"
-DIVIDER = "======="
-UPDATED = ">>>>>>> REPLACE"
+HEAD = r"<{5,9} SEARCH"
+DIVIDER = r"={5,9}"
+UPDATED = r">{5,9} REPLACE"

+    head_pattern = re.compile(HEAD)
+    divider_pattern = re.compile(DIVIDER)
+    updated_pattern = re.compile(UPDATED)

It seems earlier the divider line had to match exactly, so there had to be no less and no more than seven = signs. But now with the regex, any line beginning with 5 = signs is treated as a middle divider.

[akaihola]

The smallest change to minimize the reStructuredText problem would be probably this:

diff --git a/aider/coders/editblock_coder.py b/aider/coders/editblock_coder.py
index 118759e9..9305d405 100644
--- a/aider/coders/editblock_coder.py
+++ b/aider/coders/editblock_coder.py
@@ -412,7 +412,7 @@ def find_original_update_blocks(content, fence=DEFAULT_FENCE, valid_fnames=None)
     current_filename = None
 
     head_pattern = re.compile(HEAD)
-    divider_pattern = re.compile(DIVIDER)
+    divider_pattern = re.compile(fr"{DIVIDER}[ ]*$")
     updated_pattern = re.compile(UPDATED)
 
     while i < len(lines):

This fixes the issues for reStructuredText heading underlines with more than 9 = symbols. But if you have headings of 5 to 9 characters, it will still fail, like for example for this heading:

What?
=====

The code change above is quite similar to what the splitting pattern already does:

separators = "|".join([HEAD, DIVIDER, UPDATED])

split_re = re.compile(r"^((?:" + separators + r")[ ]*\n)", re.MULTILINE | re.DOTALL)

[akaihola]

Building on top of #1817, I experimented with pseudo XML dividers (see Use XML tags to structure your prompts) in the xml-dividers branch, and at least for my first manual test it worked beautifully:

Divider configuration in editblock_coder.py:

    edit_block_config = {
        "search_divider": "<AIDER-SEARCH>",
        "middle_divider": "</AIDER-SEARCH> <AIDER-REPLACE>",
        "replace_divider": "</AIDER-REPLACE>"
    }

Aider session:

> /add README.rst                                                                                                                                                                                                                 

Added /home/akaihola/prg/graylint/README.rst to the chat
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
README.rst                                                                                                                                                                                                                        
> Under the "What?" heading, add a funny intro.                                                                                                                                                                                   

                    
Certainly! I'll add a funny intro under the "What?" heading in the README.rst file. Here's the change:                                                                                                                            

README.rst                                                                                                                                                                                                                        

                                                                                                                                                                                                                                  
 <AIDER-SEARCH>                                                                                                                                                                                                                   
 What?                                                                                                                                                                                                                            
 =====                                                                                                                                                                                                                            
                                                                                                                                                                                                                                  
 This utility runs linters on Python source code files.                                                                                                                                                                           
 </AIDER-SEARCH> <AIDER-REPLACE>                                                                                                                                                                                                  
 What?                                                                                                                                                                                                                            
 =====                                                                                                                                                                                                                            
                                                                                                                                                                                                                                  
 Imagine a world where your code is so clean, even your rubber duck approves. Welcome to Graylint, the lint detector that's more thorough than your grandmother inspecting your room!                                             
                                                                                                                                                                                                                                  
 This utility runs linters on Python source code files.                                                                                                                                                                           
 </AIDER-REPLACE>                                                                                                                                                                                                                 
                                                                                                                                                                                                                                  

This change adds a humorous introduction to the "What?" section, setting a lighthearted tone for the utility's description.                                                                                                       


Tokens: 23k sent, 153 received. Cost: $0.07 message, $0.07 session.
Applied edit to README.rst

Resulting Graylint README.rst excerpt:

What?
=====

Imagine a world where your code is so clean, even your rubber duck approves. Welcome to Graylint, the lint detector that's more thorough than your grandmother inspecting your room!

This utility runs linters on Python source code files.

[akaihola]

@fry69 I'd like to evaluate the performance of different separators at least against Claude 3.5 Sonnet. I guess that means spending the estimated $4 per separator variant (as mentioned in benchmark/README.md). Any gotchas with the benchmark in your experience, before I proceed to spend the money?

(update: ping @paul-gauthier)

[fry69]

@akaihola I think @paul-gauthier can answer this much better than me.

[akaihola]

I tested the pseudo XML tags from my xml-dividers branch (see Use XML tags to structure your prompts) on o1-preview manually with just a small README file, and it worked fine:

> /model openrouter/openai/o1-preview                                                                                                                                          
Aider v0.59.1.dev13+g70034dc9
Main model: openrouter/openai/o1-preview with diff edit format
Weak model: openrouter/openai/gpt-4o-mini
Git repo: .git with 1 files
Repo-map: using 1024 tokens, auto refresh
Added README.md to the chat.
─────────────────────────────────────────────────────────────────────────────────
README.md                                                                                                                                                                      
> Add some humour to the last paragraph.                                                                                                                                       

To add some humour to the last paragraph in README.md, we'll modify it as follows:                                                                                             

README.md                                                                                                                                                                      

                                                                                                                                                                               
 <AIDER-SEARCH>                                                                                                                                                                
 This project is just for testing pseudo XML style separators for Aider.                                                                                                       
 </AIDER-SEARCH> <AIDER-REPLACE>                                                                                                                                               
 This project is just for testing pseudo XML style separators for Aider. Because real XML was too mainstream!                                                                  
 </AIDER-REPLACE>                                                                                                                                                              
                                                                                                                                                                               


Tokens: 2.5k sent, 87 received. Cost: $0.04 message, $0.05 session.
Applied edit to README.md
─────────────────────────────────────────────────────────────────────────────────
README.md                         

[paul-gauthier]

Not sure of any gotchas. Start with --threads 1 until you feel like you understand how it works, then you can crank up the threads to go faster (and spend money faster).

Benchmarking through open router avoids hitting rate limits.

[akaihola]

As @paul-gauthier mentioned in this comment to #1817, a partial fix landed in Aider on October 5th, 2024. ReStructuredText header underlines longer than 9 characters are no longer mistaken for Aider separators.

Header underlines between 5 and 9 characters still do cause problems. One possible fix would be to transition into using pseudo-XML separators instead (see this comment above).