Repository navigation
[Bug]: Edit silently corrupts non-UTF-8 bytes outside the replacement target #3320
Description
Prerequisites
- I searched existing issues and pull requests for encoding, UTF-8, Edit and corruption.
- This is a reproducible bug, not a usage question.
Background / Description
Editing an ASCII setting in a Latin-1 file silently corrupts unrelated text. Both direct Edit and Read followed by Edit with an AgentState report success, but replace the original 0xe9 byte with UTF-8 0xef 0xbf 0xbd (U+FFFD).
Read decodes with errors="replace" and caches that lossy text. Edit then writes the cache as UTF-8. Its no-state path also uses replacement decoding before writing.
Error Messages
There is no error. Both paths return state="running":
cached False edit running bytes b"name = 'caf\xef\xbf\xbd'\ncount = 2\n"
read running " 1\tname = 'caf\ufffd'\n 2\tcount = 1"
cached True edit running bytes b"name = 'caf\xef\xbf\xbd'\ncount = 2\n"
Steps to Reproduce
Run this against current main; it uses only temporary local files and the public tools, with no model or network requests:
import asyncio
import tempfile
from pathlib import Path
from agentscope.state import AgentState
from agentscope.tool import Edit, Read
async def main():
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "config.py"
original = b"name = 'caf\xe9'\ncount = 1\n"
for cached in (False, True):
path.write_bytes(original)
state = AgentState() if cached else None
if state is not None:
read = await Read()(file_path=str(path), _agent_state=state)
print("read", read.state, ascii(read.content[0].text))
result = await Edit()(
file_path=str(path),
old_string="count = 1",
new_string="count = 2",
_agent_state=state,
)
print("cached", cached, "edit", result.state, "bytes", path.read_bytes())
asyncio.run(main())Expected Behavior / Proposed Fix
An edit must not silently replace bytes outside the target. Without an explicit encoding option, rejecting non-UTF-8 text before caching/writing seems safer than guessing an encoding. A narrow fix would use strict UTF-8 decoding in Read and the no-state Edit path, letting their existing error handling return an error while preserving the original bytes. Alternatively, read-only previews could remain lossy, but those previews would need to be excluded from the edit cache.
Environment
- AgentScope:
2.0.11.dev0, main9b94a63407ec59881fde19300b61e469a0a7d8b2 - Python: 3.12.8
- OS: Windows
Reproduced with Codex assistance using the actual local backend, not mocked I/O.
Activity
fancyboi999 commentedon Oct 15, 2026
/assign
github-actions commentedon Oct 15, 2026
It's yours, @fancyboi999. If there is no pull request and no word from you by 2026-10-29 the claim is released so nobody is blocked — a comment here renews it.
fancyboi999 commentedon Oct 15, 2026
Opened #3322.
Readnow decodes strictly. Non-UTF-8 files still get a replacement-character preview with a note, but the lossy text is no longer cached, soEdit/Writerefuse the file instead of rewriting it.Editpath decodes strictly and returns its existing read error, leaving the file untouched.I followed the second option from the issue (lossy preview, kept out of the edit cache) so read-only access to non-UTF-8 files keeps working. Rejecting non-UTF-8 files for editing follows the approach used by Codex, whose file reads use strict
String::from_utf8and fail withInvalidDatainstead of decoding lossily. The reproduction script now returnserroron both paths and the original bytes are preserved.