Skip to content

[Bug]: Edit silently corrupts non-UTF-8 bytes outside the replacement target #3320

Open

Description

@ch-z-hc

Prerequisites

  • I searched existing issues and pull requests for encoding, UTF-8, Edit and corruption.
  • This is a reproducible bug, not a usage question.

Background / Description

Editing an ASCII setting in a Latin-1 file silently corrupts unrelated text. Both direct Edit and Read followed by Edit with an AgentState report success, but replace the original 0xe9 byte with UTF-8 0xef 0xbf 0xbd (U+FFFD).

Read decodes with errors="replace" and caches that lossy text. Edit then writes the cache as UTF-8. Its no-state path also uses replacement decoding before writing.

Error Messages

There is no error. Both paths return state="running":

cached False edit running bytes b"name = 'caf\xef\xbf\xbd'\ncount = 2\n"
read running "     1\tname = 'caf\ufffd'\n     2\tcount = 1"
cached True edit running bytes b"name = 'caf\xef\xbf\xbd'\ncount = 2\n"

Steps to Reproduce

Run this against current main; it uses only temporary local files and the public tools, with no model or network requests:

import asyncio
import tempfile
from pathlib import Path

from agentscope.state import AgentState
from agentscope.tool import Edit, Read


async def main():
    with tempfile.TemporaryDirectory() as directory:
        path = Path(directory) / "config.py"
        original = b"name = 'caf\xe9'\ncount = 1\n"
        for cached in (False, True):
            path.write_bytes(original)
            state = AgentState() if cached else None
            if state is not None:
                read = await Read()(file_path=str(path), _agent_state=state)
                print("read", read.state, ascii(read.content[0].text))
            result = await Edit()(
                file_path=str(path),
                old_string="count = 1",
                new_string="count = 2",
                _agent_state=state,
            )
            print("cached", cached, "edit", result.state, "bytes", path.read_bytes())


asyncio.run(main())

Expected Behavior / Proposed Fix

An edit must not silently replace bytes outside the target. Without an explicit encoding option, rejecting non-UTF-8 text before caching/writing seems safer than guessing an encoding. A narrow fix would use strict UTF-8 decoding in Read and the no-state Edit path, letting their existing error handling return an error while preserving the original bytes. Alternatively, read-only previews could remain lossy, but those previews would need to be excluded from the edit cache.

Environment

  • AgentScope: 2.0.11.dev0, main 9b94a63407ec59881fde19300b61e469a0a7d8b2
  • Python: 3.12.8
  • OS: Windows

Reproduced with Codex assistance using the actual local backend, not mocked I/O.

Activity

  1. added
    triage/verifyingA bot is verifying whether this bug is real
    triage/confirmedVerified: the reported defect exists
    and removed
    triage/verifyingA bot is verifying whether this bug is real
    on Oct 15, 2026
  2. fancyboi999 commented on Oct 15, 2026

    @fancyboi999
    Contributor

    /assign

  3. github-actions commented on Oct 15, 2026

    @github-actions
    Contributor

    It's yours, @fancyboi999. If there is no pull request and no word from you by 2026-10-29 the claim is released so nobody is blocked — a comment here renews it.

  4. fancyboi999 commented on Oct 15, 2026

    @fancyboi999
    Contributor

    Opened #3322.

    • Read now decodes strictly. Non-UTF-8 files still get a replacement-character preview with a note, but the lossy text is no longer cached, so Edit/Write refuse the file instead of rewriting it.
    • The no-state Edit path decodes strictly and returns its existing read error, leaving the file untouched.

    I followed the second option from the issue (lossy preview, kept out of the edit cache) so read-only access to non-UTF-8 files keeps working. Rejecting non-UTF-8 files for editing follows the approach used by Codex, whose file reads use strict String::from_utf8 and fail with InvalidData instead of decoding lossily. The reproduction script now returns error on both paths and the original bytes are preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

triage/confirmedVerified: the reported defect exists

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions