Skip to content

fix: pass encoding='utf-8' to Path.write_text on all text outputs#34

Open
Ori-Levi1 wants to merge 1 commit into
sentient-agi:mainfrom
Ori-Levi1:fix/utf8-encoding-on-write-text
Open

fix: pass encoding='utf-8' to Path.write_text on all text outputs#34
Ori-Levi1 wants to merge 1 commit into
sentient-agi:mainfrom
Ori-Levi1:fix/utf8-encoding-on-write-text

Conversation

@Ori-Levi1

Copy link
Copy Markdown

Summary

Path.write_text() falls back to locale.getpreferredencoding(False) when the encoding argument is omitted. On Windows machines running a non-UTF-8 system locale (Hebrew/cp1255, Cyrillic/cp1251, Japanese/cp932, etc.), writing strings that contain characters outside the ANSI codepage raises UnicodeEncodeError.

This PR pins encoding='utf-8' on every write_text() call site that emits text content.

Reproduction (Windows + Hebrew locale)

pipx install evoskill
cd <empty-dir>
evoskill init

Crash:

File "...\src\cli\commands\init.py", line 355, in init_cmd
    (evoskill_dir / 'task.md').write_text(TASK_MD_TEMPLATE)
  ...
  File "C:\Python314\Lib\encodings\cp1255.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '→' in position 141: character maps to <undefined>

TASK_MD_TEMPLATE contains (U+2192), which is not representable in cp1255.

Files changed

File Content written Risk if not UTF-8
src/cli/commands/init.py:355 task.md (TASK_MD_TEMPLATE — contains ) Crashes evoskill init immediately
src/cli/commands/init.py:368 state.json Defensive — JSON values may include non-ASCII branch names
src/cli/report.py:62 run-<timestamp>.md (run report) Reports often contain emoji/arrows from agent traces
src/harness/opencode/options.py:107 OpenCode JSON config Defensive — paths/instructions may include non-ASCII
src/harness/opencode/skill_utils.py:76 Evolved SKILL.md (frontmatter + body) Will crash on any evolved skill containing arrows, em-dashes, smart quotes
src/loop/helpers.py:227 Evolved prompt files Will crash on any prompt the proposer generates with non-ASCII
src/loop/runner.py:196 loop_checkpoint.json Defensive — checkpoint values are numeric but should be UTF-8 explicit

Each change is write_text(x)write_text(x, encoding='utf-8'). No behavior change on UTF-8-default systems (most Linux/macOS); fixes the crash on Windows non-UTF-8 locales.

Why this matters beyond the immediate crash

Even if a user gets past evoskill init (e.g., on a US-English Windows install), the SKILL.md and prompt files are written by the LLM-driven proposer. The proposer routinely emits Unicode (em-dashes, arrows, curly quotes) when it rewrites a skill. So evoskill run is also affected on any non-UTF-8 locale, just on a less predictable iteration. Pinning UTF-8 once removes the entire class of bugs.

Out of scope

Two more write_text() call sites exist that were not patched here because I didn't exercise their code paths during reproduction:

  • src/docker/launcher.py:111 (compose file)
  • src/remote/base.py:28 (remote run record)

The same change is recommended in a follow-up.

Test plan

  • Manual: evoskill init succeeds on Windows 11 + Hebrew locale (cp1255) + Python 3.14
  • Manual: evoskill run writes SKILL.md with em-dashes and arrows without error
  • No new tests added — change is a one-character explicit-encoding hint per call site, behavior identical on UTF-8-default systems

Path.write_text() falls back to locale.getpreferredencoding(False) when no
encoding is given. On a Windows machine running a non-UTF-8 system locale
(e.g. Hebrew/cp1255, Cyrillic/cp1251, Japanese/cp932), writing strings that
contain characters outside the ANSI codepage raises UnicodeEncodeError.

Repro on Windows with Hebrew locale (cp1255):
    pipx install evoskill
    cd <empty-dir>
    evoskill init
    # Crash at init.py:355 — TASK_MD_TEMPLATE contains '->' arrow (U+2192)
    # which cannot be encoded to cp1255.

Six write_text() call sites are affected. Every one writes content that may
contain non-ASCII characters (UTF-8 templates, evolved skill bodies,
proposer prompts, JSON checkpoints, markdown reports, OpenCode JSON
config). Pinning encoding='utf-8' fixes the immediate crash and makes the
output deterministic regardless of the operating system locale.

Files:
  src/cli/commands/init.py        # task.md + state.json
  src/cli/report.py               # run-<timestamp>.md
  src/harness/opencode/options.py # opencode JSON config
  src/harness/opencode/skill_utils.py  # SKILL.md frontmatter
  src/loop/helpers.py             # evolved prompt files
  src/loop/runner.py              # loop_checkpoint.json

The two write_text() sites in src/docker/launcher.py (compose file) and
src/remote/base.py (remote run record) are out of scope for this fix
since the user paths that hit them haven't been verified on a Hebrew
locale, but the same change is recommended in a follow-up PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant