Skip to content

Fix UnicodeDecodeError When Reading Job Log Files#45

Merged
AryanAhadinia merged 2 commits into
mainfrom
AryanAhadinia/LogsDecodingErrorFix
Apr 1, 2026
Merged

Fix UnicodeDecodeError When Reading Job Log Files#45
AryanAhadinia merged 2 commits into
mainfrom
AryanAhadinia/LogsDecodingErrorFix

Conversation

@AryanAhadinia
Copy link
Copy Markdown
Member

Problem

Reading log.out / log.err from Slurm and FirecREST jobs could raise a UnicodeDecodeError in two scenarios:

  • The file is read mid-flush, leaving a truncated multi-byte UTF-8 sequence at the end
  • The log contains other malformed byte sequences (e.g. from terminal escape codes or non-UTF-8 tool output)

Example error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 232889: invalid continuation byte

Fix

Adds a decode_log(data: bytes) -> str utility that:

  1. Strips any trailing incomplete multi-byte UTF-8 sequence (handles mid-flush reads cleanly, without replacement characters)
  2. Falls back to errors="replace" for any other malformed bytes elsewhere in the content

Both SlurmLauncher and FirecRESTLauncher now read log files as bytes and pass them through decode_log.

@AryanAhadinia AryanAhadinia requested a review from robmsmt March 30, 2026 13:34
@AryanAhadinia AryanAhadinia self-assigned this Mar 30, 2026
@AryanAhadinia AryanAhadinia added the bug Something isn't working label Mar 30, 2026
@AryanAhadinia AryanAhadinia linked an issue Mar 30, 2026 that may be closed by this pull request
@AryanAhadinia
Copy link
Copy Markdown
Member Author

The Lightweight CI / Tests (Python 3.12) (pull_request) is normal due to the ongoing efforts in #36.

@AryanAhadinia AryanAhadinia force-pushed the AryanAhadinia/LogsDecodingErrorFix branch from 62bfd3b to 7f2385f Compare March 31, 2026 16:13
@AryanAhadinia
Copy link
Copy Markdown
Member Author

AryanAhadinia commented Apr 1, 2026

The Lightweight CI / Tests (Python 3.12) (pull_request) is normal due to the ongoing efforts in #36.

Now fixed.

@AryanAhadinia AryanAhadinia merged commit 8cf087b into main Apr 1, 2026
8 checks passed
@AryanAhadinia AryanAhadinia deleted the AryanAhadinia/LogsDecodingErrorFix branch April 1, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnicodeDecodeError on Live Display

1 participant