Windows UTF-8 output crash in search.py + content_extract misclassifies MinerU JSON errors

[icy-forever]

Summary

While testing openclaw-search-skills on Windows, I found two reproducible issues:

1. search-layer/scripts/search.py can crash on Windows consoles with UnicodeEncodeError because stdout/stderr encoding is not forced to UTF-8.
2. content-extract/scripts/content_extract.py misclassifies MinerU wrapper failures when mineru_parse_documents.py returns structured JSON errors with exit code 2 (for example missing MINERU_TOKEN).

Environment

• OS: Windows 11
• Python: system Python on Windows shell
• Repo: current main as of 2026-03-17

Issue 1: Windows encoding crash in search.py

Repro

Run:

python search-layer/scripts/search.py "OpenClaw config docs" --mode deep --source exa,tavily --num 2

On Windows terminals using GBK / cp936, JSON output containing Unicode characters can fail with UnicodeEncodeError when printing to stdout.

Expected

The script should print UTF-8 JSON reliably across platforms.

Actual

The script may crash while writing output.

Minimal fix

At startup, reconfigure stdout/stderr to UTF-8 when available:

try:
    if hasattr(sys.stdout, "reconfigure"):
        sys.stdout.reconfigure(encoding="utf-8")
    if hasattr(sys.stderr, "reconfigure"):
        sys.stderr.reconfigure(encoding="utf-8")
except Exception:
    pass

Issue 2: content_extract.py treats structured MinerU errors as wrapper crashes

Repro

Run without MINERU_TOKEN configured:

python content-extract/scripts/content_extract.py --url "https://example.com"

mineru_parse_documents.py prints a valid JSON error payload and exits with code 2, but content_extract.py currently treats return codes outside (0, 1) as a wrapper crash before attempting to parse stdout.

Expected

If stdout contains valid JSON, content_extract.py should parse it first and surface the structured error.

Actual

It returns:

• notes: ["mineru wrapper crashed", ...]

which is misleading because the wrapper did not crash — it returned a meaningful JSON error.

Suggested fix

Parse p.stdout as JSON first. Only label it as wrapper crashed when stdout is not JSON and the process truly failed unexpectedly.

Additional note

The current MinerU integration appears to require MINERU_TOKEN only. Users who have MinerU Access Key / Secret Key credentials cannot use them directly with the current implementation. That may be a useful feature request, but I am not filing it as a bug here because the current docs explicitly mention MINERU_TOKEN.

[blessonism]

Opened PR #3 to fix this issue: #3

Included fixes:

• best-effort UTF-8 stdio setup in
• structured MinerU JSON errors are no longer misclassified as wrapper crashes in

Validated locally with:

• UTF-8 stdio reconfigure path
• JSON error payload + exit code 2
• non-JSON crash + non-zero exit

[blessonism]

Follow-up with clean formatting:

Opened PR #3 to fix this issue: #3

Included fixes:

• best-effort UTF-8 stdio setup in search.py
• structured MinerU JSON errors are no longer misclassified as wrapper crashes in content_extract.py

Validated locally with:

• UTF-8 stdio reconfigure path
• JSON error payload + exit code 2
• non-JSON crash + non-zero exit

[blessonism]

Merged via PR #3: #3

Closing this issue as fixed.