Skip to content

[codex] Test encoding edge cases across request stacks#1551

Merged
maurosoria merged 1 commit into
rust-enginefrom
codex/encoding-edge-case-fixes
May 29, 2026
Merged

[codex] Test encoding edge cases across request stacks#1551
maurosoria merged 1 commit into
rust-enginefrom
codex/encoding-edge-case-fixes

Conversation

@maurosoria
Copy link
Copy Markdown
Owner

Summary

Adds regression coverage for network encoding edge cases across the sync Python, async Python, and native Rust request stacks.

Changes

  • Preserve and classify malformed response/read errors more precisely in Python requesters.
  • Decode text bodies with replacement instead of dropping invalid bytes silently.
  • Add display-only sanitization/truncation for bidi controls, ZWJ-heavy emoji paths, redirects, and queued directories.
  • Preserve raw HTTP request targets in the native backend when dot segments would otherwise be normalized by the URL layer.
  • Add tests for CJK byte length, German case expansion, UTF-16 BOM artifacts, lying charset bytes, Shift-JIS backslash overlap, RTL controls, and large ZWJ emoji sequences.

Validation

  • VIRTUAL_ENV=/home/mauro/dirsearch/.venv PATH=/home/mauro/dirsearch/.venv/bin:$PATH /home/mauro/dirsearch/.venv/bin/python -m maturin develop --manifest-path native/Cargo.toml
  • /home/mauro/dirsearch/.venv/bin/python -m unittest tests.connection.test_response tests.connection.test_native_response tests.connection.test_requester tests.utils.test_crawl tests.utils.test_terminal
  • /home/mauro/dirsearch/.venv/bin/python -m unittest discover tests
  • /home/mauro/dirsearch/.venv/bin/python testing.py
  • cargo test --manifest-path native/Cargo.toml
  • git diff --check

Note: test_schemedet still emits existing ResourceWarnings for unclosed SSL sockets; the suite passes.

@maurosoria maurosoria marked this pull request as ready for review May 29, 2026 11:41
@maurosoria maurosoria merged commit 15f1a19 into rust-engine May 29, 2026
0 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant