Use external gzip utilities for FASTQ IO - #16
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces optimized gzip compression for FASTQ file I/O by leveraging external utilities (pigz/gzip) when available, falling back to Python's native gzip module otherwise. The changes wrap subprocess pipes in file-like interfaces to maintain compatibility with existing code paths and ensure proper resource cleanup.
- Adds subprocess-based gzip reading/writing using pigz (parallel) or gzip utilities
- Implements
_SubprocessFileWrapperto provide file-like interface for subprocess pipes - Adds stream cleanup logic to close input/output streams after command execution
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/heyfastqlib/argparse_types.py | Adds external gzip utility detection, subprocess wrapper class, and integration with GzipFileType for reading/writing compressed files |
| src/heyfastqlib/command.py | Adds stream cleanup logic with try-finally blocks and standard stream protection to ensure proper resource release |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| return_code = self._process.wait() | ||
| if return_code != 0: | ||
| process_exc = subprocess.CalledProcessError(return_code, self._process.args) |
There was a problem hiding this comment.
The subprocess return code check doesn't handle the case where wait() might raise an exception. Additionally, creating CalledProcessError without raising it immediately means subsequent exceptions could suppress this error. Consider raising the CalledProcessError immediately when detected.
Summary
Testing
https://chatgpt.com/codex/tasks/task_e_68f12f6338608323a232b1100a4d1ca6