Skip to content

Use external gzip utilities for FASTQ IO - #16

Closed
Ulthran wants to merge 2 commits into
mainfrom
codex/update-gzip-processing-for-speed
Closed

Use external gzip utilities for FASTQ IO#16
Ulthran wants to merge 2 commits into
mainfrom
codex/update-gzip-processing-for-speed

Conversation

@Ulthran

@Ulthran Ulthran commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

Summary

  • open gzipped FASTQ files by shelling out to pigz when available and fall back to gzip or Python's gzip module
  • wrap subprocess pipes so file-like interfaces remain compatible with existing code paths
  • close CLI input and output streams after each run to flush compression pipelines and release resources

Testing

  • pytest

https://chatgpt.com/codex/tasks/task_e_68f12f6338608323a232b1100a4d1ca6

Copilot AI review requested due to automatic review settings October 16, 2025 18:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces optimized gzip compression for FASTQ file I/O by leveraging external utilities (pigz/gzip) when available, falling back to Python's native gzip module otherwise. The changes wrap subprocess pipes in file-like interfaces to maintain compatibility with existing code paths and ensure proper resource cleanup.

  • Adds subprocess-based gzip reading/writing using pigz (parallel) or gzip utilities
  • Implements _SubprocessFileWrapper to provide file-like interface for subprocess pipes
  • Adds stream cleanup logic to close input/output streams after command execution

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/heyfastqlib/argparse_types.py Adds external gzip utility detection, subprocess wrapper class, and integration with GzipFileType for reading/writing compressed files
src/heyfastqlib/command.py Adds stream cleanup logic with try-finally blocks and standard stream protection to ensure proper resource release

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment on lines +103 to +105
return_code = self._process.wait()
if return_code != 0:
process_exc = subprocess.CalledProcessError(return_code, self._process.args)

Copilot AI Oct 16, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The subprocess return code check doesn't handle the case where wait() might raise an exception. Additionally, creating CalledProcessError without raising it immediately means subsequent exceptions could suppress this error. Consider raising the CalledProcessError immediately when detected.

Copilot uses AI. Check for mistakes.
@Ulthran Ulthran closed this Oct 21, 2025
@Ulthran
Ulthran deleted the codex/update-gzip-processing-for-speed branch October 21, 2025 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants