perf: parallel ISA-L compression and pwrite output pipeline

[KimYannn]

Summary

• Parallel ISA-L gzip compression: Worker threads compress output in parallel using ISA-L, with flight batch management to coordinate ordering, replacing the serial single-writer-thread bottleneck
• Parallel pwrite output: Workers write directly via pwrite() using a lock-free offset ring buffer, eliminating the writer thread as a serialization point
• Bounded mismatch counting: Early-exit overlap analysis rejects non-overlapping offsets after comparing just the first few dozen bytes instead of the full 150bp (~5-10x reduction in comparison work)
• Adaptive timeout with spin backoff: EMA-based ingress rate estimation auto-tunes flush timeout; progressive backoff replaces tight spin-wait in pwrite offset coordination
• Decoupled FASTQ reader/writer with zstd support: Reader and writer paths are decoupled, with zstd input/output support and seekable output with autotuned workers
• Pipeline backpressure and writer flight control: Improved flow control between pipeline stages
• AArch64 static linking fix: Removed full -static linking that caused ISA-L relocation overflow on ARM64
• End-to-end benchmark suite (scripts/bench_e2e.py): PE/SE/stdin-stdout modes, hardware profiling, dual-mode (baseline vs optimized) comparison with JSON merge capability

Key Performance Improvements

Mode	Before	After	Speedup
PE fq→fq (-w 12)	8.21s	6.33s	1.30x
PE fq→gz (parallel ISA-L)	—	significant	eliminates writer bottleneck

Changed Files

• Core: peprocessor.cpp/h, seprocessor.cpp/h, writerthread.cpp/h, writer.cpp/h
• New: isal_compress.h, flight_batch_manager.cpp/h, trace_profiler.cpp/h
• Build: Makefile, .github/workflows/ci.yml
• Bench: scripts/bench_e2e.py (replaces bench_e2e.sh)
• Docs: 3 RFCs in docs/rfc/, 2 benchmark results in docs/bench/

Test Plan

• CI passes on Linux (x86_64) and macOS (ARM64)
• E2E benchmark: python scripts/bench_e2e.py ./fastp_baseline ./fastp_opt
• MD5 verification of output correctness across all modes (PE/SE, fq/gz/zst)
• Memory usage (peak RSS) does not regress significantly
• Verify pwrite gz output is byte-identical to serial writer output

🤖 Generated with Claude Code