Commit e635eec
committed
Replace ReadaheadPipeline with anyio producer + sync iterator
The ThreadPoolExecutor-based ReadaheadPipeline and the
_PrefetchIterator wrapper around _variant_chunks_gen are gone. In
their place:
- _produce_variant_chunks: async producer running on the reader's
BlockingPortal. An outer anyio task group manages variant-chunk
fetches; each fetch task uses an inner task group to fan out the
field reads concurrently. The byte-budget refill semantics
(bootstrap chunk runs solo, subsequent chunks scheduled until the
in-flight count exceeds readahead_bytes / per-chunk-bytes) are
preserved. After fetching, the producer applies the variant filter,
materialises the output dict, and sends it through a 1-buffer
MemoryObjectStream. Telemetry — max_in_flight, last_chunk_bytes,
the final iteration log line — is reported via a shared dict.
- _AsyncBackedIterator: sync iterator wrapping the channel via
portal.call. close() cancels the producer task and shuts the
channel; __del__ closes defensively. BaseExceptionGroup is unwrapped
to a single exception so handle_exception in cli.py still surfaces
the original ValueError.
- weakref.finalize arms close on garbage collection. Without it the
portal's daemon thread joins on the asyncio default executor's
non-daemon decode workers and wedges process exit when the user
forgets to use the reader as a context manager.
CLI: --readahead-workers is removed; --io-concurrency caps concurrent
store.get calls (default 32) and --decode-threads sizes the decode
pool (default os.cpu_count()), separating IO from CPU concurrency.
Tests: TestReadaheadPipeline, TestPrefetchIteratorDirect, and the
_DepthTrackingPipeline / _make_pipeline / _shared_test_portal
helpers are deleted. A new TestVariantChunksIterator covers eager
validation, empty-fields short-circuit, exception propagation, close
cancellation, and max_in_flight semantics through the public
variant_chunks() API. The static-field-not-in-pipeline check now
monkeypatches _read_block_async.
Performance benchmarks against the four backends are deferred to
before merging.1 parent 2703b43 commit e635eec
4 files changed
Lines changed: 530 additions & 1054 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
951 | 951 | | |
952 | 952 | | |
953 | 953 | | |
954 | | - | |
955 | | - | |
956 | | - | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
957 | 961 | | |
958 | 962 | | |
959 | 963 | | |
| |||
983 | 987 | | |
984 | 988 | | |
985 | 989 | | |
986 | | - | |
| 990 | + | |
987 | 991 | | |
988 | 992 | | |
989 | 993 | | |
990 | 994 | | |
991 | 995 | | |
992 | 996 | | |
993 | | - | |
| 997 | + | |
| 998 | + | |
994 | 999 | | |
995 | 1000 | | |
996 | 1001 | | |
| |||
1000 | 1005 | | |
1001 | 1006 | | |
1002 | 1007 | | |
1003 | | - | |
| 1008 | + | |
1004 | 1009 | | |
1005 | 1010 | | |
1006 | 1011 | | |
1007 | 1012 | | |
1008 | 1013 | | |
1009 | | - | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
1010 | 1019 | | |
1011 | 1020 | | |
1012 | 1021 | | |
| |||
1015 | 1024 | | |
1016 | 1025 | | |
1017 | 1026 | | |
1018 | | - | |
| 1027 | + | |
1019 | 1028 | | |
1020 | 1029 | | |
1021 | 1030 | | |
1022 | 1031 | | |
1023 | 1032 | | |
1024 | | - | |
| 1033 | + | |
| 1034 | + | |
1025 | 1035 | | |
1026 | 1036 | | |
1027 | 1037 | | |
| |||
0 commit comments