Add opt-in multi-threading support for file-based archive extraction#1143
Add opt-in multi-threading support for file-based archive extraction#1143adamhathcock with Copilot wants to merge 6 commits into
Conversation
Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>
|
@copilot add a boolean flag into Archive that let's people know multithreaded is supported. Add an explicit option on the ArchiveOptions class to say if a user wants to allow support if it makes sense |
…on flag - Added IArchive.SupportsMultiThreadedExtraction property to indicate if multi-threading is supported - Added ReaderOptions.EnableMultiThreadedExtraction option to opt-in to multi-threading - Updated SeekableZipFilePart, TarFilePart, and SeekableFilePart to check the flag - Added test to verify multi-threading flag behavior - Multi-threading is now disabled by default for backward compatibility Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>
Added in commit 4a6e523:
All format-specific classes now check this flag before creating independent streams. |
When an archive is opened from a
FileInfoor file path, multiple threads can now extract different entries concurrently when explicitly enabled. Previously, all extractions shared a singleFileStream, causing position conflicts and corruption.Changes
Core Infrastructure
SourceStream: AddedCreateIndependentStream(volumeIndex)to create newFileStreaminstances from trackedFileInfoobjectsSemaphoreSlimfor async) to prevent concurrent header load racesOpt-in Multi-threading Support
IArchive.SupportsMultiThreadedExtraction: Boolean property indicating if multi-threaded extraction is supported for the archive instancetruewhen archive is opened from a file, multi-threading is enabled, and archive is not SOLIDReaderOptions.EnableMultiThreadedExtraction: Explicit opt-in flag to enable multi-threaded extraction (defaults tofalsefor backward compatibility)Format Support
Modified entry extraction to create independent streams when file-based and multi-threading is enabled:
SeekableZipFilePart: Both header loading and data extraction use independent streams when flag is enabledTarFilePart: Data extraction uses independent streams when flag is enabledSeekableFilePart(Rar): Data extraction uses independent streams when flag is enabledFallback Behavior
Streaminstead ofFileInfo) continue using shared streams - backward compatible, no multi-threading supportExample Usage
Known Limitations
Testing
Added
ZipMultiThreadTestsandTarMultiThreadTestsdemonstrating concurrent extraction patterns. Tests verify that:SupportsMultiThreadedExtractionproperty accurately reflects multi-threading availability<issue_title>Support multi-threading when supplying a path/FileInfo</issue_title>
><issue_description>Since the ultimate source is a local file, there's no reason multi-threading won't work for some formats. What is required:
>
> - a format that supports header reading separate from bytes or one that can skip entries (e.g. Zip, Rar, Tar)
> - compressed entries can be read from separate parts simultaneously (probably most?)
> - known non-supporting as things are single compressed streams that usually wrap a Tar (Gzip, BZip2, etc)
> - unknown: "SOLID" support. SOLID Rars and 7ZIps compress headers with entry bytes. 7Zip might be able to work but differently. If the entire Rar is SOLID then it's like a Gzipped Tar which can't really work either.</issue_description>
>
><agent_instructions>make sure things are FileInfo and RandomAccess. Allow multiple independent streams to ensure isolation across threads. new streams should probably seek back to the file to get the stream of bytes</agent_instructions>
>
> ## Comments on the Issue (you are @copilot in this section)
>
>
>
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.