Multi threaded for Files and supported formats (Zip, 7Zip and Rar)#1350
Open
adamhathcock wants to merge 4 commits into
Open
Multi threaded for Files and supported formats (Zip, 7Zip and Rar)#1350adamhathcock wants to merge 4 commits into
adamhathcock wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a concurrency-aware extraction pipeline to SharpCompress, enabling parallel extraction for file-backed archives when the format/layout can prove it’s safe (notably Zip, Rar, and 7z), and adds API surface/options for users to control that behavior.
Changes:
- Introduces
ExtractionParallelismplusIExtractionParallelismOptions, and extendsExtractionOptionswithParallelismandMaxDegreeOfParallelism. - Adds an internal concurrency-negotiation model (
ArchiveExtractionConcurrencyInfo, groups/modes, andIArchiveExtractionConcurrencyProvider) and implements it for Zip/Rar/7z. - Refactors async directory extraction to route through
ArchiveExtractionCoordinator, and adds asyncArchiveFactory.ExtractToDirectoryAsyncoverloads + new test coverage for parallel extraction.
Review notes (issues found):
ArchiveExtractionCoordinator.ValidateOptionsthrowsArgumentOutOfRangeExceptionwithparamNameset tonameof(options)instead of the invalid property (MaxDegreeOfParallelism). This makes the exception less actionable and is inconsistent with typical .NET usage.- Parallel Zip extraction currently re-opens the archive per entry and linearly scans
EntriesAsyncto find the matching key each time (ExtractZipEntryFromNewArchiveAsync), which is O(entries²) in the number of entries. For larger archives, that can dominate runtime and potentially offset the benefits of parallelism.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs | Adds async extraction tests that exercise new parallelism options for Zip. |
| tests/SharpCompress.Test/Zip/ZipArchiveAsyncParallelExtractionTests.cs | New coverage for parallel extraction edge cases (duplicate output paths, cancellation, per-entry failure messaging). |
| tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncTests.cs | Adds tests for RequireParallel behavior on non-solid vs solid 7z archives. |
| tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncFactoryExtractionTests.cs | New factory-level async extraction tests for 7z with RequireParallel. |
| tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs | Adds tests for RequireParallel behavior on non-solid vs solid Rar archives. |
| tests/SharpCompress.Test/Rar/RarArchiveAsyncFactoryExtractionTests.cs | New factory-level async extraction tests for Rar with RequireParallel. |
| tests/SharpCompress.Test/GZip/GZipArchiveAsyncFactoryExtractionTests.cs | Adds a negative test ensuring RequireParallel fails for non-parallelizable formats (GZip). |
| tests/SharpCompress.Test/ArchiveFactoryAsyncExtractionTests.cs | Adds async factory extraction tests including FileInfo + progress forwarding. |
| src/SharpCompress/Common/Options/IExtractionParallelismOptions.cs | New interface to standardize parallelism knobs on option types. |
| src/SharpCompress/Common/IEntryExtensions.cs | Widens GetEntryDestinationFileName access for use by the extraction coordinator. |
| src/SharpCompress/Common/ExtractionParallelism.cs | New enum defining parallel extraction policy options. |
| src/SharpCompress/Common/ExtractionOptions.cs | Extends public extraction options with parallelism controls. |
| src/SharpCompress/Archives/Zip/ZipArchive.cs | Implements concurrency capability reporting for Zip. |
| src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs | Implements concurrency capability reporting and grouping for 7z. |
| src/SharpCompress/Archives/Rar/RarArchive.cs | Implements concurrency capability reporting for Rar. |
| src/SharpCompress/Archives/IAsyncArchiveExtensions.cs | Routes async WriteToDirectoryAsync through the new coordinator. |
| src/SharpCompress/Archives/Extraction/IArchiveExtractionConcurrencyProvider.cs | New internal capability provider contract for formats. |
| src/SharpCompress/Archives/Extraction/ArchiveExtractionGroup.cs | New internal representation for extraction work groups. |
| src/SharpCompress/Archives/Extraction/ArchiveExtractionException.cs | Wraps per-entry/group failures with additional context. |
| src/SharpCompress/Archives/Extraction/ArchiveExtractionCoordinator.cs | Core new extraction orchestrator implementing sequential vs parallel strategies. |
| src/SharpCompress/Archives/Extraction/ArchiveExtractionConcurrencyInfo.cs | Internal capability descriptor for extraction parallelism. |
| src/SharpCompress/Archives/Extraction/ArchiveConcurrencyMode.cs | Internal enum describing extraction concurrency modes. |
| src/SharpCompress/Archives/ArchiveFactory.Async.cs | Adds async convenience overloads for extracting archives to directories. |
| src/SharpCompress/Archives/AbstractArchive.cs | Exposes file-backed source information (SourceFiles) for concurrency decisions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses: #1001
This pull request introduces a new, extensible framework for concurrent (parallel) extraction of archive entries, laying the groundwork for improved extraction performance when possible. It adds interfaces, options, and implementations to detect when parallel extraction is safe for different archive formats (Zip, Rar, 7z), and exposes new configuration options for users to control extraction parallelism. The changes also refactor the extraction code to use this new system, replacing older, format-specific logic with a unified approach.
The most important changes are:
Parallel Extraction Framework and Options:
Introduced the
ExtractionParallelismenum and theIExtractionParallelismOptionsinterface, and updated theExtractionOptionsrecord to includeParallelismandMaxDegreeOfParallelismproperties, allowing users to control the level of concurrency during extraction. [1] [2] [3]Added the
ArchiveExtractionConcurrencyInfo,ArchiveExtractionGroup,ArchiveConcurrencyMode, andIArchiveExtractionConcurrencyProvidertypes to describe and negotiate the concurrency capabilities of an archive at runtime. [1] [2] [3] [4]Archive Format Support for Parallel Extraction:
IArchiveExtractionConcurrencyProviderinterface forZipArchive,RarArchive, andSevenZipArchive, allowing each format to report whether parallel extraction is safe and how entries can be grouped for concurrent extraction. [1] [2] [3] [4] [5] [6]Extraction Pipeline Refactoring:
WriteToDirectoryAsyncextension method to use a newArchiveExtractionCoordinator(not shown in the diff, but implied by usage), replacing the previous logic for handling solid and non-solid archives with a unified, concurrency-aware approach.API Enhancements and Utilities:
ExtractToDirectoryAsyncinArchiveFactory, making it easier to extract archives to directories with optional progress reporting and cancellation.Error Handling Improvements:
ArchiveExtractionExceptionfor improved error reporting during extraction operations.