Skip to content

Multi threaded for Files and supported formats (Zip, 7Zip and Rar)#1350

Open
adamhathcock wants to merge 4 commits into
masterfrom
adam/multi-threaded
Open

Multi threaded for Files and supported formats (Zip, 7Zip and Rar)#1350
adamhathcock wants to merge 4 commits into
masterfrom
adam/multi-threaded

Conversation

@adamhathcock

Copy link
Copy Markdown
Owner

Addresses: #1001

This pull request introduces a new, extensible framework for concurrent (parallel) extraction of archive entries, laying the groundwork for improved extraction performance when possible. It adds interfaces, options, and implementations to detect when parallel extraction is safe for different archive formats (Zip, Rar, 7z), and exposes new configuration options for users to control extraction parallelism. The changes also refactor the extraction code to use this new system, replacing older, format-specific logic with a unified approach.

The most important changes are:

Parallel Extraction Framework and Options:

  • Introduced the ExtractionParallelism enum and the IExtractionParallelismOptions interface, and updated the ExtractionOptions record to include Parallelism and MaxDegreeOfParallelism properties, allowing users to control the level of concurrency during extraction. [1] [2] [3]

  • Added the ArchiveExtractionConcurrencyInfo, ArchiveExtractionGroup, ArchiveConcurrencyMode, and IArchiveExtractionConcurrencyProvider types to describe and negotiate the concurrency capabilities of an archive at runtime. [1] [2] [3] [4]

Archive Format Support for Parallel Extraction:

  • Implemented the IArchiveExtractionConcurrencyProvider interface for ZipArchive, RarArchive, and SevenZipArchive, allowing each format to report whether parallel extraction is safe and how entries can be grouped for concurrent extraction. [1] [2] [3] [4] [5] [6]

Extraction Pipeline Refactoring:

  • Refactored the WriteToDirectoryAsync extension method to use a new ArchiveExtractionCoordinator (not shown in the diff, but implied by usage), replacing the previous logic for handling solid and non-solid archives with a unified, concurrency-aware approach.

API Enhancements and Utilities:

  • Added new async overloads for ExtractToDirectoryAsync in ArchiveFactory, making it easier to extract archives to directories with optional progress reporting and cancellation.

Error Handling Improvements:

  • Introduced ArchiveExtractionException for improved error reporting during extraction operations.

Copilot AI review requested due to automatic review settings June 6, 2026 14:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a concurrency-aware extraction pipeline to SharpCompress, enabling parallel extraction for file-backed archives when the format/layout can prove it’s safe (notably Zip, Rar, and 7z), and adds API surface/options for users to control that behavior.

Changes:

  • Introduces ExtractionParallelism plus IExtractionParallelismOptions, and extends ExtractionOptions with Parallelism and MaxDegreeOfParallelism.
  • Adds an internal concurrency-negotiation model (ArchiveExtractionConcurrencyInfo, groups/modes, and IArchiveExtractionConcurrencyProvider) and implements it for Zip/Rar/7z.
  • Refactors async directory extraction to route through ArchiveExtractionCoordinator, and adds async ArchiveFactory.ExtractToDirectoryAsync overloads + new test coverage for parallel extraction.

Review notes (issues found):

  • ArchiveExtractionCoordinator.ValidateOptions throws ArgumentOutOfRangeException with paramName set to nameof(options) instead of the invalid property (MaxDegreeOfParallelism). This makes the exception less actionable and is inconsistent with typical .NET usage.
  • Parallel Zip extraction currently re-opens the archive per entry and linearly scans EntriesAsync to find the matching key each time (ExtractZipEntryFromNewArchiveAsync), which is O(entries²) in the number of entries. For larger archives, that can dominate runtime and potentially offset the benefits of parallelism.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs Adds async extraction tests that exercise new parallelism options for Zip.
tests/SharpCompress.Test/Zip/ZipArchiveAsyncParallelExtractionTests.cs New coverage for parallel extraction edge cases (duplicate output paths, cancellation, per-entry failure messaging).
tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncTests.cs Adds tests for RequireParallel behavior on non-solid vs solid 7z archives.
tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncFactoryExtractionTests.cs New factory-level async extraction tests for 7z with RequireParallel.
tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs Adds tests for RequireParallel behavior on non-solid vs solid Rar archives.
tests/SharpCompress.Test/Rar/RarArchiveAsyncFactoryExtractionTests.cs New factory-level async extraction tests for Rar with RequireParallel.
tests/SharpCompress.Test/GZip/GZipArchiveAsyncFactoryExtractionTests.cs Adds a negative test ensuring RequireParallel fails for non-parallelizable formats (GZip).
tests/SharpCompress.Test/ArchiveFactoryAsyncExtractionTests.cs Adds async factory extraction tests including FileInfo + progress forwarding.
src/SharpCompress/Common/Options/IExtractionParallelismOptions.cs New interface to standardize parallelism knobs on option types.
src/SharpCompress/Common/IEntryExtensions.cs Widens GetEntryDestinationFileName access for use by the extraction coordinator.
src/SharpCompress/Common/ExtractionParallelism.cs New enum defining parallel extraction policy options.
src/SharpCompress/Common/ExtractionOptions.cs Extends public extraction options with parallelism controls.
src/SharpCompress/Archives/Zip/ZipArchive.cs Implements concurrency capability reporting for Zip.
src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs Implements concurrency capability reporting and grouping for 7z.
src/SharpCompress/Archives/Rar/RarArchive.cs Implements concurrency capability reporting for Rar.
src/SharpCompress/Archives/IAsyncArchiveExtensions.cs Routes async WriteToDirectoryAsync through the new coordinator.
src/SharpCompress/Archives/Extraction/IArchiveExtractionConcurrencyProvider.cs New internal capability provider contract for formats.
src/SharpCompress/Archives/Extraction/ArchiveExtractionGroup.cs New internal representation for extraction work groups.
src/SharpCompress/Archives/Extraction/ArchiveExtractionException.cs Wraps per-entry/group failures with additional context.
src/SharpCompress/Archives/Extraction/ArchiveExtractionCoordinator.cs Core new extraction orchestrator implementing sequential vs parallel strategies.
src/SharpCompress/Archives/Extraction/ArchiveExtractionConcurrencyInfo.cs Internal capability descriptor for extraction parallelism.
src/SharpCompress/Archives/Extraction/ArchiveConcurrencyMode.cs Internal enum describing extraction concurrency modes.
src/SharpCompress/Archives/ArchiveFactory.Async.cs Adds async convenience overloads for extracting archives to directories.
src/SharpCompress/Archives/AbstractArchive.cs Exposes file-backed source information (SourceFiles) for concurrency decisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants