Skip to content

Add opt-in multi-threading support for file-based archive extraction#1143

Draft
adamhathcock with Copilot wants to merge 6 commits into
masterfrom
copilot/support-multi-threading-path
Draft

Add opt-in multi-threading support for file-based archive extraction#1143
adamhathcock with Copilot wants to merge 6 commits into
masterfrom
copilot/support-multi-threading-path

Conversation

Copilot AI commented Jan 18, 2026

Copy link
Copy Markdown
Contributor

When an archive is opened from a FileInfo or file path, multiple threads can now extract different entries concurrently when explicitly enabled. Previously, all extractions shared a single FileStream, causing position conflicts and corruption.

Changes

Core Infrastructure

  • SourceStream: Added CreateIndependentStream(volumeIndex) to create new FileStream instances from tracked FileInfo objects
  • Thread-safe header loading: Added synchronization (lock for sync, SemaphoreSlim for async) to prevent concurrent header load races

Opt-in Multi-threading Support

  • IArchive.SupportsMultiThreadedExtraction: Boolean property indicating if multi-threaded extraction is supported for the archive instance
    • Returns true when archive is opened from a file, multi-threading is enabled, and archive is not SOLID
  • ReaderOptions.EnableMultiThreadedExtraction: Explicit opt-in flag to enable multi-threaded extraction (defaults to false for backward compatibility)

Format Support

Modified entry extraction to create independent streams when file-based and multi-threading is enabled:

  • SeekableZipFilePart: Both header loading and data extraction use independent streams when flag is enabled
  • TarFilePart: Data extraction uses independent streams when flag is enabled
  • SeekableFilePart (Rar): Data extraction uses independent streams when flag is enabled

Fallback Behavior

  • Stream-based archives (opened from Stream instead of FileInfo) continue using shared streams - backward compatible, no multi-threading support
  • File-based archives with multi-threading disabled (default) use existing single-stream behavior

Example Usage

var options = new ReaderOptions { EnableMultiThreadedExtraction = true };
using var archive = ZipArchive.OpenArchive("large.zip", options);

// Check if multi-threading is supported
if (archive.SupportsMultiThreadedExtraction)
{
    // Extract multiple entries concurrently
    Parallel.ForEach(archive.Entries.Where(e => !e.IsDirectory), entry => {
        using var stream = entry.OpenEntryStream();
        using var output = File.Create($"output/{entry.Key}");
        stream.CopyTo(output);
    });
}

Known Limitations

  • Async concurrent operations have timing issues with header loading coordination
  • SOLID archives (some Rar, all 7Zip) should use sequential extraction for performance
  • Streaming compression formats (GZip, BZip2 wrapping Tar) cannot benefit from concurrent extraction
  • Multi-threading is opt-in and disabled by default for backward compatibility

Testing

Added ZipMultiThreadTests and TarMultiThreadTests demonstrating concurrent extraction patterns. Tests verify that:

  • Multi-threading is disabled by default
  • SupportsMultiThreadedExtraction property accurately reflects multi-threading availability
  • Extraction works correctly with and without multi-threading enabled

<issue_title>Support multi-threading when supplying a path/FileInfo</issue_title>
><issue_description>Since the ultimate source is a local file, there's no reason multi-threading won't work for some formats. What is required:
>
> - a format that supports header reading separate from bytes or one that can skip entries (e.g. Zip, Rar, Tar)
> - compressed entries can be read from separate parts simultaneously (probably most?)
> - known non-supporting as things are single compressed streams that usually wrap a Tar (Gzip, BZip2, etc)
> - unknown: "SOLID" support. SOLID Rars and 7ZIps compress headers with entry bytes. 7Zip might be able to work but differently. If the entire Rar is SOLID then it's like a Gzipped Tar which can't really work either.</issue_description>
>
><agent_instructions>make sure things are FileInfo and RandomAccess. Allow multiple independent streams to ensure isolation across threads. new streams should probably seek back to the file to get the stream of bytes</agent_instructions>
>
> ## Comments on the Issue (you are @copilot in this section)
>
>
>

Original prompt

This section details on the original issue you should resolve

<issue_title>Support multi-threading when supplying a path/FileInfo</issue_title>
<issue_description>Since the ultimate source is a local file, there's no reason multi-threading won't work for some formats. What is required:

  • a format that supports header reading separate from bytes or one that can skip entries (e.g. Zip, Rar, Tar)
  • compressed entries can be read from separate parts simultaneously (probably most?)
  • known non-supporting as things are single compressed streams that usually wrap a Tar (Gzip, BZip2, etc)
  • unknown: "SOLID" support. SOLID Rars and 7ZIps compress headers with entry bytes. 7Zip might be able to work but differently. If the entire Rar is SOLID then it's like a Gzipped Tar which can't really work either.</issue_description>

<agent_instructions>make sure things are FileInfo and RandomAccess. Allow multiple independent streams to ensure isolation across threads. new streams should probably seek back to the file to get the stream of bytes</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>
Copilot AI changed the title [WIP] Support multi-threading for path/FileInfo processing Add multi-threading support for file-based archive extraction Jan 18, 2026
Copilot AI requested a review from adamhathcock January 18, 2026 16:17
@adamhathcock

Copy link
Copy Markdown
Owner

@copilot add a boolean flag into Archive that let's people know multithreaded is supported. Add an explicit option on the ArchiveOptions class to say if a user wants to allow support if it makes sense

…on flag

- Added IArchive.SupportsMultiThreadedExtraction property to indicate if multi-threading is supported
- Added ReaderOptions.EnableMultiThreadedExtraction option to opt-in to multi-threading
- Updated SeekableZipFilePart, TarFilePart, and SeekableFilePart to check the flag
- Added test to verify multi-threading flag behavior
- Multi-threading is now disabled by default for backward compatibility

Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot AI commented Jan 18, 2026

Copy link
Copy Markdown
Contributor Author

@copilot add a boolean flag into Archive that let's people know multithreaded is supported. Add an explicit option on the ArchiveOptions class to say if a user wants to allow support if it makes sens...

Added in commit 4a6e523:

  1. IArchive.SupportsMultiThreadedExtraction property - Returns true when the archive is opened from a file, multi-threading is enabled in options, and the archive is not SOLID.

  2. ReaderOptions.EnableMultiThreadedExtraction option - Explicit opt-in flag (defaults to false for backward compatibility). When enabled, creates independent FileStreams for concurrent extraction.

All format-specific classes now check this flag before creating independent streams.

Copilot AI changed the title Add multi-threading support for file-based archive extraction Add opt-in multi-threading support for file-based archive extraction Jan 18, 2026
# Conflicts:
#	src/SharpCompress/Common/Zip/SeekableZipFilePart.cs
#	src/SharpCompress/Readers/ReaderOptions.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multi-threading when supplying a path/FileInfo

2 participants