Add async ZipFile APIs

[ericstj]

All ZipFile APIs are currently synchronous. This means manipulations to zip files will always block a thread. We should investigate using an async file and calling async IO APIs (ReadAsync/WriteAsync) to free up the thread while IO is happening. dotnet/corefx#5680

---

Edit by @carlossanlop:

System.IO.Compression

- public partial class ZipArchive : IDisposable
+ public partial class ZipArchive : IAsyncDisposable, IDisposable
  {
      public void Dispose();
      protected virtual void Dispose(bool disposing);
+     public ValueTask DisposeAsync(); // From the new implemented interface

      public ReadOnlyCollection<ZipArchiveEntry> Entries { get; }
+     public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync(CancellationToken cancellationToken = default);
+ // Should this use IAsyncEnumerable?

      public ZipArchiveEntry? GetEntry(string entryName);
+     public Task<ZipArchiveEntry?> GetEntryAsync(string entryName, CancellationToken cancellationToken = default);
...

Note there is no CreateEntryAsync above. That is because I didn't find any code inside CreateEntry (or inside DoCreateEntry) that can be made async, Aside for the cancellationToken.ThrowIfCancellationRequested() call that would have to be added at the top.

But here's the method in case we want to add it regardless. We can get it approved here, and if in the end it is not needed, we don't have to ship it:

...
      public ZipArchiveEntry CreateEntry(string entryName);
+     public Task<ZipArchiveEntry> CreateEntryAsync(string entryName, CancellationToken cancellationToken = default);  
  }

Also:

  public partial class ZipArchiveEntry
  {
      void Delete();
+     public Task DeleteAsync(CancellationToken cancellationToken = default);

      public Stream Open();
+     Task<Stream> OpenAsync(CancellationToken cancellationToken = default);
  }

Usage examples:

// Open read, get entries, get entry stream
FileStream fs = File.OpenRead("/path/to/archive.zip");
await using (ZipArchive archive =  new ZipArchive(stream: fs, ZipArchiveMode.Read, ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        Stream s = await entry.OpenAsync(ct);
        // Do something
    }
}

// Open update, get entries, delete entry
FileStream fs = File.OpenRead("/path/to/archive.zip");
await using (ZipArchive archive = new ZipArchive(stream: fs, ZipArchiveMode.Update, ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        if (/* some condition */)
        {
            await entry.DeleteAsync(ct);
        }
    }
}

System.IO.Compression.ZipFile

  public static partial class ZipFile
  {
      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static ZipArchive Open(string archiveFileName, ZipArchiveMode mode);
+     public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, CancellationToken cancellationToken = default);

      public static ZipArchive Open(string archiveFileName, ZipArchiveMode mode, Encoding? entryNameEncoding);
+     public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static ZipArchive OpenRead(string archiveFileName);
+     public static Task<ZipArchive> OpenReadAsync(string archiveFileName, CancellationToken cancellationToken = default);
  }
  
  public static partial class ZipFileExtensions
  {
      public static ZipArchiveEntry CreateEntryFromFile(this ZipArchive destination, string sourceFileName, string entryName);
+     public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CancellationToken cancellationToken = default);

      public static ZipArchiveEntry CreateEntryFromFile(this ZipArchive destination, string sourceFileName, string entryName, CompressionLevel compressionLevel);
+     public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CompressionLevel compressionLevel, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(this ZipArchive source, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(this ZipArchive source, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToFile(this ZipArchiveEntry source, string destinationFileName);
+     public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, CancellationToken cancellationToken = default);

      public static void ExtractToFile(this ZipArchiveEntry source, string destinationFileName, bool overwrite);
+     public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, bool overwrite, CancellationToken cancellationToken = default);
  }

Usage examples

/// ZipFile creation and extraction

MemoryStream ms = new();
await CreateFromDirectoryAsync(sourceDirectoryName: "C:/path/to/source/folder", destination: ms, ct);
ms.Position = 0;
await ExtractToDirectoryAsync(source: ms, destinationDirectoryName: "C:/path/to/destination/folder", ct);

/// ZipFile open, ZipFileExtensions create entry from file

await using (ZipArchive archive = await ZipFile.OpenAsync(archiveFileName: "/path/to/archive.zip", ZipArchiveMode.Update, ct))
{
    await archive.CreateEntryFromFileAsync(sourceFileName: "/path/to/file.txt", entryName: "file.txt", ct);
}

// ZipFile open read, ZipFileExtensions extract entry to file

await using (ZipArchive archive = await ZipFile.OpenReadAsync(archiveFileName: "/path/to/archive.zip", ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        string fileName = Path.Combine(rootFolder, entry.Name);
        await archive.ExtractToFileAsync(destinationFileName: fileName, overwrite: true, ct);
    }
    // or instead
    await archive.ExtractToDirectoryAsync(rootFolder, ct);
}

Plan

• Get the APIs approved for System.IO.Compression and System.IO.Compression.ZipFile.
• 1st round: Implementation of System.IO.Compression APIs:
  • Change the block structs to classes. Change ZipBlocks structs to classes #113453
  • Add zip benchmarks Add benchmarks for ZipFile.CreateFromDirectory and ZipFile.ExtractToDirectory performance#4764
  • Create the async version of the internal APIs, reuse as much code as needed from the sync version.
  • Implement the public async APIs.
  • Add unit tests.
  • Add XML comments to the public APIs.
• Find locations in the dotnet repos where the async APIs of System.IO.Compression could be used and change them.
• 2nd round: Implementation of System.IO.Compression.ZipFile APIs:
  • Implement the public async APIs.
  • Add unit tests.
  • Add XML comments to the public APIs.
• Find locations in the dotnet repos where the async APIs of System.IO.Compression could be used and change them.
• 3rd round: Create a proposal for System.IO.Packaging async APIs (Future).

[danmoseley]

This needs someone to write up a formal API proposal then we can review and if approved, offer up for implementation. Anyone?

Docs on the API review process and see Jon's API request for a great example of a strong proposal. The more concrete the proposal is (e.g. has examples of usage, real-world scenarios, fleshed out API), the more discussion it will start and the better the chances of us being able to push the addition through the review process.

[abdulbeard]

Forked the repo, and working on an api proposal.
Seems like most of the work outlined in this item is already done. e.g
https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem/src/System/IO/File.cs#L682
https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem/src/System/IO/File.cs#L732

That probably means this item is done, right?
@danmosemsft

[ianhays]

Not quite, @abdulbeard. This particular issue is around adding ZipFile async APIs.

For example, currently we have ZipFile.ExtractToDirectory(string, string). This issue is around adding async versions of those API e.g. ZipFile.ExtractToDirectoryAsync(string, string).

[abdulbeard]

Gotcha @ianhays . I did a terrible job reading the title 😛

[abdulbeard]

Summary

Curently, System.IO.Compression.Zipfile doesn't have async methods for CreateFromDirectory, ExtractToDirectory, Open and OpenRead. This means that these zipfile operations are thread-blocking, and not asynchronous, and hence, comparatively less performant.

Proposed API

namespace System.IO.Compression {
    public partial class Zipfile {
        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CancellationToken cancellationToken = default(CancellationToken));

        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName,	CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default(CancellationToken));

        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName,	CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, CancellationToken cancellationToken = default(CancellationToken));

        public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenReadAsync(string archiveFileName, CancellationToken cancellationToken = default(CancellationToken));
    }
}

Expected Use and Benefits

These new functions are asynchronous versions of their synchronous counterparts. This allows the .net runtime to allow the thread to do other work while the asynchronous operation completes (as mandated by the TPL and async/await scheduling).

• Zipfile operations will no longer be thread-blocking, improving performance.
• Long-running Zipfile operations can now be cancelled via the CancellationToken, by the user, allowing them more control.
• Asynchronous FileStream or other OS/runtime optimizations can be utilized to improve performance here.

[abdulbeard]

Here's a proposal ^. Looking forward to constructive feedback and insight.

[svick]

Should async APIs be added to ZipArchive at the same time? For example, consider the following code:

using (var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update))
{
    archive.CreateEntry("empty.txt");
}

using (var archive = await ZipFile.OpenReadAsync("archive.zip"))
{
    foreach (var entry in archive.Entries)
    {
        Console.WriteLine(entry.FullName);
    }
}

As far as I can tell from skimming the source code, even though the above code uses the proposed async APIs as much as it can, it still blocks when accessing Entries and, more insidiously, when disposing the first archive.

[abdulbeard]

Here's a proposal for ZipArchive class:

namespace System.IO.Compression {
    public class ZipArchive {
        public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync();
        public Task<ZipArchiveEntry> GetEntryAsync(string entryName, CancellationToken cancellationToken = default(cancellationToken));
    }
}

[svick]

@abdulbeard How does that help? As far as I can tell, CreateEntry does not block in the current implementation. On the other hand, accessing Entries and calling Dispose() can block, depending on the mode.

[abdulbeard]

@svick you're right. Async versions of CreateEntry don't do much, since their underlying functionality is not blocking.
On the matter of Dispose() being async, not sure there's a good way to do that.
In the matter of Entries, I can imagine replacing that with something like:

public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync();
public Task<ZipArchiveEntry> GetEntryAsync(string entryName, CancellationToken cancellationToken = default(cancellationToken));

Amended the proposal above.

[danmoseley]

@JeremyKuhne could you give feedback on this proposal? Perhaps we can move it forward

[svick]

@abdulbeard

On the matter of Dispose() being async, not sure there's a good way to do that.

Yeah, there isn't a great approach until dotnet/roslyn#114 is added. But in the meantime, I think there are ways to make it work. The approaches I considered are:

• Add FlushAsync(). This would write the current state of the ZipArchive to disk, which means that calling Dispose() right after FlushAsync() would not block.

  Example usage:

  using (var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update))
  {
      archive.CreateEntry("empty.txt");
      await archive.FlushAsync();
  }

• Add DisposeAsync(). Once IAsyncDisposable, using statements, and async/await roslyn#114 is added, you would be able to use it with using.

  Example usage:

  var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update)
  try
  {
      archive.CreateEntry("empty.txt");
  }
  finally
  {
      await archive.DisposeAsync();
  }

Looking at the two examples above, I think adding FlushAsync() is the better option.

[abdulbeard]

@svick
To me it makes sense to wait on resolution of dotnet/roslyn/#114 before deciding on FlusyAsync vs DisposeAsync.
But I can also see value in finishing out this stage with FlushAsync, and then once we have an async dispose, let that replace FlushAsync.
But I'd love to hear from @JeremyKuhne etc. as well.