Add Support for Brotli Compression with Shared Dictionaries in .NET

[vsfeedback]

This issue has been moved from a ticket on Developer Community.

---

The current implementation of Brotli compression in .NET Core (System.IO.Compression.BrotliEncoder) lacks support for using shared dictionaries during compression and decompression.
This feature is essential for scenarios where creating compressed diff files based on a shared dictionary (derived from another file) is necessary.

A key scenario, for example, is when using Compression Dictionary Transport feature (new browser feature) it is required to create brotli diff files (dcb) that can be sent to the client instead of the full compressed file payload. This helps reducing download time and therefore improve application performance.

Without this, the alternative for engineer to achieve that is to wrap the brotli cli tool or build a native DLL and creating binding to it from .Net code. Both a complex and have multiple downsides.

Possible API extension could be similar to the BrotliEncoder.Compress API definition:
BrotliEncoder.CompressWithDictionary (source, desination, dicitonary, ...)

Looking at the implementation of BrotliEncoder.cs it is unclear to me whether the native library that is used with interop supports shared dictionaries, so if not it needs to be updated to the latest brotli implementation (brotli 1.1.0).

More digging reveals that the native brotli linked code is recent and has the necessary support for shared dictionary. See for example BrotliEncoderPrepareDictionary.

---

Original Comments

Feedback Bot on 2/4/2025, 01:11 AM:

We have directed your feedback to the appropriate engineering team for further evaluation. The team will review the feedback and notify you about the next steps.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

[ericstj]

@carlossanlop - this is similar to what we were talking about with ZStandard.

[carlossanlop]

Needs an API proposal so we can advance this.

[nhelfman]

@carlossanlop

Below is a proposal for extending the Brotli encoder API to support compressing with a shared dictionary buffer. This proposal builds on the existing API (as specified in the .NET documentation) while preserving backward compatibility.

---

Overview

The goal of this proposal is to allow callers to provide a shared dictionary buffer that the BrotliEncoder can use to “prime” its compression state. This is especially useful when compressing multiple related data blocks, where a common dictionary can improve compression ratios by providing additional context.

---

Proposed API Changes

We propose adding a new overload of the existing Compress method that accepts an additional parameter—a read-only span representing the shared dictionary. The new method signature is as follows:

public System.Buffers.OperationStatus Compress(
    ReadOnlySpan<byte> source,
    Span<byte> destination,
    out int bytesConsumed,
    out int bytesWritten,
    bool isFinalBlock,
    ReadOnlySpan<byte> sharedDictionary);

Parameter Details

• source ReadOnlySpan<byte>
  A read-only span of bytes containing the source data to compress.

• destination Span<Byte>
  When this method returns, a byte span where the compressed is stored.

• bytesConsumed Int32
  When this method returns, the total number of bytes that were read from source.

• bytesWritten Int32
  When this method returns, the total number of bytes that were written to destination.

• isFinalBlock Boolean
  true to finalize the internal stream, which prevents adding more input data when this method returns; false to allow the encoder to postpone the production of output until it has processed enough input.

• sharedDictionary ReadOnlySpan<byte>
  A read-only span containing the dictionary data.

  • Usage: The provided dictionary buffer is used to initialize or update the encoder’s internal state before processing the current block.
  • Requirements:
    • The dictionary must be in a format compatible with Brotli’s expected shared dictionary format.
    • It is the caller’s responsibility to ensure that the dictionary remains valid (or is not mutated) during the operation.
    • If an empty span is provided, the encoder should operate as if no dictionary is provided.

Return Value

The method returns an OperationStatus which indicates the result of the compression operation (e.g., Done, DestinationTooSmall, or NeedMoreData or InvalidSharedDictionary).

---

Backward Compatibility

• Existing Method Remains Unchanged:
  The original Compress method (without a dictionary parameter) remains available for clients that do not require shared dictionary support.

• Overload Resolution:
  Clients wishing to use dictionary-based compression can opt-in by calling the new overload. This design avoids breaking changes while offering enhanced functionality.

---

Behavior and Implementation Notes

• Dictionary Integration:
  When a non-empty sharedDictionary is provided, the encoder should use its content to seed the compression state. This means that the dictionary will influence the compression context, potentially allowing for better matching when similar data appears in subsequent input.

• Error Handling:
  The method should validate the dictionary buffer:

  • If the dictionary format is invalid or its size exceeds supported limits, the method should return an appropriate error status or throw an exception.
  • If the dictionary is not required (i.e., an empty span), the method should behave identically to the original API.

---

Example Usage

Below is an example illustrating how a caller might use the new API overload:

// Prepare input data and shared dictionary.
ReadOnlySpan<byte> inputData = GetInputData();
ReadOnlySpan<byte> sharedDict = GetSharedDictionary();

// Allocate a destination buffer.
Span<byte> outputBuffer = new byte[expectedOutputSize];

// Variables to capture operation status.
int bytesConsumed;
int bytesWritten;

// Compress using the shared dictionary.
var status = brotliEncoder.Compress(
    inputData,
    outputBuffer,
    out bytesConsumed,
    out bytesWritten,
    isFinalBlock: false,
    sharedDictionary: sharedDict);

if (status == System.Buffers.OperationStatus.Done)
{
    // Process the compressed output.
}
else
{
    // Handle cases such as DestinationTooSmall or NeedMoreData.
}

---

Conclusion

This proposal adds a new overload to the BrotliEncoder API that accepts a shared dictionary buffer. By introducing a ReadOnlySpan<byte> sharedDictionary parameter, the encoder can now leverage pre-existing context to potentially improve compression performance for related data streams. This design maintains backward compatibility while providing additional functionality for advanced use cases.

[nhelfman]

Another consideration point - I couldn't find any open-source alternative in .Net which supports shared dictionary compression.
There are some older libraries with Brotli compression support but they don't support shared dictionary.

E.g.:
https://github.com/XieJJ99/brotli.net
https://github.com/master131/BrotliSharpLib

A widely used open source implementation of Brotli is the brotli cli (https://github.com/google/brotli/blob/master/c/tools/brotli.md ) which has support for shared dictionary based compression with the -D option.

The cli tool uses the same library that is ultimately used by BrotliEncoder.Compress with interop bindings. But the shared dictionary option is not exposed from the current .Net API as described in the item.