Examples/docs for modern streaming patterns

[AArnott]

I'm trying to find the right APIs to accomplish the following two things.
Do these APIs already exist somewhere that I haven't been able to find?
If not, can I construct them somehow? I particularly would like to keep buffer copying and allocations to a minimum. In both cases, it seems like APIs exist that get me partway there but not all the way.

Synchronous encode/decode

Establish a pipeline that accepts input via an IBufferWriter<byte> that this library implements, and emits output via an IBufferWriter<byte> that I provide. This pipeline should be able to encode or decode.

I'm hoping for something like this:

class LZ4Writer(IBufferWriter<byte> compressedReceiver) : IBufferWriter<byte>
{
  public void Complete();
}

class LZ4Reader(IBufferWriter<byte> decompressedReceiver) : IBufferWriter<byte>
{
}

I could then pass the LZ4Writer object to anything that writes binary data, and it would automatically compress before being forwarded on to whatever disk, IPC or network needs it.
Similar with LZ4Reader but in reverse.

Async encode/decode

This is pretty much just the first example, but async using PipeReader/PipeWriter. And ideally it would be a trivial hookup:

public static class LZ4Extensions
{
   public static PipeReader LZ4Encode(this PipeReader uncompressedData);
   public static PipeReader LZ4Decode(this PipeReader compressedData);
}

So then I could create a Pipe and chain its reader through LZ4, start writing bunches of bytes to it, and concurrently LZ4 will be compressing or decompressing it on its way somewhere.

[MiloszKrajewski]

I have potential problem with this one

class LZ4Reader(IBufferWriter<byte> decompressedReceiver) : IBufferWriter<byte> { }

Currently compression in modelled as write-only, while decompression is modelled as read-only.
I had this itch from very beginning to implement decompression as read-write (so you write compressed data, and it comes out as decompressed) but I never implemented it.

These components you suggest write compressed data to LZ4Reader and get decompressed data out of it.

I'll take a look.

In the meantime, you can find all potential compression target and decompression sources here:

https://github.com/MiloszKrajewski/K4os.Compression.LZ4/blob/master/src/K4os.Compression.LZ4.Streams/LZ4Frame.encode.cs

https://github.com/MiloszKrajewski/K4os.Compression.LZ4/blob/master/src/K4os.Compression.LZ4.Streams/LZ4Frame.decode.cs

[MiloszKrajewski]

Interestingly tickets #105 and #106 are both about vNext, seems like time has come to finally do it:

For half of this one I will need reader/writer state machine (you would not consider suspending on await a solution, right?).

Those below are doable straight away as they fit push-to-write and pull-to-read paradigms:

class LZ4Writer(IBufferWriter<byte> compressedReceiver) : IBufferWriter<byte>
{
  public void Complete();
}

public static class LZ4Extensions
{
   public static PipeReader LZ4Decode(this PipeReader compressedData);
}

but the other half, well... don't hold your breath as it would require some refactoring.

[AArnott]

I think the direction of the chunky streaming isn't super important to me. I just want APIs for the common streaming objects (Pipelines and IBufferWriter) so I can consume them from my new Nerdbank.MessagePack library.