LLM JSON stream to NDJSON stream

[matteobortolazzo]

Introduction

A simple Minimal API that parses LLM-generated JSON while still streaming from Semantic Kernel, and produces an NDJSON stream for the client.

This is especially useful to populate a user interface incrementally.

Here's the example code!

The README describes the approach and the challenges.

Features and Screenshots

• Ask an LLM to generate a JSON
• Parse the JSON while streaming
• Generate NDJSON events and send them to the client

Technical Details

• C# Minimal API
• Semantic Kernel
• Utf8JsonReader
• State machine to parse the JSON

Challenges and Solutions

• Semantic Kernel returns an IAsyncEnumerable<StreamingTextContent>, which can't be used directly with Utf8JsonReader
• Utf8JsonReader is a ref struct, which makes async code challenging to use
• Utf8JsonReader reads tokens and forwards. So, a state machine is used to understand when to create an event for the client
• System.Json polymorphism and source generators are used to generate JSON events
• NDJSON is used because it makes things easy enough for both server and client

[amynic]

Thanks for sharing this streaming option @matteobortolazzo with the Foundry Developer Community!