(upload): comprehensive upload system overhaul with validation, progress tracking, and chunked audio support

[nielsbosma]

Overview

This PR represents a comprehensive overhaul of the file upload system, introducing robust validation, progress tracking, upload limits, and chunked audio recording support. The changes span 40 files with 2,313 insertions and 649 deletions.

Major Features

📁 File Upload System Enhancements

• File Type & Size Validation: Server-side and client-side validation with toast notifications
• Upload Limits: MaxFiles enforcement across multiple uploads with user-friendly error messages
• Progress Tracking: Enhanced upload progress reporting with real-time status updates
• Upload Cancellation: Proper cancellation support with OperationCanceledException handling
• Multi-file Support: Improved handling of single vs. multiple file uploads with proper state management

🎙️ Audio Recording Improvements

• Chunked Upload Support: New ChunkedMemoryStreamUploadHandler for streaming audio uploads
• Accumulation Logic: Automatically accumulates audio chunks into a single growing file
• Two Upload Modes:
  • Chunked (streaming while recording)
  • Single (complete upload when stopped)
• Progress Tracking: Real-time progress and chunk count visualization

📋 Form System Improvements

• View Context Access: Form input factories now receive IViewContext for accessing hooks like UseUpload()
• ScaffoldColumn Support: Forms and DataTables now respect [ScaffoldColumn(false)] attribute
• FileUpload Field Handling: FileUpload fields are no longer auto-scaffolded (requires manual configuration)

⚡ Event System Improvements

• Event Dispatch Queue: New per-session EventDispatchQueue for non-blocking event processing
• Async Event Support: Event handlers now support asynchronous invocation
• Thread Pool Optimization: Dedicated background thread avoids consuming ThreadPool workers during bursts

Breaking Changes

🔴 FileInput API Changes

Before:

return fileState.ToFileInput()
    .Placeholder("Select a file");

After:

var upload = this.UseUpload(MemoryStreamUploadHandler.Create(fileState))
    .Accept(".pdf,.doc,.docx")
    .MaxFileSize(5 * 1024 * 1024);

return fileState.ToFileInput(upload)
    .Placeholder("Select a file");

• ToFileInput() now requires IState<UploadContext> parameter
• Upload configuration (Accept, MaxFileSize, MaxFiles) moved to UseUpload() for consistency
• Provides better separation of concerns between upload logic and UI

🔴 AudioRecorder API Changes

Before:

new AudioRecorder("Start recording", "Recording...")
    .UploadUrl(upload.Value)

After:

new AudioRecorder(upload.Value, "Start recording", "Recording...")

• UploadContext is now a required first parameter
• Removes redundant .UploadUrl() extension method

🔴 FormBuilder Input Factory Changes

Before:

.Builder(e => e.MyField, state => state.ToTextInput())

After (with backwards compatibility):

// Option 1: Use the convenience overload (no context needed)
.Builder(e => e.MyField, state => state.ToTextInput())

// Option 2: Access view context for hooks like UseUpload()
.Builder(e => e.MyField, (state, context) =>
    state.ToFileInput(context.UseUpload(handler)))

• Input factories can now optionally receive IViewContext as second parameter
• Backwards compatible via overload resolution
• Enables access to hooks like UseUpload(), UseState(), etc. within form builders

🔴 UploadContext Non-Nullable

• IState<UploadContext> is no longer nullable
• UseUpload() always returns a valid context

API Additions

New Classes

• ChunkedMemoryStreamUploadHandler: Upload handler for accumulating audio chunks

  • Create(IState<FileUpload<byte[]>?>): Accumulates chunks into single file
  • CreateArray(IState<ImmutableArray<FileUpload<byte[]>>>): Stores each chunk separately

• EventDispatchQueue: Dedicated event processing queue per session

  • Bounded channel (1024 capacity) with DropOldest strategy
  • Single-reader, multi-writer design
  • Automatic cleanup on disposal

Enhanced Classes

• UploadContext: New properties

  • Accept: File type restrictions (MIME types or extensions)
  • MaxFileSize: Maximum file size in bytes
  • MaxFiles: Maximum number of files for multi-file uploads

• MemoryStreamUploadHandler: Now supports both single and multi-file state

  • Auto-detects state type (single vs. collection)
  • Uses appropriate sink pattern (SingleFileSink or MultipleFileSink)

• FileUpload<T>: Generic variant with typed content

  • ToDetails(): Auto-formats Length and Progress fields

Non-Upload API Changes

🟡 Event System

• EventDispatchQueue: New per-session event queue (Ivy/Core/EventDispatchQueue.cs)

  • Prevents ThreadPool starvation during event bursts
  • Bounded channel with DropOldest strategy for backpressure
  • Used automatically by AppSession for event processing

• Async Event Invocation: Event handlers can now be async

  • Properly awaited instead of fire-and-forget
  • Better error handling and cancellation support

🟡 Form & DataTable Scaffolding

• ScaffoldColumn Attribute: Now respected in auto-scaffolding

  [ScaffoldColumn(false)]
  public Guid InternalId { get; set; }  // Not shown in forms/tables

  • Applied to both FormBuilder<T> and DataTableBuilder<T>
  • Prevents sensitive or internal fields from appearing automatically

• FormBuilder Input Factory Context:

  // New overload with IViewContext
  .Builder(e => e.Field, (state, context) => {
      var upload = context.UseUpload(handler);
      return state.ToFileInput(upload);
  })
  
  // Old overload still works
  .Builder(e => e.Field, state => state.ToTextInput())

🟡 Utilities

• Utils.ToListAsync() Fix: No longer blocks on Task.Result in synchronous fallback path
  • Prevents potential deadlocks
  • Better async/sync boundary handling

Implementation Details

Upload Validation Flow

1. Client-side (FileInputWidget.tsx):

   • Validates file size before upload
   • Enforces maxFiles limit
   • Shows toast notifications for validation errors

2. Server-side (UploadService.cs):

   • Re-validates file type and size
   • Shows toast notifications via IClientProvider
   • Returns OK to prevent frontend error handling

Upload State Management

• Single File: IState<FileUpload<byte[]>?>
• Multiple Files: IState<ImmutableArray<FileUpload<byte[]>>>
• Cancellation: Removes canceled file from state or resets to default

Event Processing

• Events are enqueued to EventDispatchQueue
• Processed on dedicated background thread
• Exceptions logged but don't crash the worker
• Graceful shutdown with 2-second timeout

Documentation Updates

• Uploads.md: Comprehensive guide to upload system with all features
• File.md: Updated examples with new API, fixed XML parsing issue with angle brackets
• AudioRecorder.md: Added chunked vs. single upload examples
• Widgets.md: Updated FileInput examples

Bug Fixes

• Fixed XML parsing error in documentation (angle brackets in Callout blocks)
• Fixed Utils.ToListAsync() synchronous fallback to not block on Task.Result
• Fixed tab event handling in Forms
• Fixed file cancellation logic with proper state cleanup
• Fixed maxFiles enforcement across upload sessions

Testing

• Updated TextInputTests.cs for new FormBuilder signature
• Manual testing of:
  • Single and multiple file uploads
  • Audio recording (chunked and single modes)
  • Form validation with file uploads
  • Upload cancellation
  • MaxFiles enforcement
  • File type and size validation

Migration Guide

For FileInput Users

1. Update ToFileInput() calls to include upload context:

// Add this before your FileInput
var upload = this.UseUpload(MemoryStreamUploadHandler.Create(fileState))
    .Accept(".pdf,.doc,.docx")
    .MaxFileSize(5 * 1024 * 1024);

// Update your FileInput
return fileState.ToFileInput(upload).Placeholder("Select file");

2. Configure validation on the upload context instead of the widget

For AudioRecorder Users

Update constructor calls:

// Before
new AudioRecorder("Start", "Recording...").UploadUrl(upload.Value)

// After
new AudioRecorder(upload.Value, "Start", "Recording...")

For FormBuilder Users

If you need access to UseUpload() or other hooks in form fields:

.Builder(e => e.Document, (state, context) => {
    var upload = context.UseUpload(MemoryStreamUploadHandler.Create(state));
    return state.ToFileInput(upload);
})

Statistics

• Files Changed: 40
• Insertions: 2,313
• Deletions: 649
• New Files: 2 (EventDispatchQueue.cs, ChunkedMemoryStreamUploadHandler.cs)
• Commits: 27

---

Reviewers: Please pay special attention to:

• Breaking changes and migration path
• Event dispatch queue performance characteristics
• Upload validation edge cases
• Documentation accuracy
• FormBuilder context access pattern

[Copilot]

@nielsbosma I've opened a new pull request, #1327, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

@nielsbosma I've opened a new pull request, #1328, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

@nielsbosma I've opened a new pull request, #1329, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

@nielsbosma I've opened a new pull request, #1330, to work on those changes. Once the pull request is ready, I'll request review from you.