Skip to content

feat: implement visual pipeline debugger#269

Open
Grayking1905 wants to merge 1 commit intoc2siorg:mainfrom
Grayking1905:pipeline
Open

feat: implement visual pipeline debugger#269
Grayking1905 wants to merge 1 commit intoc2siorg:mainfrom
Grayking1905:pipeline

Conversation

@Grayking1905
Copy link
Copy Markdown

Feature Description

This PR introduces the Visual Pipeline Debugger, allowing users to visually inspect their image processing pipelines step-by-step. By capturing intermediate state images after each operator executes, users can scrub forward and backward through their pipeline history. Corresponding Blockly blocks actively glow on the canvas to help trace execution logic visually.

Fix - #268

Full Approach in Detail

The approach relies on modifying the backend to pass back intermediate DebugStepState outputs when a debug=True flag is appended to the Pipeline Request. Instead of returning raw high-definition PNGs for every intermediate step (which would bloat the response size), the backend compresses these snapshots aggressively. Each is downscaled to a max dimension of 512px and encoded to JPEG with a quality limit of 70. This ensures snappy UI feedback.

On the frontend, the feature pivots around the React usePipelineStore (via Zustand). A new DebugScrubber UI component intercepts the standard output preview and enables navigation using the debugCursor and debugStates array. We rely on the block.id natively provided by Blockly to match each state with its authoring node, dispatching CSS animations onto the active SVG path in Layout.tsx using a React useEffect callback without modifying the core blockly library.

State Diagram

stateDiagram-v2
    [*] --> Idle
    Idle --> Executing : Click "Run" (Debug On)
    
    state Executing {
        DecodingImage --> ProcessingStep1
        ProcessingStep1 --> CompressThumb1 : Extract State
        CompressThumb1 --> ProcessingStep2
        ProcessingStep2 --> CompressThumb2 : Extract State
        CompressThumb2 --> EncodingFinalImage
    }
    
    Executing --> DebugModeActive : Return PipelineResponse
    Executing --> ErrorState : Operator Error (Return Partial States)
    
    state DebugModeActive {
        ScrubForward
        ScrubBackward
        HighlightActiveBlock
    }
    
    DebugModeActive --> Idle : Exit Debug Mode (Esc)
    ErrorState --> DebugModeActive : Error Intercepted (Debug Rendered)
Loading

Task List Implemented

  • Backend: Update PipelineStep to accept optional id mapping.
  • Backend: Update PipelineRequest to handle debug flag.
  • Backend: Update PipelineResponse to include debug_states.
  • Backend: Add intermediate image capture, resizing (_thumbnail_for_debug), and JPEG encoding.
  • Backend: Update image.py to support variable quality for JPEG export encoding.
  • Frontend: Define types for DebugStepState.
  • Frontend: Expand Zustand usePipelineStore to manage debug modes, cursor states, and iteration controls.
  • Frontend: Modify usePipeline hook to export block.id from Blockly inputs.
  • Frontend: Develop DebugScrubber.tsx component to parse intermediate results visually.
  • Frontend: Refactor PreviewPane.tsx to conditionally render debug UI or standard UI.
  • Frontend: Implement SVG targeting inside Layout.tsx for .debug-highlighted-block.
  • Frontend: Add core Keyboard shortcuts (Command/Ctrl + D, Arrow Left, Arrow Right, Esc).

Manual Testing

  1. Launch frontend and backend servers.
  2. Ensure you are on the pipeline branch.
  3. Open the UI, drag an "Image -> Read Image" block onto the canvas.
  4. Drag several filters (e.g. Gaussian Blur, Convert to Grayscale, Sobel Edge Detection).
  5. Wire all the blocks together to form a legal pipeline.
  6. Toggle the Debug icon in the Toolbar (the Bug icon should turn Amber).
  7. Execute the Pipeline.
  8. Assert that the DebugScrubber appears on the right pane instead of a single output.
  9. Press Right Arrow and Left Arrow to scrub through the outputs. Check that the highlighted block on the canvas moves in parallel.
  10. Intentionally cause an error (e.g., mismatched type) and verify that all debug states up to the error step are successfully returned and navigable.
  11. Press Escape or the exit button to successfully drop back to standard execution view.

Video Result

Screen.Recording.2026-03-22.164728.mp4

@Grayking1905
Copy link
Copy Markdown
Author

@ivantha
please review it

@ivantha
Copy link
Copy Markdown
Member

ivantha commented Apr 13, 2026

Thanks for the PR! It currently has merge conflicts with main (or is behind). Could you rebase onto the latest main and push the updated branch? Happy to merge once it's auto-rebasable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants