Pipeline Flow

A visual drag-and-drop pipeline builder with AI-powered pipeline generation for designing and analyzing node-based workflows.

Features • Tech Stack • Quick Start • Project Structure • Documentation • Deployment • Contributing • License

---

Overview

Pipeline Flow is a full-stack application that lets you visually design data processing pipelines using a drag-and-drop interface. Connect different node types to build workflows, then submit your pipeline for structural analysis — including DAG (Directed Acyclic Graph) validation.

Built with React, ReactFlow, shadcn/ui, Tailwind CSS, and a FastAPI backend.

---

Features

• Visual Pipeline Builder — Drag and drop nodes onto an interactive canvas
• AI Pipeline Generator — Describe a workflow in natural language and auto-generate the entire pipeline
• OpenAI Integration — Optional GPT-4o-mini powered generation for complex workflows (falls back to smart keyword matching)
• 9 Node Types — Input, Output, LLM, Text, Filter, Merge, API, Condition, Transform
• Dynamic Text Node — Auto-detects {{variables}} and creates corresponding input handles
• Edge Connections — Connect nodes with animated smoothstep edges
• DAG Validation — Submit pipelines to check for valid directed acyclic graph structure
• In-App Documentation — Built-in docs dialog with overview, node types, usage, API reference, and tips
• Modern UI — Clean interface built with shadcn/ui components and Tailwind CSS
• Real-time Feedback — Toast notifications and modal dialogs for analysis results
• Responsive Layout — Full-height canvas with scrollable dialogs that adapt to screen size

---

Tech Stack

Frontend

Technology	Purpose
React 18	UI framework
Vite	Build tool & dev server
ReactFlow	Node-based graph editor
Tailwind CSS	Utility-first CSS
shadcn/ui	UI component library
Zustand	State management
Lucide React	Icon library
Sonner	Toast notifications

Backend

Technology	Purpose
FastAPI	Python web framework
Pydantic	Data validation
Uvicorn	ASGI server
python-dotenv	Environment variable loading
OpenAI API	Optional AI-powered pipeline generation

---

Quick Start

Prerequisites

• Node.js >= 18.x
• Python >= 3.9
• npm or yarn

1. Clone the repository

git clone https://github.com/Gdhanush-13/Pipeline-Builder.git
cd Pipeline-Builder

2. Start the Backend

cd backend

# Create a virtual environment
python -m venv venv

# Activate it
# Windows:
venv\Scripts\activate
# macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env

# (Optional) Add your OpenAI API key for AI-powered generation
# Edit .env and set: OPENAI_API_KEY=sk-your-key-here

# Start the server
uvicorn main:app --reload --port 8000

The API will be available at http://localhost:8000. Visit http://localhost:8000/docs for the interactive Swagger documentation.

3. Start the Frontend

cd frontend

# Install dependencies
npm install

# Start development server
npm run dev

The app will open at http://localhost:5173.

---

Project Structure

pipeline-flow/
├── README.md                   # This file
├── LICENSE                     # MIT License
├── .gitignore                  # Git ignore rules
│
├── docs/                       # Documentation
│   ├── getting-started.md      # Setup guide
│   ├── architecture.md         # System architecture
│   ├── api-reference.md        # Backend API docs
│   └── deployment.md           # Deployment guide
│
├── backend/                    # FastAPI backend
│   ├── main.py                 # API server, DAG logic & AI generator
│   ├── requirements.txt        # Python dependencies
│   ├── .env.example            # Environment template
│   └── .env                    # Local env (gitignored, add OpenAI key here)
│
└── frontend/                   # React frontend
    ├── index.html              # Entry HTML (Vite)
    ├── package.json            # Node dependencies
    ├── vite.config.js          # Vite configuration
    ├── tailwind.config.js      # Tailwind configuration
    ├── postcss.config.js       # PostCSS configuration
    ├── jsconfig.json           # Path aliases
    ├── components.json         # shadcn/ui config
    │
    └── src/
        ├── main.jsx            # React entry point
        ├── App.jsx             # Root component
        ├── index.css           # Global styles & CSS vars
        │
        ├── lib/
        │   └── utils.js        # Utility functions (cn)
        │
        ├── store/
        │   └── index.js        # Zustand store
        │
        └── components/
            ├── toolbar.jsx         # Node toolbar + Generate & Docs buttons
            ├── pipeline-ui.jsx     # ReactFlow canvas
            ├── submit-button.jsx   # Submit action
            ├── draggable-node.jsx  # Draggable node chip
            ├── result-dialog.jsx   # Results modal
            ├── generate-dialog.jsx # AI pipeline generator dialog
            ├── docs-dialog.jsx     # In-app documentation dialog
            │
            ├── ui/                 # shadcn/ui components
            │   ├── button.jsx
            │   ├── card.jsx
            │   ├── input.jsx
            │   ├── textarea.jsx
            │   ├── badge.jsx
            │   ├── separator.jsx
            │   └── dialog.jsx
            │
            └── nodes/              # Pipeline node types
                ├── base-node.jsx   # Base node component
                ├── input-node.jsx
                ├── output-node.jsx
                ├── llm-node.jsx
                ├── text-node.jsx
                ├── filter-node.jsx
                ├── merge-node.jsx
                ├── api-node.jsx
                ├── condition-node.jsx
                └── transform-node.jsx

---

How It Works

Manual Mode

1. Drag node types from the toolbar onto the canvas
2. Connect nodes by drawing edges between their handles (colored dots)
3. Configure each node using its built-in form fields
4. Submit the pipeline to analyze its structure
5. View results showing node count, edge count, and DAG validation

AI Generator Mode

1. Click the Generate button in the toolbar
2. Describe your workflow in plain English (e.g., "Monitor Gmail for invoices, extract attachments, upload to Drive, use OCR, save to Sheets, notify Slack if amount > $1000")
3. Click Generate or press Ctrl+Enter
4. The pipeline is auto-built on the canvas with proper nodes, edges, and branching

Two generation modes:

Mode	When Used	How It Works
Smart Matching	Always available (default)	Parses numbered steps & keywords to build pipelines
OpenAI GPT	When OPENAI_API_KEY is set	Uses GPT-4o-mini for more accurate, complex pipelines

Node Types

Node	Description	Inputs	Outputs
Input	Entry point for data	—	value
Output	Exit point for results	value	—
LLM	Language model processing	system, prompt	response
Text	Text with {{variable}} support	dynamic	output
Filter	Conditional data filtering	input	passed, failed
Merge	Combine multiple inputs	input1, input2, input3	output
API	External API call	headers, body	response, error
Condition	Conditional branching	value	true, false
Transform	Data transformation	input	output

---

API Reference

GET /

Health check endpoint.

Response:

{
  "status": "ok",
  "service": "Pipeline Flow API",
  "version": "1.0.0"
}

POST /pipelines/parse

Analyze a pipeline's structure.

Request Body:

{
  "nodes": [
    { "id": "customInput-1", "type": "customInput", "data": {} }
  ],
  "edges": [
    { "source": "customInput-1", "target": "llm-1" }
  ]
}

Response:

{
  "num_nodes": 1,
  "num_edges": 1,
  "is_dag": true
}

POST /pipelines/generate

Generate a pipeline from a natural language prompt.

Request Body:

{
  "prompt": "Build a chatbot that takes user input, sends it to an LLM, and outputs the response"
}

Response:

{
  "nodes": [...],
  "edges": [...],
  "num_nodes": 4,
  "num_edges": 3,
  "mode": "openai",
  "warning": null
}

mode is "openai" when GPT was used, or "keyword" for smart matching. warning contains any OpenAI fallback message.

For the full interactive API docs, start the backend and visit http://localhost:8000/docs.

---

Deployment

Frontend (Static Hosting)

cd frontend
npm run build

Deploy the dist/ folder to any static host: Vercel, Netlify, GitHub Pages, Cloudflare Pages, etc.

Backend (Server)

cd backend
uvicorn main:app --host 0.0.0.0 --port 8000

Use Docker, Railway, Render, or any Python hosting. See docs/deployment.md for detailed guides.

Environment Variables

Variable	Default	Description
ALLOWED_ORIGINS	*	Comma-separated CORS origins
HOST	0.0.0.0	Server bind address
PORT	8000	Server port
OPENAI_API_KEY	(empty)	Optional — enables AI-powered pipeline generation via GPT-4o-mini

---

Development

Frontend Commands

npm run dev       # Start dev server
npm run build     # Production build
npm run preview   # Preview production build

Backend Commands

uvicorn main:app --reload          # Dev server with hot reload
uvicorn main:app --host 0.0.0.0   # Production server

---

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Guidelines

• Follow existing code style and conventions
• Use Tailwind CSS classes (avoid inline styles)
• Add new shadcn/ui components in src/components/ui/
• Keep node components in src/components/nodes/
• Test both frontend and backend before submitting

---

License

This project is licensed under the MIT License — see the LICENSE file for details.

---

Built with React, FastAPI, ReactFlow, shadcn/ui, and OpenAI