Skip to content

WORKFLOW_SETUP.md

Latest commit

 

History

History
349 lines (275 loc) · 9.68 KB

WORKFLOW_SETUP.md

File metadata and controls

349 lines (275 loc) · 9.68 KB
Raw

Building the InstaGist Workflow in n8n - Step by Step

This guide walks you through building the workflow manually in n8n, which gives you better control and understanding than importing JSON.

Prerequisites

  • ✅ n8n running (you have this!)
  • ✅ OpenAI API key
  • ✅ yt-dlp installed on server
  • ✅ FFmpeg installed on server

Step-by-Step Workflow Creation

Step 1: Create Webhook Node

  1. Add new workflow in n8n
  2. Add node → Search "Webhook"
  3. Configure:
    • HTTP Method: POST
    • Path: instagram-gist
    • Response Mode: Using 'Respond to Webhook' Node
  4. Save the node

Your webhook URL will be: http://your-n8n-domain/webhook/instagram-gist

Step 2: Add URL Validation (IF Node)

  1. Add node → Search "IF"
  2. Connect from Webhook node
  3. Configure:
    • Condition 1:
      • Value 1: {{ $json.body.url }}
      • Operation: contains
      • Value 2: instagram.com
  4. Save

This creates two paths: TRUE (valid) and FALSE (invalid)

Step 3: Add Error Response (for invalid URLs)

  1. Add node → Search "Respond to Webhook"
  2. Connect from IF node's FALSE output
  3. Configure:
    • Respond With: JSON
    • JSON Response: 
      {
  "error": "Invalid Instagram URL. Please provide a valid Instagram reel URL."
}
  4. Save

Step 4: Download Video (Execute Command)

  1. Add node → Search "Execute Command"
  2. Connect from IF node's TRUE output
  3. Configure:
    • Command: 
      yt-dlp -f "best[ext=mp4]/best" --no-playlist -o "/tmp/insta-gist-{{ $json.body.url.match(/\/(?:reel|p)\/([A-Za-z0-9_-]+)/)[1] }}.mp4" "{{ $json.body.url }}"
  4. Save

Note: The regex extracts the video ID (e.g., DOgC-ZjiEF3) from the URL and downloads to /tmp/. This works with query parameters like ?igsh=...

Step 5: Set Variables (Set Node)

  1. Add node → Search "Set"
  2. Connect from Execute Command
  3. Configure - Add these fields:
    • videoId:
      • Type: String
      • Value: {{ $json.body.url.match(/\/(?:reel|p)\/([A-Za-z0-9_-]+)/)[1] }}
    • videoPath:
      • Type: String
      • Value: /tmp/insta-gist-{{ $json.body.url.match(/\/(?:reel|p)\/([A-Za-z0-9_-]+)/)[1] }}.mp4
    • audioPath:
      • Type: String
      • Value: /tmp/insta-gist-{{ $json.body.url.match(/\/(?:reel|p)\/([A-Za-z0-9_-]+)/)[1] }}.mp3
    • originalUrl:
      • Type: String
      • Value: {{ $json.body.url }}
  4. Save

Note: The regex expression match(/\/(?:reel|p)\/([A-Za-z0-9_-]+)/) correctly extracts the video ID from Instagram URLs regardless of query parameters or format variations.

Step 6: Extract Audio (Execute Command)

  1. Add node → Search "Execute Command"
  2. Connect from Set node
  3. Configure:
    • Command: 
      ffmpeg -i {{ $json.videoPath }} -vn -acodec libmp3lame -ac 1 -ar 16000 -ab 128k {{ $json.audioPath }} -y
  4. Save

What this does: Extracts audio, converts to mono 16kHz MP3 (optimized for Whisper)

Step 7: Read Audio File (Read Binary File)

  1. Add node → Search "Read Binary File"
  2. Connect from previous Execute Command
  3. Configure:
    • File Path: {{ $json.audioPath }}
    • Property Name: audioFile
  4. Save

Step 8: Transcribe Audio (HTTP Request)

  1. Add node → Search "HTTP Request"
  2. Connect from Read Binary File
  3. Configure:
    • Authentication: Select your OpenAI credential
    • Method: POST
    • URL: https://api.openai.com/v1/audio/transcriptions
    • Send Body: Yes
    • Body Content Type: Form-Data Multipart
    • Add Parameter:
      • Name: file
      • Input Data Field Name: audioFile
    • Add Parameter:
      • Name: model
      • Value: whisper-1
    • Add Parameter (optional):
      • Name: language
      • Value: en
  4. Save

Step 9: Generate Summary (HTTP Request)

  1. Add node → Search "HTTP Request"
  2. Connect from previous HTTP Request
  3. Configure:
    • Authentication: Select your OpenAI credential
    • Method: POST
    • URL: https://api.openai.com/v1/chat/completions
    • Send Body: Yes
    • Body Content Type: JSON
    • JSON/RAW Parameters: 
      {
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant that creates concise summaries of video transcripts. Create a summary with 3-5 bullet points highlighting the key messages, main topics, and actionable insights."
    },
    {
      "role": "user",
      "content": "Summarize the following video transcript:\n\n{{ $json.text }}"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 500
}
  4. Save

Note: Using gpt-4o-mini instead of gpt-4 saves cost (~90% cheaper)

Step 10: Format Response (Set Node)

  1. Add node → Search "Set"
  2. Connect from previous HTTP Request
  3. Configure - Add these fields:
    • summary:
      • Type: String
      • Value: {{ $json.choices[0].message.content }}
    • transcript:
      • Type: String
      • Value: {{ $node["HTTP Request"].json.text }}
    • videoId:
      • Type: String
      • Value: {{ $node["Set"].json.videoId }}
    • originalUrl:
      • Type: String
      • Value: {{ $node["Set"].json.originalUrl }}
  4. Save

Step 11: Cleanup Files (Execute Command)

  1. Add node → Search "Execute Command"
  2. Connect from Set node
  3. Configure:
    • Command: 
      rm -f {{ $node["Set"].json.videoPath }} {{ $node["Set"].json.audioPath }}
  4. Save

Step 12: Success Response (Respond to Webhook)

  1. Add node → Search "Respond to Webhook"
  2. Connect from Execute Command
  3. Configure:
    • Respond With: JSON
    • Response Data Source: Define Below for Each Property
    • Add all the fields from the previous Set node
  4. Save

Testing the Workflow

1. Activate the Workflow

Click the Active toggle in the top right

2. Test with curl

curl -X POST http://your-n8n-domain/webhook/instagram-gist \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.instagram.com/reel/DOgC-ZjiEF3/"}'

3. Test with the script

export WEBHOOK_URL="http://your-n8n-domain/webhook/instagram-gist"
./scripts/test-webhook.sh "https://www.instagram.com/reel/DOgC-ZjiEF3/"

Workflow Diagram

Webhook (POST)
    ↓
IF (validate URL)
    ├─ TRUE →  Download Video (yt-dlp)
    │              ↓
    │          Set Variables
    │              ↓
    │          Extract Audio (ffmpeg)
    │              ↓
    │          Read Audio File
    │              ↓
    │          Transcribe (Whisper API)
    │              ↓
    │          Summarize (GPT-4)
    │              ↓
    │          Format Response
    │              ↓
    │          Cleanup Files
    │              ↓
    │          Respond with Success
    │
    └─ FALSE → Error Response

Common Issues & Solutions

Issue: "yt-dlp: command not found"

Solution: yt-dlp is not installed on your n8n server. For Render.com, you need to use the custom Docker image.

# Build and deploy custom Docker image
./scripts/build-and-push.sh your-dockerhub-username

Issue: "ffmpeg: command not found"

Solution: Same as above - use the custom Docker image with FFmpeg included.

Issue: Whisper API returns 400 error

Possible causes:

  • Audio file is too large (>25MB limit)
  • Audio file format not supported
  • File path is wrong

Solution:

  • Check the audio file was created: Add a debug Execute Command node with ls -lh /tmp/insta-gist-*
  • Verify the Read Binary File node is reading the correct path

Issue: Download fails with "Video unavailable"

Possible causes:

  • Video is from a private account
  • Video was deleted
  • Instagram blocking the request

Solution:

  • Test with a different public Instagram reel
  • Add user-agent to yt-dlp command: 
    yt-dlp --user-agent "Mozilla/5.0" -f "best[ext=mp4]/best" ...

Issue: Workflow times out

Possible causes:

  • Long video (>5 minutes)
  • Slow network
  • API rate limiting

Solution:

  • Set execution timeout in n8n settings (Settings → Execution Timeout)
  • Consider adding a queue for long-running jobs

Optimizations

Use GPT-4o-mini instead of GPT-4

In Step 9, change the model from gpt-4 to gpt-4o-mini:

  • Cost: ~$0.15 per 1M input tokens (vs $5 for GPT-4)
  • Quality: Still excellent for summaries
  • Speed: Faster responses

Add Caching

To avoid reprocessing the same video:

  1. Add a "Check Cache" node before download (e.g., check Redis or database)
  2. If cached, return cached summary
  3. If not cached, process and save to cache

Handle Long Videos

For videos >10 minutes:

  1. Add a check for video duration before processing
  2. Return an error or queue for async processing
  3. Consider chunking audio for Whisper (it has a 25MB limit)

Next Steps

  1. ✅ Build the workflow following these steps
  2. ✅ Test with the Instagram URL from your test data
  3. ✅ Add error handling for each node
  4. ✅ Set up monitoring/alerting for failures
  5. ✅ Consider adding a database for caching results

Advanced: Error Handling

For production, add error handling to each node:

  1. Click on a node
  2. Settings → On Error
  3. Choose "Continue" to prevent workflow from stopping
  4. Add an error notification node (email, Slack, etc.)

Support

If you encounter issues:

  1. Check n8n execution logs (Executions tab)
  2. Test each node individually using the "Test" button
  3. Verify credentials are correctly linked
  4. Check server logs for yt-dlp and ffmpeg errors