Skip to content

NEXT_STEPS.md

Latest commit

 

History

History
278 lines (212 loc) · 6.6 KB

NEXT_STEPS.md

File metadata and controls

278 lines (212 loc) · 6.6 KB

Next Steps - Ready to Find Links for All Companies

Current Status

✅ Completed:

  • All 45 Winter 2026 companies scraped and stored
  • N8N workflows for evaluate-company and generate-report (working)
  • N8N workflow for find-company-links (created, needs activation)
  • Trigger script ready (trigger-link-finding.js)

❌ Blocking Issue:

  • The find-company-links workflow returns 404 Not Found
  • This means the workflow needs to be imported and activated in n8n Cloud

Error Encountered

[1/45] 21st
  Triggering workflow...
  ✗ Failed: Request failed with status code 404

Cause: The webhook https://bubblyducks.app.n8n.cloud/webhook/find-company-links doesn't exist yet because the workflow hasn't been imported to n8n.

What You Need to Do (5 minutes)

Step 1: Get Tavily API Key

If you don't have one yet:

  1. Go to https://tavily.com
  2. Sign up (free tier: 1,000 searches/month)
  3. Get your API key (format: tvly-...)

Step 2: Import Workflow to N8N

  1. Log into n8n Cloud: https://bubblyducks.app.n8n.cloud

  2. Import the workflow:

    • Click "Workflows" in sidebar
    • Click "+ New Workflow" or "Import from file"
    • Select: backend/n8n/workflows/find-company-links.json
    • Workflow will load with 6 nodes

  3. Add Tavily credentials:

    • Go to Settings → Credentials
    • Click "+ New Credential"
    • Search for and select "Tavily API"
    • Enter your API key: tvly-YOUR_KEY_HERE
    • Click Save

  4. Configure the Tavily Search node:

    • In the workflow, click the "Tavily Search" node
    • In Credentials dropdown, select your Tavily API credential
    • Verify Query parameter shows: ={{ $json.query }}

  5. Activate the workflow:

    • Click the toggle switch (top-right) to turn it ON
    • Verify webhook URL: https://bubblyducks.app.n8n.cloud/webhook/find-company-links
    • Click Save

Step 3: Test the Workflow

Test with a single company first:

cd backend

# Test with curl
curl -X POST https://bubblyducks.app.n8n.cloud/webhook/find-company-links \
  -H "Content-Type: application/json" \
  -d '{
    "company": {
      "id": 1,
      "name": "Bubble Lab",
      "website": "https://bubblelab.ai/",
      "description": "Open-source workflow automation",
      "sector": "Artificial Intelligence",
      "yc_batch": "Winter 2026"
    },
    "searchConfig": {
      "maxLinks": 5
    }
  }'

Expected Response:

{
  "success": true,
  "company": {
    "id": 1,
    "name": "Bubble Lab"
  },
  "foundLinks": [
    {
      "type": "news",
      "url": "https://www.ycombinator.com/companies/bubble-lab",
      "title": "Bubble Lab | Y Combinator",
      ...
    }
  ],
  "totalSearches": 3,
  "finalLinkCount": 5
}

If you get this response, the workflow is working!

Step 4: Run for All Companies

Once the test succeeds:

cd backend
npm run find:links:auto

What happens:

  • Processes all 45 companies (those needing links)
  • Each company: 3 Tavily searches
  • Automatically stores results in database
  • Takes ~2-3 hours total (with rate limiting)

Expected Output:

========================================
Trigger Link Finding Workflow
========================================

Found 45 Winter 2026 companies
Companies needing links: 45

[1/45] 21st
  Triggering workflow...
  ✓ Found 5 links
  Storing links in database...
  ✓ Added 5 new links

[2/45] Arzule
  Triggering workflow...
  ✓ Found 5 links
  Storing links in database...
  ✓ Added 5 new links

...

========================================
Summary
========================================

Total companies processed: 45
Successful: 45
Failed: 0
Total links added to database: 225

✓ Link finding complete!

Workflow Details

How It Works

Webhook (receives company data)
  ↓
Extract Data (parse payload)
  ↓
Generate Search Queries (creates 3 queries)
  ↓ ↓ ↓ (3 parallel Tavily searches)
Tavily Search (n8n Tavily node)
  ↓
Parse Results (extracts links from Tavily response)
  ↓
Aggregate Results (combines, deduplicates, ranks by score)
  ↓
Respond to Webhook (returns top 5 links)

Search Queries Per Company

  1. [Company Name] YC Winter 2026
  2. [Company Name] startup news 2026
  3. [Company Name] funding announcement

What Gets Found

  • Official YC company page (always included)
  • News from tech publications (TechCrunch, Bloomberg, etc.)
  • PitchBook/Crunchbase profiles
  • Funding announcements
  • Product launches
  • Any other relevant articles

Top 5 links ranked by Tavily's relevance score.

Troubleshooting

Still Getting 404?

Check:

  1. Workflow is imported in n8n
  2. Workflow is activated (toggle ON)
  3. Webhook URL matches exactly: /webhook/find-company-links
  4. Save the workflow after activating

Workflow Fails with Authentication Error?

Check:

  1. Tavily API credentials are configured
  2. API key is correct (starts with tvly-)
  3. Credential is selected in the Tavily Search node

Timeout Errors?

Solution: Increase timeout in trigger-link-finding.js:

timeout: 120000  // Change from 60000 to 120000 (2 minutes)

Alternative: Manual Link Finding

If you prefer not to use the automated workflow, you can still find links manually:

# List companies needing links
npm run find:links

# Then use the add-links-helper to add links manually
# See LINK_FINDING_GUIDE.md for examples

Files Reference

File Purpose
backend/n8n/workflows/find-company-links.json N8N workflow file (import this)
backend/n8n/workflows/WORKFLOW_UPDATE_INSTRUCTIONS.md Detailed setup guide
backend/scripts/trigger-link-finding.js Automated trigger script
backend/scripts/add-links-helper.js Manual link adding helper
LINK_FINDING_GUIDE.md Manual link finding guide

Summary

Current Blocker: Workflow not imported/activated in n8n

Time to Fix: ~5 minutes

Steps:

  1. Get Tavily API key (free tier)
  2. Import workflow to n8n
  3. Add Tavily credentials
  4. Activate workflow
  5. Test with curl
  6. Run npm run find:links:auto

Result: All 45 companies will have 3-5 relevant links automatically found and stored!

Once Links Are Complete

After all companies have links, you can:

  1. Generate reports: npm run trigger:workflows

    • Triggers both evaluate-company and generate-report workflows
    • Reports will now include the found links as context

  2. Check progress:

    npm run db:status

  3. View results in database:

    • Companies with links
    • Evaluations completed
    • Reports generated

The project will be 100% complete! 🎉