Skip to content

QUICK_IMPORT.md

Latest commit

 

History

History
122 lines (85 loc) · 2.96 KB

QUICK_IMPORT.md

File metadata and controls

122 lines (85 loc) · 2.96 KB

Quick Import Guide - YC Winter 2026

This is a streamlined guide to get you up and running quickly with importing YC Winter 2026 companies.

Prerequisites Checklist

  • Neon database credentials configured
  • n8n webhook URL configured
  • npm dependencies installed
  • Database connection tested

Quick Start (3 Commands)

# 1. Install dependencies (if not already done)
cd backend && npm install

# 2. Test database connection
npm run db:test

# 3. Import YC Winter 2026 companies
npm run import:yc-w26

What Happens During Import

The script will:

  1. Find 40+ companies from YC Winter 2026 batch
  2. Extract details for each company:
    • Company name
    • Description
    • Website URL
    • Sector/category
    • Founders' LinkedIn profiles
  3. Store in Neon database with duplicate detection
  4. Trigger n8n workflows (if configured) for:
    • AI report generation
    • Company evaluation

Expected Runtime

  • Scraping: ~2-3 minutes for 40 companies
  • Rate limiting: 3 seconds between requests
  • Total time: 5-7 minutes

Environment Variables

Your .env file is already configured with:

# Neon Database
DATABASE_URL=postgresql://neondb_owner:npg_2kZoxdber7Fl@ep-ancient-wildflower-a4c1trj9-pooler.us-east-1.aws.neon.tech/neondb?sslmode=require&channel_binding=require

# n8n Webhooks
N8N_WEBHOOK_BASE_URL=https://bubblyducks.app.n8n.cloud/webhook

Testing Before Full Import

To test the scraper without importing to database:

npm run test:scraper

This will:

  • Scrape the first page of Winter 2026 companies
  • Show details for the first company
  • Verify scraping is working correctly

Monitoring Progress

The import script provides real-time progress:

[1/40] Processing: Veriad
  Fetching details for veriad...
  ✓ Imported successfully

[2/40] Processing: Corelayer
  Fetching details for corelayer...
  ⊘ Skipped (already_exists)

[3/40] Processing: Chasi
  Fetching details for chasi...
  ✓ Imported successfully

After Import

Once complete, you can:

  1. View in database:

    psql 'postgresql://neondb_owner:npg_2kZoxdber7Fl@ep-ancient-wildflower-a4c1trj9-pooler.us-east-1.aws.neon.tech/neondb?sslmode=require&channel_binding=require'

SELECT name, website FROM companies WHERE yc_batch = 'Winter 2026';

  2. Check n8n workflows: Log in to https://bubblyducks.app.n8n.cloud and verify workflows are processing

  3. View in frontend: Start the frontend app and browse companies

Common Issues

"No companies found"

  • YC website structure may have changed
  • Try running npm run test:scraper to diagnose

"Database connection failed"

  • Run npm run db:test to verify connection
  • Check .env file for correct DATABASE_URL

"n8n webhook failed"

  • Verify workflows are activated in n8n dashboard
  • Check N8N_WEBHOOK_BASE_URL in .env
  • This is non-critical - companies will still be imported

Need Help?

See the comprehensive guide: IMPORT_GUIDE.md