Skip to content
BranchesTags

Repository files navigation

ClienTell

Version: 1.0.0 | Status: Live | Use: Internal only | Deployment: clientell.fly.dev

01. What It Does

Sales, partnerships, and research workflows require knowing who a company's customers are — but that information is scattered across case study pages, logo grids, testimonials, press releases, and partner directories. Compiling it manually takes hours. ClienTell automates that process: enter a URL, get back a structured list of client/customer mentions in under a minute.

Primary use cases:

  • Pre-call competitive research — who's already a customer of a vendor or competitor?
  • Partnership prospecting — mapping a potential partner's existing customer ecosystem
  • Market research — identifying the customer profile of players in a given category

02. Features

Feature Description
URL input Accept a seed URL. Auto-prepends https:// and www. to bare domains.
Crawl modes Homepage only — single page. Smart scan — targeted multi-page crawl (see §05). Deep scan — full recursive crawl up to 50 pages.
Client/customer extraction Identify company names from logo alt text, testimonial bylines, case study headers, and inline body text. Normalize, deduplicate, and score confidence. Explicitly excludes partner/integration companies.
HTML section prioritization High-signal HTML sections (matching client, customer, testimonial, logo-grid etc. class/id attributes) are extracted and front-loaded in the 20k character token budget before sending to Claude.
Results table & CSV export Sortable, filterable table: company name, confidence, vertical, source type, context snippet, source URL. One-click CSV export.
Shareable results Every completed scan has a permanent URL at /scan/:id with a "Copy link" button. The page replays the stored results (and live-polls if the scan is still running).
Confidence scoring High — dedicated customer/case study page. Medium — testimonial or logo grid. Low — passing mention.
Industry verticals Claude infers industry verticals per company. Multi-value, filterable.
Warnings Amber banner when: page content was truncated before Claude, Claude hit the output token limit, or Smart scan hit the page cap before visiting all qualifying pages.
Methodology page /methodology — explains how crawling and scoring works, what's returned and what's excluded.
Rate limiting Per-IP limit of 5 crawls/hour (express-rate-limit) plus a server-wide cap of 3 concurrent crawls. There is no login — access is controlled by rate limits, not a password.
Cancel crawl Mid-crawl cancel button interrupts between page visits and closes browser cleanly.
Crash recovery Crawls left at running by a server restart/crash are automatically marked failed on startup, so the UI shows an error instead of polling forever.

03. Stack

Single Fly.io app. Express serves both the API and the built React frontend as static files.

Layer Technology Notes
Frontend React 18 + Vite 6 Built and served as static files by Express
Backend Node.js (ESM, v20) + Express 4 Port 8080
Crawler HTTP-first + Playwright fallback fetch() for server-rendered pages; Playwright Chromium headless only for JS SPAs
Extraction Claude API (claude-sonnet-4-6) Parallel API calls via Promise.allSettled
Database Postgres 16 Crawls and results
Rate limiting express-rate-limit 5 crawls/IP/hour; 3 concurrent server-wide

04. Technical Approach

Crawl Layer

Smart scan (default, 20 pages) — two-phase flat crawl:

  1. Discover — fetches the homepage, extracts same-origin href links, and plain-fetches sitemap.xml for additional candidates.
  2. Score — every candidate URL is scored against SMART_SCAN_PATTERNS. Only URLs scoring ≥ 5 are queued.
  3. Visit — qualifying URLs are visited in score-descending order, in parallel batches of 5, up to the max pages cap. No further link discovery happens — the queue is fixed after phase 1.

SMART_SCAN_PATTERNS — page scoring:

Score Page types Example URL patterns
10 Customers, Clients, Case studies, Work, Portfolio, Success stories /customers, /case-studies, /our-work, /portfolio
8 Testimonials, Reviews, Logo walls, Trusted-by /testimonials, /reviews, /trusted-by
6 Industries, Solutions, Spotlights, About/Clients /industries, /solutions, /about/customers
≤ 4 Partners, Integrations, Marketplace, Press, News /partners, /integrations, /press — never fetched

Deep scan — recursively discovers all links from every page, scored and prioritized, up to 50 pages.

Homepage only — crawls the seed URL only.

Per-page fetch strategy:

  1. Try plain fetch() first — fast path (~200–400ms). Works for the majority of B2B sites with server-rendered HTML.
  2. If the response is an empty JS SPA shell (<200 chars of visible text after stripping tags), fall back to Playwright Chromium.
  3. Playwright is lazy-initialized — no browser is launched at all if all pages are served via HTTP.

Additional crawler behaviours:

  • Robots.txt compliance (checked before any page visit)
  • Rate limiting: 150ms between requests
  • Playwright pages: scroll to bottom (500ms wait) to trigger lazy-loaded carousels and logo grids
  • Cancel token: DELETE /api/crawl/:id interrupts between page visits, closes browser cleanly

Extraction Layer

  • All pages extracted concurrently via Promise.allSettled (not sequential)
  • Claude API for NER — extracts company names with surrounding context (max_tokens: 8096)
  • Image alt text and filename parsing for logo grids (e.g. Logo_HomeDepot.svg → Home Depot)
  • HTML section prioritization — high-signal sections front-loaded before the 20k char truncation
  • Deduplication: highest-confidence mention wins per company name
  • Claude explicitly instructed not to include technology partners, integration vendors, or API marketplace listings — only customers/end-users
  • JSON resilience: partial output recovered if Claude hits the output token limit mid-response

Access Control & Rate Limiting

ClienTell has no login or password gate. Access is bounded by rate limits instead:

  • POST /api/crawl is wrapped in express-rate-limit — 5 crawls per IP per hour (returns 429 past the limit)
  • A server-wide in-memory cap of 3 concurrent crawls (MAX_CONCURRENT) — the 4th concurrent request gets a 429 "Server is busy"
  • app.set("trust proxy", 1) so per-IP limiting works correctly behind Fly.io's TLS-terminating proxy

Shareable Results

  • Every scan persists to Postgres and is addressable at /scan/:id
  • ScanView (React) fetches GET /api/crawl/:id: renders stored results if complete, surfaces an error for failed/cancelled scans, or live-polls if the scan is still running
  • "Copy link" button copies the current /scan/:id URL to the clipboard

Database Schema

crawls   (id, url, mode, max_pages, status, created_at, completed_at, warnings TEXT[], error TEXT)
results  (id, crawl_id, company_name, source_url, page_type, source_type, context, confidence, created_at, verticals TEXT[])

A vestigial "session" table is still created by migrate.js (left over from a removed auth layer) but is no longer used.

05. File Structure

ClienTell/
├── Dockerfile                  # mcr.microsoft.com/playwright:v1.59.1-jammy base
├── fly.toml                    # performance-1x, 2gb, region iad
├── package.json
├── vite.config.js
├── .env                        # DATABASE_URL, ANTHROPIC_API_KEY, PORT, NODE_ENV
├── .env.example                # template for the above
├── scripts/
│   ├── inspect-element.js      # dev helper — inspect a page's HTML
│   ├── test-crawler.js         # standalone crawler test
│   └── test-extractor.js       # standalone extractor test
├── server/
│   ├── index.js                # Express entry point, trust proxy, static serve, stale-crawl cleanup on boot
│   ├── routes/
│   │   └── crawl.js            # POST/GET/DELETE /api/crawl — rate limit + concurrency cap
│   ├── services/
│   │   ├── crawler.js          # HTTP-first fetch, Playwright fallback, smart scan batching
│   │   └── extractor.js        # Claude extraction, parallel API calls, dedup, confidence
│   └── db/
│       ├── migrate.js          # Creates/updates schema (crawls, results)
│       └── queries.js          # saveCrawl, saveResults, getCrawl, markFailed, markCancelled, failStaleRunningCrawls
└── client/src/
    ├── App.jsx
    ├── main.jsx                # Routing — App, /methodology, /scan/:id
    ├── index.css
    ├── components/
    │   ├── CrawlForm.jsx       # URL input, mode toggle, max pages, spider animation
    │   ├── ResultsTable.jsx    # Sortable/filterable table, warnings, CSV export
    │   ├── ScanView.jsx        # /scan/:id shareable results view
    │   ├── Methodology.jsx     # /methodology page
    │   ├── Logo.jsx            # SVG wordmark
    │   └── WebCorner.jsx       # Decorative corner webs
    └── lib/api.js              # startCrawl, pollCrawl, cancelCrawl

06. Running Locally

# Start Postgres (if not running)
brew services start postgresql@16

# Copy the env template and fill in values
cp .env.example .env

# Run DB migration (first time or after schema changes)
npm run db:migrate

# Run both Express (8080) and Vite dev server (5173) together
npm run dev

Open http://localhost:5173. No password is required.

npm run dev runs dev:server (node --watch --env-file=.env server/index.js) and dev:client (vite) concurrently. To run the production server alone: npm start.

Known environment quirks:

  • The --env-file flag in the npm scripts requires Node 20+. If a tool can't load it, export manually: export $(grep -v '^#' .env | xargs)
  • SSL: if you hit a local cert issue, NODE_TLS_REJECT_UNAUTHORIZED=0 in .env works around it
  • Port 8080 lingering: lsof -ti:8080 | xargs kill -9

07. Deployment (Fly.io)

App is live at clientell.fly.dev. Machine: performance-1x, 2GB RAM, region iad. Auto stop/start enabled with min_machines_running = 0.

Required secrets (all set):

Secret Description
DATABASE_URL Injected automatically by fly postgres attach
ANTHROPIC_API_KEY Claude API key

To redeploy after changes:

fly deploy

Migration runs automatically on every deploy via fly.toml release_command.

08. Constraints & Known Limitations

  • Public pages only — no login bypass or authenticated scraping
  • No app-level auth — access is rate-limited (5/IP/hour, 3 concurrent), not gated by a password
  • Robots.txt is respected; disallowed pages are skipped
  • HTTP-first fetch has no cookie jar — pages requiring a cookie consent click before content loads may return less data (Playwright fallback handles some of these)
  • Logo extraction is best-effort; alt text quality varies by site
  • Smart scan relies on URL patterns — sites with non-standard structures may be mis-scored
  • discoverLinks() uses a regex for href="..." — single-quoted hrefs supplemented by sitemap.xml

09. Success Metrics

Metric Target
Extraction precision ≥ 85% of extracted companies are genuine clients of the crawled site
Recall rate ≥ 80% of publicly listed customers surfaced vs. manual audit
Time to results Smart scan (20 pages) completes in under 2 minutes; typical under 45 seconds
Team adoption At least one non-builder team member runs a crawl within two weeks of launch

About

clientell.fly.dev/

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages