PaperRadar

AI-powered academic paper discovery platform that ingests papers from ArXiv, enriches them with Semantic Scholar metadata and Claude analysis, scores them across six weighted dimensions, and delivers personalized feeds using vector similarity.

Features

• ArXiv Ingestion Pipeline -- Fetches papers daily from 6 ArXiv categories (cs.AI, cs.CL, cs.LG, cs.CV, cs.RO, stat.ML) via the Atom API, deduplicates across categories, and upserts into Supabase with conflict handling on arxiv_id
• AI Enrichment -- Each paper is enriched with: a 2-sentence TLDR, a keyword-dense TLDR for semantic search, 5 key findings, novelty assessment, practical applicability analysis, difficulty rating, and category classification -- all generated by Claude Haiku
• Semantic Scholar Integration -- Fetches citation counts, influential citations, author h-indexes, and institutional affiliations from the Semantic Scholar API. Links canonical author records with bibliometric data
• GitHub Code Discovery -- Automatically searches GitHub for code repositories associated with each paper, surfacing repos by title/arXiv ID match sorted by star count
• Gravity Scoring Engine -- Scores every paper 0-100 across 6 weighted dimensions: Novelty (25%, Claude-scored), Social Buzz (20%, logarithmic normalization from Reddit/HN/HuggingFace), Builder Relevance (20%, heuristic: code availability, practical language, difficulty), Citation Velocity (15%, citations/day with 30-day cap), Author Reputation (10%, max h-index + top lab affiliation boost), Technical Depth (10%, abstract analysis + keyword detection)
• Vector Embeddings -- Generates 1536-dimension embeddings (OpenAI text-embedding-3-small) from keyword-dense TLDRs. Stored in pgvector with IVFFlat cosine distance indexes for efficient similarity search
• Personalized Feed -- Authenticated users get a feed ranked by gravity_score * cosine_similarity(user_embedding, paper_embedding). Falls back to category-boosted gravity ranking for users without a profile embedding
• Smart Clustering -- Groups semantically similar papers into clusters with labels, average similarity scores, and top gravity scores for trend discovery
• Stripe Payments -- Three-tier pricing (Free/Pro/Team) with Stripe Checkout, customer portal, and webhook-driven tier management. Price-to-tier mapping supports both monthly and yearly billing
• Email Digests -- Configurable daily or weekly email digests via Resend, delivering top-scored personalized papers with React Email templates
• Background Jobs -- Full 5-step pipeline (ingest, enrich, embed, cluster, score) orchestrated by Trigger.dev tasks with retry policies and pipeline run observability logging
• Row-Level Security -- Comprehensive Supabase RLS policies: papers/clusters are publicly readable, profiles/saved papers/interactions are user-scoped, pipeline runs are service-role only

Tech Stack

Category	Technology
Framework	Next.js 16 (App Router)
Language	TypeScript 5
Database	Supabase (PostgreSQL + pgvector)
Auth	Supabase Auth
AI Models	Claude Haiku (enrichment), Claude Sonnet (novelty scoring)
Embeddings	OpenAI text-embedding-3-small (1536 dims)
AI SDK	Vercel AI SDK 6
Payments	Stripe (checkout, webhooks, customer portal)
Email	Resend + React Email
Background Jobs	Trigger.dev v4
Styling	Tailwind CSS v4
RSS Parsing	rss-parser (ArXiv Atom feeds)
Validation	Zod v4
Testing	Vitest + v8 coverage (78 unit tests)

Getting Started

Prerequisites

• Node.js 20+
• Supabase project (with pgvector extension enabled)
• Stripe account
• Anthropic API key
• OpenAI API key (for embeddings)
• Resend API key (for email digests)
• Trigger.dev account (for background jobs)

Installation

git clone https://github.com/yourusername/paperradar.git
cd paperradar
npm install

Environment Variables

Create a .env.local file:

# Supabase
NEXT_PUBLIC_SUPABASE_URL="https://your-project.supabase.co"
NEXT_PUBLIC_SUPABASE_ANON_KEY="your-anon-key"
SUPABASE_SERVICE_ROLE_KEY="your-service-role-key"

# AI
ANTHROPIC_API_KEY="sk-ant-..."
OPENAI_API_KEY="sk-..."

# Stripe
STRIPE_SECRET_KEY="sk_..."
STRIPE_WEBHOOK_SECRET="whsec_..."
STRIPE_PRO_MONTHLY_PRICE_ID="price_..."
STRIPE_PRO_YEARLY_PRICE_ID="price_..."

# Email
RESEND_API_KEY="re_..."

# Trigger.dev
TRIGGER_SECRET_KEY="tr_..."

Database Setup

Apply migrations to your Supabase project:

npx supabase db push

Run

npm run dev

Open http://localhost:3000.

Build

npm run build

Testing

The pure scoring and auth logic is split from its I/O (database, AI, Stripe) into a self-contained "functional core", so it can be unit-tested without mocks or a live database.

npm test                # Vitest unit suite — 78 tests
npm run test:coverage   # + v8 coverage (functional core: ~99% lines, 100% functions)
npm run typecheck       # tsc --noEmit

Tested modules: the Gravity Engine dimension scorers (pipeline/score-dimensions), ArXiv id parsing (pipeline/utils), tier/paywall rules (paywall/check), and the HMAC unsubscribe tokens (email/unsubscribe-token). CI runs typecheck → lint → test → build on every push and pull request.

Project Structure

paperradar/
├── supabase/
│   └── migrations/
│       ├── 001_initial_schema.sql     # Papers, clusters, social signals, authors, profiles,
│       │                              # saved papers, user interactions, pipeline runs + RLS
│       ├── 002_vector_functions.sql   # pgvector similarity search RPCs
│       ├── 003_profile_functions.sql  # Profile embedding update functions
│       └── 004_stripe_indexes.sql     # Payment-related indexes
├── src/
│   ├── app/
│   │   ├── (auth)/login/             # Supabase Auth login page
│   │   ├── (main)/
│   │   │   ├── feed/                 # Personalized paper feed with filters and pagination
│   │   │   ├── paper/[id]/           # Paper detail view (TLDR, key findings, gravity breakdown)
│   │   │   ├── clusters/             # Cluster discovery page
│   │   │   ├── cluster/[id]/         # Cluster detail with grouped papers
│   │   │   ├── billing/              # Stripe billing and subscription management
│   │   │   └── settings/             # User preferences (categories, digest frequency)
│   │   ├── admin/                    # Pipeline management dashboard
│   │   ├── api/
│   │   │   ├── papers/[id]/          # Paper interactions and save/unsave endpoints
│   │   │   ├── pipeline/             # Pipeline trigger and status endpoints
│   │   │   ├── stripe/               # Checkout, portal, and webhook handlers
│   │   │   └── email/                # Unsubscribe endpoint
│   │   └── page.tsx                  # Marketing landing page with pricing
│   ├── components/                   # Paper cards, gravity badges, save buttons, paywall gates
│   ├── lib/
│   │   ├── ai/                       # AI model configuration (Claude + OpenAI)
│   │   ├── pipeline/
│   │   │   ├── arxiv-ingest.ts       # ArXiv Atom API fetcher with rate limiting
│   │   │   ├── enrich.ts             # Semantic Scholar + GitHub + Claude enrichment
│   │   │   ├── embed.ts              # Batch vector embedding generation
│   │   │   ├── cluster.ts            # K-means-style paper clustering
│   │   │   ├── score.ts              # Gravity scoring engine (imperative shell)
│   │   │   └── score-dimensions.ts   # Pure 6-dimension scoring math (unit-tested)
│   │   ├── feed/personalized.ts      # Vector-ranked and category-boosted feed generation
│   │   ├── stripe/                   # Stripe client and tier mapping
│   │   ├── email/                    # Resend client, digest sender, React Email templates
│   │   ├── paywall/                  # Tier-based access control
│   │   └── db/                       # Supabase client factories
│   ├── trigger/
│   │   ├── full-pipeline.ts          # End-to-end 5-step pipeline orchestrator
│   │   ├── ingest-arxiv.ts           # Scheduled ArXiv ingestion task
│   │   ├── enrich-papers.ts          # Batch enrichment task
│   │   ├── embed-papers.ts           # Batch embedding task
│   │   ├── cluster-update.ts         # Cluster refresh task
│   │   ├── score-papers.ts           # Batch scoring task
│   │   └── send-digests.ts           # Email digest delivery task
│   ├── types/database.ts             # TypeScript interfaces mirroring the Supabase schema
│   └── middleware.ts                 # Auth middleware for protected routes
├── trigger.config.ts                 # Trigger.dev configuration
└── package.json

Pipeline Architecture

The data pipeline runs as a sequence of Trigger.dev tasks, each idempotent and independently retryable:

1. Ingest -- Fetch latest papers from ArXiv across 6 categories (rate-limited to 1 req/3s per ArXiv guidelines). Deduplicate across categories. Upsert in batches of 25.
2. Enrich -- For each un-enriched paper: query Semantic Scholar for citations and author data, search GitHub for code repos, generate structured analysis via Claude Haiku. Rate-limited to 1 paper/second.
3. Embed -- Generate 1536-dim embeddings from keyword-dense TLDRs in batches of 20 using OpenAI embedMany.
4. Cluster -- Group semantically similar papers using cosine similarity thresholds. Update cluster metadata.
5. Score -- Calculate the 6-dimension Gravity Score for each enriched paper. Rescore stale papers (>24h) to reflect new social signals.

Notes

• The Gravity Engine's dimension weights (novelty 25%, social buzz 20%, builder relevance 20%, citation velocity 15%, author reputation 10%, technical depth 10%) are tuned for the AI/ML builder community.
• Personalized feeds degrade gracefully: vector-ranked feed for users with profile embeddings, category-boosted gravity ranking for users without, and pure gravity ranking for unauthenticated users.
• The paywall gates access to full gravity breakdowns, daily digests, and advanced filters behind the Pro tier.

License

MIT