Polymarket X Ingestor

This service polls @polymarket on X, normalizes newly detected posts, stores them, and exposes a small read API. Local development uses Playwright + SQLite. Vercel deployment uses X auth cookies + Postgres + Vercel Cron.

Requirements

• Node.js 22+
• An X account dedicated to monitoring
• Playwright browser binaries installed locally
• Postgres for Vercel deployment

Setup

npm install
npx playwright install chromium
cp .env.example .env

Edit .env as needed. Defaults already match the v1 plan.

BROWSER_CHANNEL=chrome is the default because X login often behaves better in the locally installed Google Chrome than in Playwright's bundled Chromium.

Commands

One-time interactive login:

npm run login

Use X's native username/email + password flow in that browser window. Do not use Sign in with Google, because Google commonly blocks OAuth from automation-controlled browsers.

Run one scrape cycle and print a summary:

npm run poll-once

Backfill posts from January 1, 2026 through the present:

npm run backfill

Use a custom cutoff by passing an ISO date:

npm run backfill -- 2026-01-01T00:00:00.000Z

Backfill scrolls the authenticated profile timeline and captures X UserTweets responses until it reaches the cutoff date or stops making progress. Stored posts are deduped by post_id.

Run the long-lived worker:

npm run worker

Run the worker and the read API together:

npm run serve

Run only the read API against the existing database:

npm run api

API

• GET /health
• GET /posts/latest
• GET /posts?since_detected_at=<ISO timestamp>
• GET /posts?since_created_at=<ISO timestamp>
• GET /meme-analyses?status=success&limit=50
• GET /meme-signals?min_score=70&limit=50
• GET /dex-discoveries?min_score=0&limit=50
• GET /posts/<post_id>/meme-analysis
• GET /posts/<post_id>/dex-discovery

On Vercel, the same endpoints are under /api:

• GET /api/health
• GET /api/posts/latest
• GET /api/posts?since_detected_at=<ISO timestamp>
• GET /api/posts?since_created_at=<ISO timestamp>
• GET /api/meme-analyses?status=success&limit=50
• GET /api/meme-signals?min_score=70&limit=50
• GET /api/dex-discoveries?min_score=0&limit=50
• GET /api/posts/<post_id>/meme-analysis
• GET /api/posts/<post_id>/dex-discovery
• GET /api/cron/poll
• GET /api/cron/dex-discovery

AI memecoin signal analysis

The optional AI layer analyzes saved Polymarket posts and produces search intelligence for possible existing memecoins. It does not launch coins, search token markets, verify contracts, or make trading recommendations.

Enable it with:

AI_ENABLED=true
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://share-ai.ckbdev.com
OPENAI_MODEL=gpt-5.4
OPENAI_REASONING_EFFORT=medium
OPENAI_DISABLE_RESPONSE_STORAGE=true
OPENAI_TIMEOUT_MS=30000
AI_MAX_POSTS_PER_POLL=1
MEME_SIGNAL_THRESHOLD=70

When enabled, each successful poll analyzes up to AI_MAX_POSTS_PER_POLL posts that do not yet have a meme signal analysis. AI failures are stored on the relevant post and do not fail the scraping poll. The recommended cron-safe profile is gpt-5.4 with medium reasoning and one post per poll.

DEX discovery layer

The optional DEX discovery layer runs independently from X polling and Telegram trading. It consumes stored high-score meme signals, searches DexScreener for existing token pairs, refreshes already identified pairs on an interval, stores ranked matches, and exposes them through the API/dashboard. It does not quote or execute trades.

Enable it with:

DEX_DISCOVERY_ENABLED=true
DEX_DISCOVERY_MIN_SIGNAL_SCORE=70
DEX_DISCOVERY_MAX_SIGNALS_PER_RUN=5
DEX_DISCOVERY_MAX_QUERIES_PER_SIGNAL=8
DEX_DISCOVERY_CACHE_TTL_MINUTES=30
DEX_CANDIDATE_REFRESH_TTL_MINUTES=10
DEX_CANDIDATE_REFRESH_LIMIT=100
DEX_RUG_CHECK_TTL_MINUTES=10
DEX_RUG_CHECK_LIMIT=100
DEX_DISCOVERY_MIN_LIQUIDITY_USD=5000
DEX_DISCOVERY_MIN_VOLUME_24H_USD=1000
DEXSCREENER_BASE_URL=https://api.dexscreener.com

Run locally:

npm run dex-discovery

Each run refreshes stale known pairs by exact DexScreener pair address, recalculates no-key rug-risk scores for stale candidates, and re-searches stale narrative signals after DEX_DISCOVERY_CACHE_TTL_MINUTES, so later launches can be picked up. Tokens with growing price, volume, liquidity, or already strong market activity receive priority reasons shown in the dashboard; tokens with liquidity, FDV/liquidity, volume/liquidity, price-collapse, DexScreener transaction imbalance, metadata, or age red flags receive rug-risk scores and details. Solana tokens also get free public-RPC checks for mint authority, freeze authority, and top-holder concentration when SOLANA_RPC_URL is reachable.

On Vercel, /api/cron/dex-discovery is scheduled in vercel.json every 10 minutes. External schedulers can call the same path with the Authorization: Bearer <CRON_SECRET> header.

Vercel Deployment

Vercel cannot run the local long-lived worker, persist ./data/app.db, or reuse the local Chrome profile. The deployable path uses Vercel Cron to call /api/cron/poll once per minute and stores state in Postgres. Vercel's built-in once-per-minute cron requires a Pro or Enterprise plan; on Hobby, use an external scheduler to call /api/cron/poll with the same Authorization header.

Required Vercel env vars:

POSTGRES_URL=postgres://...
X_TARGET_HANDLE=polymarket
X_COOKIE_HEADER=<full Cookie request header from X>
X_USER_TWEETS_URL=<optional captured /UserTweets? URL>
CRON_SECRET=<optional random secret>
LOG_LEVEL=info
AI_ENABLED=true
OPENAI_API_KEY=<OpenAI API key>
OPENAI_BASE_URL=https://share-ai.ckbdev.com
OPENAI_MODEL=gpt-5.4
OPENAI_REASONING_EFFORT=medium
OPENAI_DISABLE_RESPONSE_STORAGE=true
OPENAI_TIMEOUT_MS=30000
AI_MAX_POSTS_PER_POLL=1
MEME_SIGNAL_THRESHOLD=70

To get X_COOKIE_HEADER, log into X in a normal browser with the monitoring account, open DevTools, inspect the network request containing /UserTweets?, and copy the full Cookie request header. As a fallback, you can set X_AUTH_TOKEN and X_CSRF_TOKEN from the auth_token and ct0 cookies, but the full cookie header is more reliable. These cookies expire or can be invalidated by X, so monitoring health must be watched.

X_USER_TWEETS_URL is recommended because X does not always expose the GraphQL operation URL in static HTML. Capture it locally from a logged-in session by watching network requests to https://x.com/polymarket and copying the request URL containing /UserTweets?.

Deploy:

vercel
vercel env add POSTGRES_URL
vercel env add X_COOKIE_HEADER
vercel env add X_USER_TWEETS_URL
vercel env add CRON_SECRET
vercel --prod

Manual poll test after deployment:

curl -H "Authorization: Bearer $CRON_SECRET" https://<project>.vercel.app/api/cron/poll

Telegram Solana Buy Bot

The optional Telegram trading layer is independent from the signal monitor. It only buys tokens explicitly supplied by the authorized Telegram user; it does not auto-buy from AI signals.

V1 supports exact-input Solana buys with a confirmation step:

/buy <token_mint> <amount> <SOL|USDC|USDT>
/confirm <trade_id>
/cancel <trade_id>
/balance
/help

Required env vars:

TRADING_ENABLED=true
TELEGRAM_BOT_TOKEN=<telegram bot token>
TELEGRAM_WEBHOOK_SECRET=<random webhook secret>
TELEGRAM_ALLOWED_USER_IDS=<your numeric Telegram user id>
PUBLIC_BASE_URL=https://<project>.vercel.app
POSTGRES_URL=postgres://...
SOLANA_RPC_URL=https://api.mainnet-beta.solana.com
SOLANA_WALLET_SECRET_KEY=<base58 secret key or JSON number array>
JUPITER_API_KEY=<jupiter api key>

Recommended safety env vars:

DEFAULT_SLIPPAGE_BPS=500
MAX_SLIPPAGE_BPS=1000
MAX_PRICE_IMPACT_PCT=15
MIN_SOL_FEE_BALANCE=0.005
MIN_SOL_RESERVE=0.02
MAX_BUY_SOL=1
MAX_BUY_USDC=500
MAX_BUY_USDT=500
TRADE_INTENT_TTL_SECONDS=60

Set the Telegram webhook after deployment:

npm run telegram:set-webhook

Use a limited-balance hot wallet only. Do not use a primary wallet private key.

Runtime data

The service stores runtime state under ./data/:

• app.db: SQLite database
• browser-profile-chrome/ or browser-profile-chromium/: persisted Playwright session for the selected browser channel
• artifacts/: screenshots and HTML captures on parse/login failures

Notes

• This is a polling scraper, not a push feed. Detection timing is bounded by the poll interval.
• X page structure changes can break parsing. When that happens, inspect artifacts under data/artifacts/.
• npm run login must be completed before headless polling can work reliably.