|
| 1 | +# Polymarket Market Explorer |
| 2 | + |
| 3 | +OpenBB Workspace backend for public Polymarket prediction-market data. |
| 4 | + |
| 5 | +A FastAPI backend that OpenBB Workspace registers as a custom data connector. |
| 6 | +Workspace reads `widgets.json` for the widget definitions and `apps.json` for |
| 7 | +the prebuilt **Polymarket Market Explorer** app layout. |
| 8 | + |
| 9 | +Every parameter choice in the app (tag → event → market) is derived from a |
| 10 | +single cached snapshot of **active events**, built by walking Polymarket's |
| 11 | +keyset-paginated Gamma endpoint: |
| 12 | + |
| 13 | +- [`GET /events/keyset`](https://docs.polymarket.com/api-reference/events/list-events-keyset-pagination) — |
| 14 | + active events ranked by 24h volume, each embedding its tags and nested markets. |
| 15 | + Pagination is **cursor-based** (`after_cursor`); the legacy `offset` is rejected. |
| 16 | + |
| 17 | +Because the choice lists come from this snapshot rather than from re-scanning |
| 18 | +live endpoints, filtering by tag and event is fast and consistent with the data |
| 19 | +shown in the tables. Polymarket's `/tags` endpoint carries no activity metric, |
| 20 | +so the **tag list is derived from the events that are actually active**, ranked |
| 21 | +by aggregated volume. The Browse search box routes to |
| 22 | +[`GET /public-search`](https://docs.polymarket.com/api-reference/search/search-markets-events-and-profiles) |
| 23 | +for full-text search across all events. |
| 24 | + |
| 25 | +## Quick Start (Docker) |
| 26 | + |
| 27 | +```bash |
| 28 | +docker compose up --build |
| 29 | +``` |
| 30 | + |
| 31 | +This builds the image and runs the backend on `http://localhost:7779`, with the |
| 32 | +on-disk cache (`POLYMARKET_CACHE_DIR=/data/cache`) persisted to the named volume |
| 33 | +`polymarket-cache`. Because the active-events snapshot lives on that volume, a |
| 34 | +`docker compose restart` reuses it instead of re-scanning the upstream API. The |
| 35 | +container runs as a non-root user, `--init` reaps signals, and the server exits |
| 36 | +cleanly on `docker stop` (bounded graceful shutdown). |
| 37 | + |
| 38 | +Plain Docker with a named volume: |
| 39 | + |
| 40 | +```bash |
| 41 | +docker build -t openbb-polymarket . |
| 42 | +docker run -d -p 7779:7779 --init -v polymarket-cache:/data/cache openbb-polymarket |
| 43 | +``` |
| 44 | + |
| 45 | +Mount a host directory instead of a named volume only if it is writable by |
| 46 | +uid 1000 (the container's `app` user): `-v /host/cache:/data/cache`. Set |
| 47 | +`POLYMARKET_PUBLIC_BASE_URL` if Workspace reaches the backend at a different |
| 48 | +host/port than the container sees. |
| 49 | + |
| 50 | +## Quick Start (local) |
| 51 | + |
| 52 | +```bash |
| 53 | +python -m venv .venv |
| 54 | +source .venv/bin/activate |
| 55 | +pip install -r requirements.txt |
| 56 | +uvicorn main:app --reload --port 7779 --timeout-graceful-shutdown 5 |
| 57 | +``` |
| 58 | + |
| 59 | +`--timeout-graceful-shutdown 5` bounds how long uvicorn waits for open |
| 60 | +connections on Ctrl+C. The app holds a long-lived SSE stream open |
| 61 | +(`/selection_stream`, which syncs the selected market across widgets); without |
| 62 | +the bound, shutdown stalls on *"Waiting for connections to close."* Running |
| 63 | +`python main.py` applies the same timeout without the CLI flag. |
| 64 | + |
| 65 | +Then add the backend URL in OpenBB Workspace → **Settings → Data Connectors → |
| 66 | +Add Custom Backend**: |
| 67 | + |
| 68 | +```text |
| 69 | +http://localhost:7779 |
| 70 | +``` |
| 71 | + |
| 72 | +Workspace loads `widgets.json` and `apps.json` from that URL automatically. |
| 73 | +Open **Apps** and launch **Polymarket Market Explorer**. |
| 74 | + |
| 75 | +## Project Layout |
| 76 | + |
| 77 | +``` |
| 78 | +main.py # thin entry point: exposes `app` for `uvicorn main:app` |
| 79 | +openbb_polymarket/ |
| 80 | +├── app.py # application factory: wiring + CORS + routers + /mcp mount |
| 81 | +├── config.py # Settings loaded from the environment (.env optional) |
| 82 | +├── cache.py # shared on-disk cache (diskcache) rooted at a mounted volume |
| 83 | +├── client.py # disk-cached async httpx wrapper; keyset cursor helper; rate limiter |
| 84 | +├── stats.py # EventStatsCache: active-events scan -> tags + discover/browse |
| 85 | +├── service.py # resolve + realtime: prices-history, book, trades, holders, search |
| 86 | +├── transforms.py # raw Polymarket objects -> flat widget rows |
| 87 | +├── formatting.py # value/format helpers + market_key codec + JSON-string parsing |
| 88 | +├── charts.py # Plotly figure builders |
| 89 | +├── browse.py # HTML event-card browser (iframe) |
| 90 | +├── event_page.py # HTML event details page |
| 91 | +├── marketrules.py # HTML market brief (resolution criteria + UMA) |
| 92 | +├── ladder.py # HTML orderbook ladder |
| 93 | +├── dependencies.py # FastAPI accessors for the shared singletons |
| 94 | +├── mcp_server.py # MCP server (mounted at /mcp) + market-selection pub/sub |
| 95 | +└── routers/ |
| 96 | + ├── meta.py # health, manifests, thumbnail |
| 97 | + ├── options.py # the tag -> event -> market cascade |
| 98 | + ├── discover.py # volume by tag, browse markets, event details |
| 99 | + ├── events.py # event metrics, outcomes, price history |
| 100 | + └── markets.py # rules, orderbook, trades, holders, leaderboard |
| 101 | +``` |
| 102 | + |
| 103 | +An `EventStatsCache` (`openbb_polymarket/stats.py`) pages the active-event book |
| 104 | +once in the background (cursor pagination), maps each event to its tags, and |
| 105 | +persists the snapshot to the on-disk cache. The Discover widgets serve from a |
| 106 | +per-worker mirror instantly; a startup warmer and TTL keep it fresh. |
| 107 | + |
| 108 | +## Caching & Persistence |
| 109 | + |
| 110 | +All caching is backed by [`diskcache`](https://grantjenks.com/docs/diskcache/) |
| 111 | +rooted at `POLYMARKET_CACHE_DIR` — **point this at a mounted volume in |
| 112 | +production**. Both the HTTP responses and the active-events snapshot live on |
| 113 | +disk (SQLite + spill files), so resident memory stays flat regardless of cache |
| 114 | +size, and the expensive initial scan survives restarts and is shared by every |
| 115 | +worker: |
| 116 | + |
| 117 | +- **Initial ingest** is single-flight: a cross-worker lock (`cache.add`) means |
| 118 | + only one worker scans on a cold start; the rest wait for the snapshot to land |
| 119 | + on disk, then load it — no stampede on the upstream API, no duplicate RAM. |
| 120 | +- **Restarts** reuse a still-fresh on-disk snapshot instead of re-scanning. |
| 121 | +- **TTL** is enforced by diskcache `expire` plus the snapshot's own creation |
| 122 | + timestamp; `POLYMARKET_CACHE_SIZE_LIMIT` caps disk use (LRU eviction past it). |
| 123 | + |
| 124 | +See `.env.example` for `POLYMARKET_CACHE_DIR`, `POLYMARKET_CACHE_SIZE_LIMIT`, |
| 125 | +and `POLYMARKET_STATS_SCAN_LOCK_TTL`. |
| 126 | + |
| 127 | +## How the Cascade Works |
| 128 | + |
| 129 | +Each dropdown is populated by an options endpoint that depends only on the |
| 130 | +choice above it, and every data widget falls back to the most active live |
| 131 | +instrument when nothing is selected: |
| 132 | + |
| 133 | +| Choice | Endpoint | Derived from | |
| 134 | +|--------|----------|--------------| |
| 135 | +| Tag | `/options?field=tag` | active-events scan (ranked by volume) | |
| 136 | +| Event | `/options?field=event_id&tag=` | active-events scan (filtered by tag) | |
| 137 | +| Market | `/options?field=market_key&event_id=` | resolved event's markets | |
| 138 | + |
| 139 | +`market_key` is the opaque value passed between market widgets, encoded as |
| 140 | +`event_id|condition_id`. The YES/NO CLOB token ids are re-derived from the |
| 141 | +resolved market. |
| 142 | + |
| 143 | +## Data Sources |
| 144 | + |
| 145 | +| Data | API | Endpoint | |
| 146 | +|------|-----|----------| |
| 147 | +| Events / markets / tags | Gamma | `/events/keyset`, `/events/{id}`, `/markets/keyset` | |
| 148 | +| Search | Gamma | `/public-search` | |
| 149 | +| Price history | CLOB | `/prices-history` | |
| 150 | +| Order book | CLOB | `/book` | |
| 151 | +| Trade tape | Data | `/trades` | |
| 152 | +| Top holders | Data | `/holders` | |
| 153 | +| Leaderboard | Leaderboard | `/volume`, `/pnl` | |
| 154 | + |
| 155 | +## Data Source and Safety |
| 156 | + |
| 157 | +The backend uses only Polymarket public market-data endpoints. It does not |
| 158 | +submit orders, read private portfolio data, or require credentials. Responses |
| 159 | +are cached in memory for short intervals to reduce repeated API calls. |
| 160 | + |
| 161 | +## Validate Locally |
| 162 | + |
| 163 | +With the server running: |
| 164 | + |
| 165 | +```bash |
| 166 | +curl http://localhost:7779/ |
| 167 | +curl http://localhost:7779/widgets.json |
| 168 | +curl "http://localhost:7779/options?field=tag" |
| 169 | +curl "http://localhost:7779/volume_by_tag" |
| 170 | +curl "http://localhost:7779/options?field=event_id&tag=politics" |
| 171 | +``` |
0 commit comments