Skip to content

Commit 01e22b2

Browse files
jwahdatehaghjwahdatehagh
committed
Add docs
1 parent 6784629 commit 01e22b2

8 files changed

Lines changed: 379 additions & 32 deletions

File tree

‎docs/.vitepress/config.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,20 @@ export default defineConfig({
116116
{ text: 'punksmarket.app', link: '/ui/punksmarket-app' },
117117
],
118118
},
119+
{
120+
text: 'Indexer',
121+
items: [
122+
{
123+
text: 'Overview',
124+
link: '/indexer',
125+
collapsed: false,
126+
items: [
127+
{ text: 'Schema', link: '/indexer/schema' },
128+
{ text: 'API', link: '/indexer/api' },
129+
],
130+
},
131+
],
132+
},
119133
],
120134
},
121135
})

‎docs/index.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,12 @@ All contracts target Ethereum mainnet only.
6060
- [punksmarket.app](/ui/punksmarket-app) covers the web interface for
6161
trading Punks on the broken June 9th 2017 `CryptoPunks` contract through
6262
`PunksMarket`.
63+
64+
## Indexer
65+
66+
- [Indexer](/indexer) covers the Ponder service that tracks both markets,
67+
the wrappers, `PunksMarket`, and `PunksAuction` in one process.
68+
- [Schema](/indexer/schema) covers the unified event log, current-state
69+
tables, the `PunksMarket` and auction tables, and denormalized USD pricing.
70+
- [API](/indexer/api) covers the GraphQL, raw-SQL, and purpose-built REST
71+
routes the service exposes.

‎docs/indexer.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Indexer
2+
3+
The indexer is a [Ponder](https://ponder.sh) service that tracks the entire
4+
Punks stack — both markets, every wrapper, `PunksMarket`, `PunksAuction`,
5+
and the vault/stash custody contracts — in one process. It is the data
6+
source behind the activity feeds, profile pages, and bid/offer matching in
7+
[punks.auction](/ui/punks-auction) and
8+
[punksmarket.app](/ui/punksmarket-app).
9+
10+
It lives at `indexer/`. An example deployment runs at
11+
[`indexer.punksmarket.app`](https://indexer.punksmarket.app). The package
12+
README covers the operational side — local Postgres, fork mode, snapshots,
13+
and Kamal deployment — that these pages do not.
14+
15+
## What it watches
16+
17+
| Contract | Address | Tracked as |
18+
| --------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
19+
| `CryptoPunks` | [`0x6Ba6f2207e343923BA692e5Cae646Fb0F566DB8D`](https://evm.now/address/0x6Ba6f2207e343923BA692e5Cae646Fb0F566DB8D) | The June 9th 2017 contract and its native market |
20+
| `CryptoPunksMarket` | [`0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB`](https://evm.now/address/0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB) | The canonical June 22nd 2017 market |
21+
| `WrappedPunk` | [`0xb7f7F6C52F2e2fdb1963Eab30438024864c313F6`](https://evm.now/address/0xb7f7F6C52F2e2fdb1963Eab30438024864c313F6) | ERC-721 wrapper for canonical Punks |
22+
| `CryptoPunks721` | [`0x000000000000003607fce1aC9E043a86675C5C2F`](https://evm.now/address/0x000000000000003607fce1aC9E043a86675C5C2F) | ERC-721 wrapper for canonical Punks |
23+
| `PunksV1Wrapper` | [`0x282BDD42f4eb70e7A9D9F40c8fEA0825B7f68C5D`](https://evm.now/address/0x282BDD42f4eb70e7A9D9F40c8fEA0825B7f68C5D) | ERC-721 wrapper for June 9th 2017 Punks |
24+
| `PunksMarket` | [`0x64e507FEBF26521b73FbdfA533106B2042533218`](https://evm.now/address/punksmarket.eth) | Criteria-bid market over the June 9th contract |
25+
| `PunksAuction` | [`0x6f99d7E85b4Ba6fFD9ff60A09fc12201027b7873`](https://evm.now/address/0x6f99d7E85b4Ba6fFD9ff60A09fc12201027b7873) | Auction house: lots, auctions, offers |
26+
| `PunksVaultFactory` | [`0xf3381B259B2FE142c0A87bffF463695d935D6F66`](https://evm.now/address/0xf3381B259B2FE142c0A87bffF463695d935D6F66) | Auction vault custody |
27+
| `StashFactory` | [`0x000000000000A6fA31F5fC51c1640aAc76866750`](https://evm.now/address/0x000000000000A6fA31F5fC51c1640aAc76866750) | Stash custody |
28+
29+
## Provenance model
30+
31+
Canonical Punks and June 9th 2017 Punks are tracked as two separate
32+
current-state collections:
33+
34+
- `punks` — the canonical collection plus its two ERC-721 wrappers.
35+
- `v1_punks` — the June 9th 2017 contract plus its ERC-721 wrapper.
36+
37+
While a Punk is wrapped, the user-facing `owner` column reflects the
38+
ERC-721 owner, and the underlying native owner (the wrapper itself) is
39+
preserved in `native_owner`. Unwrapping restores the native owner via an
40+
on-chain read. Both collections follow the same public-owner model.
41+
42+
## Documentation Sections
43+
44+
| Section | Use it for |
45+
| ---------------------------------- | ------------------------------------------------------------------------------------------------ |
46+
| [Schema](/indexer/schema) | The unified event log, current-state tables, `PunksMarket` and auction tables, and USD pricing |
47+
| [API](/indexer/api) | The GraphQL, raw-SQL, and purpose-built REST routes the service exposes |
48+
49+
For setup, local Postgres, fork mode, and deployment, see the
50+
[`indexer/` README](https://github.com/networked-art/cryptopunks/tree/master/indexer).

‎docs/indexer/api.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Indexer API
2+
3+
The service exposes both generic and purpose-built read surfaces over the
4+
indexed [schema](/indexer/schema). The generic ones cover ad-hoc queries;
5+
the REST routes wrap the queries the apps run constantly (recent sales, bid
6+
matching, collection stats) so clients don't re-implement them. For the
7+
high-level role of the service, see the [overview](/indexer).
8+
9+
All routes are read-only. The example deployment is at
10+
[`indexer.punksmarket.app`](https://indexer.punksmarket.app).
11+
12+
## Generic surfaces
13+
14+
| Route | Returns |
15+
| -------------- | --------------------------------------------------------------- |
16+
| `/` (GraphQL) | Ponder's generated GraphQL schema over every table |
17+
| `/sql/*` | Read-only SQL over the public schema (Ponder's SQL-over-HTTP client) |
18+
| `/profiles/*` | ENS profile resolution (name, avatar) backed by the offchain cache |
19+
20+
GraphQL and `/sql/*` reach every table in the [schema](/indexer/schema), so
21+
queries the REST routes below don't cover can be expressed directly.
22+
23+
## Sales
24+
25+
```text
26+
GET /sales?limit=&offset=
27+
```
28+
29+
Recent `sale` events, newest first. Each item carries `usd_value_cents`
30+
already on the row — no JOIN, no follow-up RPC. `limit` defaults to `50`
31+
and caps at `200`. Returns `{ items, limit, offset }`, each item with
32+
`source`, `punk_id`, `buyer`, `seller`, `wei_amount`, `usd_value_cents`,
33+
`tx_hash`, `block_number`, `timestamp`, and `day_unix`.
34+
35+
## Bids
36+
37+
The `PunksMarket` criteria-bid surface. `serialize` returns each bid
38+
hydrated with its decoded `criteria` (trait/color masks plus pixel/color
39+
ranges) and its `includeIds` / `excludeIds`.
40+
41+
```text
42+
GET /bids?active=&bidder=&limit=&offset=&sort=
43+
```
44+
45+
Paginated bid list. `active` filters by the active flag; `bidder` filters
46+
by address; `sort ∈ { bid_wei, timestamp, bid_id } × { -asc, -desc }`
47+
(default `bid_wei-desc`). Returns `{ items, total, limit, offset }`.
48+
49+
```text
50+
GET /bids/matching/punk/:punkId
51+
GET /bids/matching/trait/:traitId
52+
GET /bids/matching/color/:colorId
53+
```
54+
55+
The hosted matching predicate. `/matching/punk` returns the active bids
56+
that would accept that Punk; `/matching/trait` and `/matching/color` return
57+
the active bids that would accept at least one Punk carrying that trait or
58+
color. Each runs the full predicate against the
59+
[predicate side tables](/indexer/schema#punksmarket-criteria-bids) —
60+
include/exclude lists, required/forbidden/anyOf trait and color sets, and
61+
pixel/color-count ranges — ordered by `bid_wei` descending. `trait_id`
62+
must be `0..110`; `color_id` must be `1..221`.
63+
64+
```text
65+
GET /bids/:bidId
66+
```
67+
68+
A single bid, hydrated like the list items. `404` when the id is unknown.
69+
70+
## Stats
71+
72+
```text
73+
GET /stats
74+
GET /stats/:window
75+
GET /stats/history/:interval?limit=&cursor=&from=&to=&order=&source=
76+
```
77+
78+
`/stats` is a compact summary across all windows (no time series).
79+
`/stats/:window` returns one window with its time series. `/stats/history/:interval`
80+
is the full-history bucketed series with cursor pagination — each item is
81+
one UTC-aligned calendar bucket; `cursor` is the previous page's
82+
`nextCursor`, `from` / `to` are unix-second bounds, `order` defaults to
83+
`desc`, and `source` scopes to one event source.
84+
85+
## Punks
86+
87+
```text
88+
GET /punks/market-state?fresh=1
89+
```
90+
91+
A compact snapshot of the canonical CryptoPunks market: the `listed` ids
92+
(with parallel `listed_prices`, null for private `onlySellTo` listings),
93+
`active_bids`, and `wrapped` / `legacy_wrapped` sets. Memoized for ten
94+
seconds; pass `?fresh=1` to bypass the cache (e.g. right after a
95+
wrap/unwrap) so the caller sees the new state immediately.
96+
97+
## Accounts
98+
99+
```text
100+
GET /accounts/stats?addresses=&eoa=&scope=
101+
```
102+
103+
Per-account aggregates for a profile page. `addresses` is the
104+
comma-separated custody set (EOA + vault + stash, capped at 8); `eoa` is
105+
the address used for the per-EOA lookups (last-active, first-seen); `scope=v2`
106+
restricts the sale aggregates to the canonical CryptoPunks market, while the
107+
default sums across every indexed sale source.

‎docs/indexer/schema.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Indexer Schema
2+
3+
The indexer writes a single unified event log plus a set of current-state
4+
tables. Everything is a Ponder *onchain* table — the USD price cache lives
5+
there too, because indexing handlers can only read and write onchain
6+
tables. For the high-level role of the service, see the
7+
[overview](/indexer).
8+
9+
## Events
10+
11+
All user-facing activity flows into one `events` table — one row per
12+
indexed log, with a unified shape so an activity feed can render any event
13+
type without joining.
14+
15+
```text
16+
source ∈ { cryptopunks_v1, cryptopunks_v2, wrapped_punks, cryptopunks_721,
17+
v1_wrapper, punks_market, punks_auction, stash, vault }
18+
19+
type ∈ { assign, transfer, stashed, unstashed, vaulted, unvaulted,
20+
escrowed, listing, listing_cancelled, bid, bid_adjusted,
21+
bid_cancelled, sale, wrap, unwrap, escrow_credit,
22+
escrow_withdrawal, lot_created, lot_cancelled, lot_cleared,
23+
lot_updated, auction_started, auction_settled, offer_placed,
24+
offer_cancelled, offer_adjusted }
25+
```
26+
27+
| Column | Holds |
28+
| ------------------------------------------------------- | ---------------------------------------------------------------------- |
29+
| `id` | Primary key (deterministic per log) |
30+
| `source`, `source_event`, `type` | Origin contract, raw event name, and normalized activity type |
31+
| `punk_id` | The Punk involved, when applicable |
32+
| `actor`, `from`, `to`, `buyer`, `seller`, `bidder`, `settler` | The addresses involved, by role |
33+
| `wei_amount`, `listing_wei`, `bid_wei`, `settlement_wei` | The relevant ETH amounts |
34+
| `usd_value_cents` | Denormalized USD-cent value of `wei_amount` (see [USD pricing](#usd-pricing)) |
35+
| `only_sell_to` | Directed-listing / restricted-lot target, when set |
36+
| `bid_id`, `lot_id`, `auction_id`, `offer_id` | The `PunksMarket` / `PunksAuction` entity ids |
37+
| `offer_kind` | `collection` / `trait` / `selection` for offer rows |
38+
| `tx_hash`, `block_number`, `log_index` | Onchain location |
39+
| `timestamp`, `day_unix` | Event time and its UTC day (the JOIN key against `eth_usd_prices`) |
40+
41+
Indexes on `timestamp`, `source`, `type`, `punk_id`, and the actor columns
42+
back the common feed queries (global, per-source, per-type, per-Punk, and
43+
per-account).
44+
45+
## Current-state collections
46+
47+
Per-Punk current state is split into two tables because both contracts use
48+
ids `0..9999` and carry independent market state:
49+
50+
- `punks` — canonical CryptoPunks (`punk_id` PK, `owner`, `native_owner`,
51+
`native_standard`, `is_wrapped`, `wrapper`, `last_sale_wei`, …).
52+
- `v1_punks` — June 9th 2017 Punks, with the same shape.
53+
54+
In both, `owner` is the public owner (the ERC-721 owner while wrapped, the
55+
native owner otherwise) and `native_owner` preserves the underlying holder.
56+
57+
## Native market state
58+
59+
The two native markets each get a listings table and a per-Punk bid table,
60+
updated in place as state changes:
61+
62+
| Table | One row per | Holds |
63+
| ------------------------------ | ----------- | ---------------------------------------------- |
64+
| `listings` / `v1_listings` | Punk | `seller`, `min_value_wei`, `only_sell_to`, `active` |
65+
| `punk_bids` / `v1_punk_bids` | Punk | `bidder`, `value_wei`, `active` |
66+
67+
The per-Punk bid tables hold a single outstanding bid each, mirroring the
68+
contract, which only allows one.
69+
70+
## PunksMarket criteria bids
71+
72+
`PunksMarket` collection bids are decomposed for SQL matching. The scalar
73+
row lives in `market_bids`; the predicate is exploded into side tables so a
74+
"which bids match this Punk?" query runs as a join rather than re-deriving
75+
criteria in the client:
76+
77+
| Table | Holds |
78+
| -------------------------------------- | ---------------------------------------------------------------------------- |
79+
| `market_bids` | `bid_id` PK, `bidder`, `bid_wei`, `settlement_wei`, `active`, the trait/color masks and pixel/color-count ranges, `has_include_ids`, and the raw `criteria_json` / `include_ids_json` / `exclude_ids_json` payloads |
80+
| `bid_traits` | One row per `(bid_id, trait_id, kind)`, `kind ∈ { required, forbidden, anyOf }` |
81+
| `bid_colors` | One row per `(bid_id, color_id, kind)`, same `kind` set |
82+
| `bid_include_ids` / `bid_exclude_ids` | One row per `(bid_id, punk_id)` allow/deny entry |
83+
84+
The matching query joins these against a static Punk dataset, seeded once
85+
from the SDK offline bundle:
86+
87+
| Table | Holds |
88+
| -------------- | ------------------------------------------------ |
89+
| `punk_traits` | One row per `(punk_id, trait_id)` |
90+
| `punk_colors` | One row per `(punk_id, color_id)` |
91+
| `punk_visuals` | `punk_id` PK, `pixel_count`, `color_count` |
92+
93+
The hosted matching predicate is exposed through the
94+
[`/bids/matching/*` routes](/indexer/api#bids).
95+
96+
## Auction state
97+
98+
`PunksAuction` lots, auctions, and offers each mirror their contract struct
99+
so handlers can recover the seller/offerer at cancel or settle time without
100+
re-reading from chain (the contract has often deleted the entity by then):
101+
102+
| Table | Holds |
103+
| -------------------- | -------------------------------------------------------------------------------------- |
104+
| `auction_lots` | `lot_id` PK, `seller`, `reserve_wei`, `only_sell_to`, `item_count`, `active` |
105+
| `auction_lot_items` | One row per `(lot_id, item_index)`: `standard`, `punk_id`, `weight_bps` |
106+
| `auction_auctions` | `auction_id` PK, `lot_id`, `seller`, `latest_bidder`, `latest_bid_wei`, `end_timestamp`, `settled` |
107+
| `auction_offers` | `offer_id` PK, `offerer`, `amount_wei`, `slot_count`, `kind`, `specific_punk_id`, `active` |
108+
109+
`auction_lots` and `auction_offers` are kept around after an entity is
110+
consumed or cancelled so the corresponding handlers can still resolve the
111+
party.
112+
113+
## Accounts
114+
115+
`accounts` is a per-EOA registry, populated whenever an address appears in
116+
an event. It pins each user's deterministic custody contracts —
117+
`vault` (`PunksVaultFactory.predictVault`) and `stash`
118+
(`StashFactory.stashAddressFor`) — plus the `user_proxy` from the legacy
119+
wrapper, and tracks whether each clone is `*_deployed` on-chain. The vault,
120+
stash, and proxy columns are individually indexed for reverse lookup
121+
(custody address → owner EOA), which powers profile-URL canonicalization.
122+
123+
## USD pricing
124+
125+
Every event row carries `usd_value_cents` — the USD-cent equivalent of
126+
`wei_amount` at the day's ETH/USD price, **denormalized onto the row at
127+
indexing time**. Consumers reading recent sales never need a JOIN or a
128+
follow-up RPC.
129+
130+
The day-keyed cache lives in `eth_usd_prices` (`day_unix` PK,
131+
`eth_usd_cents` as Stripe-style integer cents, `source`, `block_number`),
132+
filled by two layered sources:
133+
134+
1. **CSV historical baseline** — `data/eth_usd_prices.csv` ships daily
135+
closes from the June 9th launch through today, seeded once on startup and
136+
guarded by a row in `backfill_markers` so the seed is idempotent across
137+
restarts.
138+
2. **Chainlink live fill** — when a sale handler finds no cached row for the
139+
event's UTC day, it reads Chainlink's ETH/USD aggregator at the event's
140+
historical block, caches the result under `day_unix`, and stamps the row.
141+
Blocks before Chainlink existed (mid-2021) fall back to the CSV baseline;
142+
if neither covers the day, `usd_value_cents` stays null.

0 commit comments

Comments
 (0)