|
| 1 | +# Deployment — testnet |
| 2 | + |
| 3 | +Ordered sequence for bringing up the contracts, the indexer, and the worker. The order matters: |
| 4 | +each step produces a value the next one needs, and two of the steps are hard to undo. |
| 5 | + |
| 6 | +For day-2 operations (alerts, key rotation, recovery) see [worker/RUNBOOK.md](worker/RUNBOOK.md). |
| 7 | + |
| 8 | +## Keys |
| 9 | + |
| 10 | +Three distinct keys. Keeping them separate is not hygiene, it is the design: |
| 11 | + |
| 12 | +| Key | Held by | Purpose | Must be funded | |
| 13 | +| -------------------- | ------------------- | ----------------------------------------------------------- | ---------------- | |
| 14 | +| **owner / deployer** | a human | deploys, and calls `approvePacket` to release held packets | yes, both chains | |
| 15 | +| **operator** | the worker process | `submitVerification`, `commitVerification`, `recordVerdict` | yes, both chains | |
| 16 | +| **feed signer** | the indexer process | signs published feeds | no | |
| 17 | + |
| 18 | +`approvePacket` is `onlyOwner` precisely so the worker cannot release the packets it chose to |
| 19 | +withhold. If owner and operator are the same key, that protection is gone — the worker could |
| 20 | +approve its own holds. The deploy script warns when they collapse. |
| 21 | + |
| 22 | +The owner key belongs in the CLI environment as `OWNER_PRIVATE_KEY`, **never** in the worker's |
| 23 | +environment. |
| 24 | + |
| 25 | +## 1. Root environment |
| 26 | + |
| 27 | +```bash |
| 28 | +cp .env.example .env |
| 29 | +``` |
| 30 | + |
| 31 | +Set in `.env`: |
| 32 | + |
| 33 | +- `PRIVATE_KEY` — the **owner/deployer** key (this is what hardhat deploys from) |
| 34 | +- `OPERATOR_ADDRESS` — the worker key's _address_ (not its private key) |
| 35 | +- `RPC_URL_BASE_SEPOLIA`, `RPC_URL_OPTIMISM_SEPOLIA` — override the public defaults if you have |
| 36 | + your own endpoints; public RPCs rate-limit and the indexer polls continuously |
| 37 | + |
| 38 | +Leave `DVN_BASE_SEPOLIA` and `DVN_OPTIMISM_SEPOLIA` empty for now — they are outputs of step 3. |
| 39 | + |
| 40 | +## 2. Preflight |
| 41 | + |
| 42 | +```bash |
| 43 | +npx hardhat dvn:preflight --network base-sepolia |
| 44 | +``` |
| 45 | + |
| 46 | +```bash |
| 47 | +npx hardhat dvn:preflight --network optimism-sepolia |
| 48 | +``` |
| 49 | + |
| 50 | +Checks the signer resolves, the RPC answers, the deployer is funded, the ReceiveUln is known, and |
| 51 | +whether an existing deployment is compatible. Sends no transactions. Fix every `ERROR` and read |
| 52 | +every `WARN` before continuing. |
| 53 | + |
| 54 | +## 3. Deploy the DVN |
| 55 | + |
| 56 | +Both chains. `OPERATOR_ADDRESS` must be set or owner and operator collapse. |
| 57 | + |
| 58 | +```bash |
| 59 | +npx hardhat deploy --network base-sepolia --tags ComplianceDVN |
| 60 | +``` |
| 61 | + |
| 62 | +```bash |
| 63 | +npx hardhat deploy --network optimism-sepolia --tags ComplianceDVN |
| 64 | +``` |
| 65 | + |
| 66 | +Then put the two addresses into `.env` as `DVN_BASE_SEPOLIA` and `DVN_OPTIMISM_SEPOLIA`. |
| 67 | + |
| 68 | +> A previously deployed ComplianceDVN cannot be reused. `submitVerification` gained verdict |
| 69 | +> parameters and the contract is not upgradeable, so the old address exposes a different ABI. |
| 70 | +> `dvn:preflight` detects this and says so. |
| 71 | +
|
| 72 | +## 4. Wire the pathway |
| 73 | + |
| 74 | +This is the step that tells the ULN which DVN each pathway **requires**. |
| 75 | + |
| 76 | +```bash |
| 77 | +npx hardhat lz:oapp:wire --oapp-config layerzero.config.ts |
| 78 | +``` |
| 79 | + |
| 80 | +`layerzero.config.ts` throws if either `DVN_*` is unset rather than defaulting to a placeholder — |
| 81 | +wiring a zero address succeeds silently and then every message on that pathway is permanently |
| 82 | +unverifiable, because the required DVN has no code to verify with. |
| 83 | + |
| 84 | +Confirm both sides: |
| 85 | + |
| 86 | +```bash |
| 87 | +npx hardhat dvn:status --network base-sepolia |
| 88 | +``` |
| 89 | + |
| 90 | +Check `operator` is the worker address and `owner` is yours. |
| 91 | + |
| 92 | +## 5. Indexer |
| 93 | + |
| 94 | +```bash |
| 95 | +cd indexer |
| 96 | +cp .env.example .env |
| 97 | +``` |
| 98 | + |
| 99 | +Set: |
| 100 | + |
| 101 | +- `FEED_SIGNING_KEY` — a fresh key, used only for signing. Note its **address**; the worker needs it. |
| 102 | +- `DVN_BASE_SEPOLIA`, `DVN_OPTIMISM_SEPOLIA` — from step 3 |
| 103 | +- `TRACKED_TOKENS` — the ERC-20s whose transfers build the graph. **Empty means no edges**, so the |
| 104 | + feed will be valid, signed, and empty. |
| 105 | +- `TOKEN_MINIMUMS` — `chain:token:minValue` per token, in the token's smallest unit. **Without an |
| 106 | + entry a token never produces an inbound label**, so leaving this empty turns |
| 107 | + `sanctions_1hop_inbound` off entirely. Outbound labels are unaffected. |
| 108 | +- `POLICY_VERSION` — must equal the worker's `POLICY_VERSION` in `worker/assess/policy.ts` (currently `1`) |
| 109 | + |
| 110 | +```bash |
| 111 | +docker compose up -d |
| 112 | +``` |
| 113 | + |
| 114 | +Migrations run at boot. Verify: |
| 115 | + |
| 116 | +```bash |
| 117 | +curl -s localhost:9091/feed/latest.json | head -c 400 |
| 118 | +``` |
| 119 | + |
| 120 | +A `503 no feed published yet` is expected until the first build (`FEED_REBUILD_MS`, default 10 min). |
| 121 | +Check the logs for the boot warnings — they name any tracked token missing a threshold. |
| 122 | + |
| 123 | +Get the signer address for the next step: |
| 124 | + |
| 125 | +```bash |
| 126 | +docker compose logs indexer | grep -i "indexer feed\|signer" |
| 127 | +``` |
| 128 | + |
| 129 | +## 6. Worker |
| 130 | + |
| 131 | +```bash |
| 132 | +cd worker |
| 133 | +cp .env.example .env |
| 134 | +``` |
| 135 | + |
| 136 | +Set: |
| 137 | + |
| 138 | +- `OPERATOR_PRIVATE_KEY` — the **operator** key (funded on both chains). The name differs from |
| 139 | + the root `.env` deliberately: the worker rejects a bare `PRIVATE_KEY` and explains why, so |
| 140 | + copying the root file here fails loudly instead of granting owner rights. |
| 141 | +- `DVN_BASE_SEPOLIA`, `DVN_OPTIMISM_SEPOLIA` — from step 3 |
| 142 | +- `INDEXER_FEED_URL` — e.g. `http://<indexer-host>:9091/feed/latest.json` |
| 143 | +- `INDEXER_SIGNERS` — the indexer's signer **address**. The worker refuses to boot with a feed URL |
| 144 | + and no allowlist: ingesting unverified labels is worse than having none. |
| 145 | +- `EMIT_VERDICT_EVENTS` — `block` by default, which means each veto costs a `recordVerdict` |
| 146 | + transaction. Set it empty to record only `allow` (which rides along on `submitVerification` for |
| 147 | + free). |
| 148 | +- `TEST_DENYLIST` — an address you hold a key for, if you want to demo a veto |
| 149 | + |
| 150 | +Do **not** set `OWNER_PRIVATE_KEY` here. |
| 151 | + |
| 152 | +```bash |
| 153 | +pnpm start |
| 154 | +``` |
| 155 | + |
| 156 | +```bash |
| 157 | +curl -s localhost:9090/readyz |
| 158 | +``` |
| 159 | + |
| 160 | +The worker refuses to verify anything until it has a fresh risk store, so `readyz` failing at |
| 161 | +first is the fail-closed design working, not a fault. |
| 162 | + |
| 163 | +## 7. Smoke test |
| 164 | + |
| 165 | +Send a clean transfer and watch it settle. `demo:send` is the convenience wrapper: |
| 166 | + |
| 167 | +```bash |
| 168 | +npx hardhat demo:send --network base-sepolia --dst opt --to 0x<recipient> --amount 1 |
| 169 | +``` |
| 170 | + |
| 171 | +Or the full LayerZero task, which takes explicit eids rather than a network: |
| 172 | + |
| 173 | +```bash |
| 174 | +npx hardhat lz:oft:send --src-eid 40245 --dst-eid 40232 --to 0x<recipient> --amount 1 |
| 175 | +``` |
| 176 | + |
| 177 | +The worker log should show `VERIFY submitted` then `COMMIT driven`. Or screen a specific |
| 178 | +transaction without sending: |
| 179 | + |
| 180 | +```bash |
| 181 | +pnpm cli verify baseSepolia 0x<txhash> --dry-run |
| 182 | +``` |
| 183 | + |
| 184 | +Then exercise a veto by putting a held address in `TEST_DENYLIST` and sending from it — expect |
| 185 | +`VETO — withholding verification` and no delivery. |
| 186 | + |
| 187 | +To exercise the manual-review path, set `OWNER_PRIVATE_KEY` in your **shell** (not the worker's |
| 188 | +env) and: |
| 189 | + |
| 190 | +```bash |
| 191 | +pnpm cli pending |
| 192 | +``` |
| 193 | + |
| 194 | +```bash |
| 195 | +pnpm cli approve optimismSepolia 0x<payloadHash> |
| 196 | +``` |
| 197 | + |
| 198 | +## Cross-checks that bite later |
| 199 | + |
| 200 | +- **Operator gas on both chains.** `submitVerification` and `commitVerification` run on the |
| 201 | + _destination_ chain, so a bidirectional pathway needs the operator funded on both. |
| 202 | +- **`POLICY_VERSION` must match** between worker code and indexer env. A mismatched feed is |
| 203 | + rejected whole, not reconciled — scores computed under different weights are not comparable. |
| 204 | +- **Feed TTL vs rebuild interval.** `FEED_TTL_SEC` must exceed `FEED_REBUILD_MS` or a document can |
| 205 | + expire before its replacement exists and screening flaps. Config enforces this. |
| 206 | +- **Staleness vs refresh.** `MAX_DENYLIST_STALENESS_MS >= DENYLIST_REFRESH_MS`, likewise enforced. |
| 207 | +- **`DEGRADED_MODE`.** Default `degrade` keeps verifying on OFAC/OpenSanctions alone when the feed |
| 208 | + is unavailable. `halt` withholds everything instead. Decide deliberately. |
| 209 | +- **Sourcify v1 is in a brownout.** The indexer targets v2; if you point `VERIFIER_URL` at a |
| 210 | + self-hosted instance, make sure it serves `/v2/contract/{chainId}/{address}`. |
| 211 | + |
| 212 | +## Dashboards |
| 213 | + |
| 214 | +For a local stack, the indexer's compose file brings up Prometheus and Grafana with both dashboards |
| 215 | +and both datasources already provisioned: |
| 216 | + |
| 217 | +```bash |
| 218 | +docker compose --profile observability up -d |
| 219 | +``` |
| 220 | + |
| 221 | +Grafana on <http://localhost:3000> (loopback only, anonymous viewer). See |
| 222 | +[indexer/README.md](indexer/README.md) for what the panels mean. |
| 223 | + |
| 224 | +To wire them into existing monitoring instead, import: |
| 225 | + |
| 226 | +- `worker/deploy/grafana-dashboard.json` — Prometheus only |
| 227 | +- `indexer/deploy/grafana-dashboard.json` — needs both Prometheus and the indexer's Postgres; |
| 228 | + the audit trail lives in Postgres and is deliberately not exported as metrics |
| 229 | + |
| 230 | +Both pin their datasource variables to the uids `dvn-prometheus` / `dvn-postgres`; if yours are |
| 231 | +named differently, repoint the variable once at the top of the dashboard. |
| 232 | + |
| 233 | +Alerts: `worker/deploy/k8s/prometheusrule.yaml`. |
0 commit comments