Skip to content

Commit f8269a4

Browse files
jim-tothjim-toth
committed
Updates README with architecture and operations documentation
1 parent dd2bd27 commit f8269a4

1 file changed

Lines changed: 224 additions & 17 deletions

File tree

‎README.md‎

Lines changed: 224 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,245 @@
1-
Operator Registry Controller
1+
# Facilitator Controller
22

3-
## Description
3+
A [NestJS](https://nestjs.com/) oracle/bridge service for the
4+
[ANYONE Protocol](https://anyone.io). It connects the protocol's reward
5+
distribution logic, which runs as [AO](https://ao.arweave.net/) processes on
6+
Arweave, to the protocol's smart contracts on Ethereum.
47

5-
Dedicated oracle-ish service to provide ANYONE Protocol Relay Operator AO
6-
Process (Smart Contract) with Relay Operator info (i.e. identity linking)
8+
In short: it watches Ethereum for reward requests, asks the AO reward processes
9+
how much each account has earned, and writes the resulting reward/allocation
10+
amounts back to Ethereum.
11+
12+
## How it works
13+
14+
When a user requests their rewards on-chain, the relevant Ethereum contract
15+
emits an event. The controller reacts to that event by:
16+
17+
1. **Listening** for the request event over a WebSocket connection to Ethereum.
18+
2. **Claiming** the account's current rewards from the AO reward processes
19+
(relay rewards and staking rewards) via signed AO messages.
20+
3. **Settling** the rewards back on Ethereum by sending a transaction from an
21+
operator wallet.
22+
23+
Because WebSocket subscriptions can miss events (disconnects, restarts, RPC
24+
hiccups), the controller also runs a **discovery** loop that periodically scans
25+
historical blocks over JSON-RPC, persists every request/fulfillment event to
26+
MongoDB, and re-queues any request that was never fulfilled. This makes event
27+
processing eventually-consistent rather than relying solely on the live socket.
28+
29+
### Two operating modes
30+
31+
The service supports two reward flows, toggled independently with the
32+
`USE_HODLER` and `USE_FACILITY` environment variables:
33+
34+
| Mode | Status | Contract | Trigger event | Settlement |
35+
| --- | --- | --- | --- | --- |
36+
| **Hodler** | **Current / production** | Hodler | `UpdateRewards` | Claims relay + staking rewards from AO, `approve()`s the token transfer from the rewards pool, then calls `reward()` on the Hodler contract (with gas accounting). |
37+
| **Facility** | Legacy | Facility | `RequestingUpdate` | Fetches the relay allocation from AO and calls `updateAllocation()` on the Facility contract. |
38+
39+
The current production deployment runs **Hodler mode only**
40+
(`USE_HODLER=true`, `USE_FACILITY=false`). The Facility path is retained for
41+
backwards compatibility.
42+
43+
### Gas accounting (Hodler mode)
44+
45+
When settling Hodler rewards the controller pays gas for two transactions (the
46+
ERC-20 `approve` and the `reward` call). It measures the actual gas spent
47+
against the user-supplied gas estimate and accumulates the running balance in
48+
the `EventsServiceState` document in MongoDB, logging when it over- or
49+
under-charges.
50+
51+
## Architecture
52+
53+
```
54+
Ethereum (Hodler / Facility contracts)
55+
│ ▲
56+
events │ │ reward() / updateAllocation()
57+
▼ │
58+
┌─────────────────────────────────────────────────┐
59+
│ facilitator-controller │
60+
│ │
61+
│ EvmProviderService ── resilient WS (Infura │
62+
│ (live events) primary / Alchemy │
63+
│ secondary) + JSON-RPC │
64+
│ │
65+
│ EventsService ── reacts to live events │
66+
│ *DiscoveryService ── backfills via JSON-RPC │
67+
│ │
68+
│ BullMQ flows (Redis) ── queues the work │
69+
│ Relay/StakingRewards ── claim rewards from AO │
70+
│ ClusterService ── Consul leader election │
71+
└─────────────────────────────────────────────────┘
72+
│ │
73+
▼ ▼
74+
AO processes (Arweave) MongoDB (event state,
75+
relay-rewards / staking- recovery, gas balance)
76+
rewards distribution
77+
```
78+
79+
Key building blocks:
80+
81+
- **EvmProviderService** — manages a resilient pair of WebSocket providers
82+
(Infura primary, Alchemy secondary) with automatic failover, plus a JSON-RPC
83+
provider used for historical event queries.
84+
- **EventsService** — subscribes to live contract events and orchestrates the
85+
reward settlement transactions.
86+
- **EventsDiscoveryService / RewardsDiscoveryService** — periodically scan
87+
historical blocks (in ≤5000-block ranges, every hour by default), store
88+
discovered events in MongoDB, match request events to their fulfillment
89+
events, and re-queue anything unfulfilled.
90+
- **RelayRewardsService / StakingRewardsService** — talk to the AO reward
91+
processes using [`@permaweb/aoconnect`](https://www.npmjs.com/package/@permaweb/aoconnect),
92+
signing messages with an Ethereum-keyed data-item signer.
93+
- **BullMQ flows** — all work runs through Redis-backed job flows, giving
94+
retries, deduplication (by `address`+`txHash` job IDs), and recovery jobs.
95+
- **ClusterService** — uses [Consul](https://www.consul.io/) KV-based leader
96+
election so that, across multiple running instances, only the elected leader
97+
acts on events. Combined with the per-host "local leader" flag
98+
(`IS_LOCAL_LEADER`), exactly one process is "the one" that performs each
99+
one-time action.
100+
101+
### Persistence & infrastructure
102+
103+
- **MongoDB** — stores discovered events, discovery checkpoints
104+
(`lastSafeCompleteBlock`), and gas-balance accounting state.
105+
- **Redis** — backs the BullMQ job queues. Supports both `standalone` and
106+
`sentinel` modes (production uses Sentinel).
107+
- **Consul** — leader election and service discovery.
108+
- **Vault / Nomad** — secret injection and deployment (see `operations/`).
109+
110+
## Configuration
111+
112+
Configuration is entirely environment-variable driven (via `@nestjs/config`).
113+
The most important variables:
114+
115+
### General
116+
117+
| Variable | Description |
118+
| --- | --- |
119+
| `IS_LIVE` | `true` enables real transactions and Consul clustering. When not `true`, the service runs single-node and skips/marks all on-chain writes as "NOT LIVE". |
120+
| `PORT` | HTTP port for the health endpoint (default `3000`). |
121+
| `VERSION` | Build/version string, logged at startup. |
122+
| `DO_CLEAN` | `true` obliterates BullMQ queues on boot. |
123+
| `DO_DB_NUKE` | `true` clears stored request/update event collections on boot. |
124+
| `USE_HODLER` | `true` enables the Hodler reward flow. |
125+
| `USE_FACILITY` | `true` enables the legacy Facility flow. |
126+
127+
### Ethereum / EVM
128+
129+
| Variable | Description |
130+
| --- | --- |
131+
| `EVM_NETWORK` | Network name passed to ethers. |
132+
| `EVM_JSONRPC` | JSON-RPC URL used for historical event discovery. |
133+
| `EVM_PRIMARY_WSS` | Primary WebSocket URL (Infura). |
134+
| `EVM_SECONDARY_WSS` | Secondary/failover WebSocket URL (Alchemy). |
135+
136+
### Hodler mode
137+
138+
| Variable | Description |
139+
| --- | --- |
140+
| `HODLER_CONTRACT_ADDRESS` | Hodler contract address. |
141+
| `HODLER_CONTRACT_DEPLOYED_BLOCK` | Block to start historical discovery from. |
142+
| `HODLER_OPERATOR_KEY` | Private key of the operator wallet that calls `reward()`. |
143+
| `REWARDS_POOL_KEY` | Private key of the wallet that approves the token transfer. |
144+
| `TOKEN_CONTRACT_ADDRESS` | ERC-20 reward token address. |
145+
146+
### Facility mode (legacy)
147+
148+
| Variable | Description |
149+
| --- | --- |
150+
| `FACILITY_CONTRACT_ADDRESS` | Facility contract address. |
151+
| `FACILITY_CONTRACT_DEPLOYED_BLOCK` | Block to start historical discovery from. |
152+
| `FACILITY_OPERATOR_KEY` | Private key of the operator wallet that calls `updateAllocation()`. |
153+
154+
### AO reward processes
155+
156+
| Variable | Description |
157+
| --- | --- |
158+
| `RELAY_REWARDS_PROCESS_ID` | AO process ID for relay rewards. |
159+
| `RELAY_REWARDS_CONTROLLER_KEY` | Signing key for relay-rewards AO messages. |
160+
| `STAKING_REWARDS_PROCESS_ID` | AO process ID for staking rewards. |
161+
| `STAKING_REWARDS_CONTROLLER_KEY` | Signing key for staking-rewards AO messages. |
162+
| `CU_URL` | AO Compute Unit URL used by aoconnect. |
163+
164+
### MongoDB & Redis
165+
166+
| Variable | Description |
167+
| --- | --- |
168+
| `MONGO_URI` | MongoDB connection string. |
169+
| `REDIS_MODE` | `standalone` (default) or `sentinel`. |
170+
| `REDIS_HOSTNAME` / `REDIS_PORT` | Used in `standalone` mode. |
171+
| `REDIS_MASTER_NAME`, `REDIS_SENTINEL_{1,2,3}_HOST/PORT` | Used in `sentinel` mode. |
172+
173+
### Clustering (Consul)
174+
175+
| Variable | Description |
176+
| --- | --- |
177+
| `CONSUL_HOST` / `CONSUL_PORT` | Consul agent address. |
178+
| `CONSUL_SERVICE_NAME` | Service name used for the leader-election key. |
179+
| `CONSUL_TOKEN_CONTROLLER_CLUSTER` | Consul ACL token. |
180+
| `IS_LOCAL_LEADER` | Marks a process as eligible to act / participate in election. |
181+
| `CPU_COUNT` | Number of worker threads when running multi-process. |
182+
183+
> Note: when `IS_LIVE` is not `true`, or when Consul host/port are unset, the
184+
> service bootstraps in single-node mode and treats itself as the leader.
7185
8186
## Project setup
9187

10188
```bash
11-
$ npm install
189+
npm install
12190
```
13191

14-
## Compile and run the project
192+
## Running
15193

16194
```bash
17195
# development
18-
$ npm run start
196+
npm run start
19197

20198
# watch mode
21-
$ npm run start:dev
199+
npm run start:dev
22200

23-
# production mode
24-
$ npm run start:prod
201+
# production build + run
202+
npm run build
203+
npm run start:prod
25204
```
26205

27-
## Run tests
206+
The service exposes a health check at `GET /health` (and `/`), which returns
207+
`OK`.
208+
209+
### Local development
210+
211+
`docker-compose.yml` provides local Redis and MongoDB:
28212

29213
```bash
30-
# unit tests
31-
$ npm run test
214+
docker compose up redis mongo
215+
```
32216

33-
# e2e tests
34-
$ npm run test:e2e
217+
Running the controller against real reward flows requires Ethereum RPC
218+
credentials, contract addresses, operator keys, and AO process IDs (see
219+
[Configuration](#configuration)). Leave `IS_LIVE` unset (or not `true`) to run
220+
single-node and avoid broadcasting real transactions. In production these values are injected from Vault/Consul (see
221+
`operations/`).
35222

36-
# test coverage
37-
$ npm run test:cov
223+
## Tests
224+
225+
```bash
226+
npm run test # unit tests
227+
npm run test:e2e # e2e tests
228+
npm run test:cov # coverage
38229
```
230+
231+
## Deployment
232+
233+
The service is containerized (`Dockerfile`) and deployed to
234+
[Nomad](https://www.nomadproject.io/). Job specs live in `operations/`:
235+
236+
- `facilitator-controller-live.hcl` / `facilitator-controller-stage.hcl` — the
237+
service jobs (run with `count = 2` for redundancy; Consul elects the leader).
238+
- `facilitator-controller-redis-sentinel-*.hcl` — the Redis Sentinel cluster.
239+
240+
Secrets are sourced from Vault and runtime configuration (contract addresses,
241+
process IDs, Mongo/Redis endpoints) from Consul KV and service discovery.
242+
243+
## License
244+
245+
[AGPL-3.0-only](./LICENSE)

0 commit comments

Comments
 (0)