Skip to content

operations.md

Latest commit

 

History

History
342 lines (261 loc) · 14.4 KB

operations.md

File metadata and controls

342 lines (261 loc) · 14.4 KB
Raw

Operations guide

English · 中文

Every operation: starting services, accounts & webhooks, request signing, addresses/balances, chain data (scan/rescan/reorg), withdrawals, concentration, internal transfers, manual corrections, coin management, and direct DB queries. Examples use http://localhost:8080 and chain SEPOLIA; Solana specifics are in §3a.

The easiest way to do all of this is the ops console at http://localhost:8080/admin (no HMAC needed). The /internal/v1/* endpoints it calls must be bound to localhost / an internal network in production. The CLI/API commands below are for integration and automation.

0. Start services

make migrate        # one-time: create schema + seed coins (ETH/USDC on Sepolia, SOL/USDC on Solana devnet)
make run-api        # serves :8080
make run-indexer    # continuous scanner (optional but recommended)
curl -s http://localhost:8080/healthz     # {"status":"ok"}

1. Create an integrator account (API credentials)

Accounts are created via the internal API (no signature; localhost-only in production).

curl -s -X POST http://localhost:8080/internal/v1/accounts \
  -H 'content-type: application/json' \
  --data '{"name":"my-exchange"}'
# {"account_id":1,"api_key":"ak_...","api_secret":"<64-hex>"}   # secret shown ONCE

Optional body fields: "mode" (live default / test) and "callback_url" (webhook endpoint, see §1.1). Save api_key and api_secret:

export APIKEY=ak_...  SECRET=<64-hex>

Account modes & secret recovery. The api_secret is stored KEK-encrypted (the server needs it to verify HMACs), but recovery is gated by mode: live (default) — never recoverable; test — recoverable by operators (go run ./cmd/reveal-secret [--api-key ak_...]). Use live for real accounts.

1.1 Webhook callbacks

With a callback_url set, the system POSTs an HMAC-signed JSON on each event (no polling needed):

curl -s -X POST http://localhost:8080/internal/v1/admin/accounts/callback \
  -H 'content-type: application/json' \
  --data '{"account_id":1,"callback_url":"https://exchange/callback"}'
  • Events: deposit.confirmed, withdrawal.sent/confirmed/failed/internal.
  • Body: {type, ref_type, ref_id, payload{…}, timestamp}; headers X-Signature = hex(HMAC_SHA256(api_secret, body)), X-Timestamp. Verify with your secret.
  • Retried up to 3× (linear backoff); each attempt recorded in webhook_deliveries.
  • GET /internal/v1/admin/webhooks?account_id= lists deliveries (also in the console "Accounts" tab).

2. Request signing (HMAC)

Every /api/v1/* request needs:

X-Api-Key:    <api_key>
X-Timestamp:  <unix seconds, within ±5 min of server time>
X-Signature:  hex( HMAC_SHA256( secret, MESSAGE ) )
MESSAGE     = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + SHA256_hex(BODY)

PATH excludes the query string; BODY is the raw JSON bytes (empty string for GET).

2.1 Pure curl + openssl signer

sign_and_call() {                              # needs APIKEY / SECRET exported
  local method="$1" fullpath="$2" body="$3"
  local path="${fullpath%%\?*}"
  local ts; ts=$(date +%s)
  local bodyhash; bodyhash=$(printf '%s' "$body" | openssl dgst -sha256 -hex | awk '{print $NF}')
  local msg; msg=$(printf '%s\n%s\n%s\n%s' "$method" "$path" "$ts" "$bodyhash")
  local sig; sig=$(printf '%s' "$msg" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $NF}')
  if [ "$method" = "GET" ]; then
    curl -s -X GET "http://localhost:8080${fullpath}" \
      -H "X-Api-Key: $APIKEY" -H "X-Timestamp: $ts" -H "X-Signature: $sig"
  else
    curl -s -X "$method" "http://localhost:8080${fullpath}" -H 'content-type: application/json' \
      -H "X-Api-Key: $APIKEY" -H "X-Timestamp: $ts" -H "X-Signature: $sig" --data "$body"
  fi
}

2.2 Built-in cmd/smoke

The smoke tool implements the same scheme. Argument order is <secret> <apikey>:

go run ./cmd/smoke address  $SECRET $APIKEY SEPOLIA
go run ./cmd/smoke balances $SECRET $APIKEY <address>

3. Addresses & balances

# create a custodial address (type: deposit default | hot | fee)
sign_and_call POST /api/v1/addresses '{"chain":"SEPOLIA"}'
# {"address":"0x…","chain":"SEPOLIA","id":3,"type":"deposit"}

sign_and_call GET /api/v1/addresses ''
sign_and_call GET "/api/v1/balances?address=0x…" ''
sign_and_call GET "/api/v1/deposits?coin=ETH" ''       # &address=…&status=detected|confirmed
  • Address types: deposit (per-user receiving), hot (operational wallet / sweep destination), fee (gas wallet that funds ERC20 sweeps; falls back to hot if absent). Same key custody; the type is just a usage tag.
  • Each address generates a fresh key, AES-GCM-encrypted into wallet_keys: secp256k1 (EVM, 0x… hex address) or ed25519 (Solana, base58 address), chosen by the chain's kind.
  • Balances: amount (confirmed available, human + base), locked (reserved by in-flight withdrawals).

3a. Solana (devnet)

The SOLANA_DEVNET chain (kind solana) supports native SOL and SPL tokens. Everything above works the same through the console/API; the differences:

  • Enable it: set SOLANA_RPC in .env (e.g. https://api.devnet.solana.com) and keep the SOLANA_DEVNET entry in config.yaml (see config.example.yaml). make migrate seeds SOL (9 decimals) and devnet USDC (SPL mint).
  • Addresses are base58 ed25519 public keys (not 0x…). Fund them from a devnet faucet: solana airdrop 2 <addr> --url devnet or https://faucet.solana.com; get devnet USDC from https://faucet.circle.com (select Solana devnet).
  • Fees: the sending / fee-paying address needs SOL. Withdraw from a funded hot wallet; the fee (or hot) wallet also funds SPL-sweep fees + ATA rent.
  • SPL transfers auto-create the recipient's associated token account (ATA) when missing (rent paid by the fee payer).
  • Confirmations: scanned at finalized (commitment), so a detected deposit is credited on first sight (confirmations: 1); expect ~15–30s latency.
  • No RBF: bump is rejected for Solana (no nonce). A dropped tx (expired blockhash) is simply resent as a new tx.
  • Explorer: the console links to Solscan with ?cluster=devnet (configurable via the chain's explorer in config.yaml).

4. Chain data updates (scanning)

The scanner turns chain events into ledger data. Inbound transfers to our addresses become deposits; confirmations accrue; once over the threshold (Sepolia default 3) the balance is credited. These endpoints are internal.

curl -s "http://localhost:8080/internal/v1/head?chain=SEPOLIA"          # head block
curl -s -X POST .../internal/v1/rescan/block -d '{"chain":"SEPOLIA","number":11149212}'
curl -s -X POST .../internal/v1/rescan/tx    -d '{"chain":"SEPOLIA","tx_hash":"0x…"}'

rescan/tx both records/updates a deposit and, if the tx backs a withdrawal, updates its confirmation. Idempotent: rescanning never double-credits.

4.1 Reconcile (correct already-booked data)

Normal rescans never touch finalized rows. After fixing a parsing/accounting bug, correct old entries with "reconcile": true — an idempotent delta correction against on-chain truth:

curl -s -X POST .../internal/v1/rescan/tx -d '{"chain":"SEPOLIA","tx_hash":"0x…","reconcile":true}'

Deposits: update the recorded amount and adjust the balance (or reverse on reorg). Withdrawals: back-charge a missed fee without re-debiting the principal.

4.2 Reorg handling

The continuous scanner detects reorgs by block-hash continuity (scan_cursors.last_block_hash): a mismatched parent rolls the cursor back reorg_depth blocks (default 6) and re-scans the canonical chain. Each tick re-verifies recently-confirmed deposits — a dropped tx reverses the credit (deposit_reorg ledger entry, deposit → failed); a moved tx updates its block.

4.3 Continuous scanner & external (other-language) scanners

make run-indexer scans forward from scan_cursors.last_scanned_block. A scanner written in any language can feed the same ledger path:

curl -s -X POST .../internal/v1/ingest/deposit -d '{
  "chain":"SEPOLIA","coin":"ETH","block_number":… ,"tx_hash":"0x…","log_index":0,
  "from":"0x…","to":"0xOurAddress","amount_base":"…","confirmations":12 }'
curl -s -X POST .../internal/v1/ingest/confirmation -d '{"chain":"SEPOLIA","tx_hash":"0x…","log_index":0,"confirmations":20}'

Internal transfers are not deposits. If a scanned transfer's from is one of our addresses (gas funding, a sweep, an internal rebalance) it is treated as internal and not credited as a customer deposit — concentration/withdraw book those moves themselves.

5. Withdrawals

sign_and_call POST /api/v1/withdrawals '{
  "client_withdraw_id":"wd-1","chain":"SEPOLIA","coin":"ETH","to":"0xDest","amount":"0.001"}'
sign_and_call GET /api/v1/withdrawals/1 ''

The response includes settlement details parsed by the scanner: fee_paid, gas_used, block_number, confirmations, tx_status.

  • Validation & limits: the to address is format-checked (invalid → rejected); per-coin max_withdraw (single) and max_withdraw_daily (per account) are enforced; before broadcast the source is checked to hold enough native coin for gas.
  • Optional "from" selects the source address (must belong to the account); omitted → an owned address with enough balance is chosen.

Stuck / bump (RBF). Gas set too low leaves a tx in sent. List stuck withdrawals (no progress 25+ blocks after broadcast) and bump them (same nonce, +25% gas):

curl -s "http://localhost:8080/internal/v1/admin/withdrawals/stuck?chain=SEPOLIA"
curl -s -X POST .../internal/v1/admin/withdrawals/bump -d '{"account_id":1,"withdrawal_id":5}'

5.1 Concentration (deposit → hot wallet)

Sweep scattered deposits into the account's hot address (create one first). Native is one step; ERC20 is a gas-station two-step (the fee/hot wallet funds gas to the deposit address, then the token is swept). Booked only on confirmation.

curl -s -X POST .../internal/v1/admin/concentration \
  -d '{"account_id":1,"chain":"SEPOLIA","coin":"ETH","address":"0x…deposit"}'   # omit address = sweep all eligible
curl -s -X POST .../internal/v1/admin/concentration/advance -d '{"chain":"SEPOLIA"}'  # manual; the indexer also advances
curl -s "http://localhost:8080/internal/v1/admin/concentrations?account_id=1"

States: native pending→sweeping→success; ERC20 pending→gas_funding→gas_funded→sweeping→success. Threshold: coins.min_concentration. Ledger reasons: concentration_out/in/fee, gas_fund_out/in/fee.

5.2 Internal transfer (same account, book-only)

If a withdrawal's to is another address of the same account, no chain tx (and no gas) is used — the ledger moves directly (internal_out/internal_in), returning sign_method=internal, status=confirmed, empty tx_hash. A destination of a different account or external still goes on-chain.

5.3 Manual correction (errors reconcile can't fix)

For wrong/duplicate/phantom credits that on-chain truth cannot derive, post a compensating entry (balances are floored at 0; everything is audited):

# void a wrongly-credited deposit (auto-reverses its credit, marks it voided)
curl -s -X POST .../internal/v1/admin/adjust -d '{"void_deposit_id":123,"operator":"alice","note":"wrong account"}'
# signed balance correction (base units; negative = debit)
curl -s -X POST .../internal/v1/admin/adjust \
  -d '{"address":"0x…","chain":"SEPOLIA","coin":"ETH","delta_amount":"-500000000000000000","operator":"alice","note":"fix"}'

5.4 Coin / token management (add ERC20)

Adding a token is one coins row (chain + contract + decimals); the scanner then auto-detects its transfers — no code change.

curl -s -X POST .../internal/v1/admin/coins -d '{
  "chain":"SEPOLIA","symbol":"DAI","contract_address":"0x…","decimals":18,
  "min_concentration":"0","max_withdraw":"0","max_withdraw_daily":"0"}'
curl -s "http://localhost:8080/internal/v1/admin/coins?chain=SEPOLIA"

Decimals must be correct; native coins leave contract_address empty. onchain-balance compares the live chain balance with the ledger:

curl -s "http://localhost:8080/internal/v1/admin/onchain-balance?chain=SEPOLIA&address=0x…&coin=ETH"

6. Inspect the DB

mysql -h127.0.0.1 -P3306 -uuser1 -p custodial_wallet

Table Purpose
accounts integrators (api_key, mode, callback_url)
addresses custodial addresses (account, chain, type)
wallet_keys encrypted private keys (signer-private)
balances per address+coin (amount, locked_amount)
deposits inbound transfers (block, tx, confirmations, status)
withdrawals outbound requests (status, sign_method, tx_hash)
transactions on-chain tx for a withdrawal (nonce, gas, fee_paid, broadcast_block)
ledger_entries append-only audit of every balance change
concentrations sweeps (from→to, state, gas_fund_tx/sweep_tx)
scan_cursors last scanned block + hash per chain
coins coin config (contract, decimals, limits)
webhook_deliveries webhook attempts 
SELECT address, coin, amount, locked_amount FROM balances;
SELECT reason, coin, delta_amount, amount_after, ref_type, ref_id, tx_hash, created_at
  FROM ledger_entries WHERE address='0x…' ORDER BY id DESC;

7. End-to-end example (real Sepolia deposit)

ACC=$(curl -s -X POST http://localhost:8080/internal/v1/accounts -H 'content-type: application/json' --data '{"name":"demo","mode":"test"}')
export APIKEY=$(echo "$ACC" | python3 -c "import sys,json;print(json.load(sys.stdin)['api_key'])")
export SECRET=$(echo "$ACC" | python3 -c "import sys,json;print(json.load(sys.stdin)['api_secret'])")

sign_and_call POST /api/v1/addresses '{"chain":"SEPOLIA"}'              # -> address A
# fund A from a Sepolia faucet, then:
curl -s -X POST http://localhost:8080/internal/v1/rescan/tx -H 'content-type: application/json' \
  --data '{"chain":"SEPOLIA","tx_hash":"<funding_tx_hash>"}'
sign_and_call GET "/api/v1/balances?address=A" ''
sign_and_call POST /api/v1/withdrawals '{"client_withdraw_id":"wd-1","chain":"SEPOLIA","coin":"ETH","to":"0xDest","amount":"0.001"}'
curl -s -X POST http://localhost:8080/internal/v1/rescan/tx -H 'content-type: application/json' \
  --data '{"chain":"SEPOLIA","tx_hash":"<withdraw_tx_hash>"}'
sign_and_call GET /api/v1/withdrawals/1 ''