Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .env.docker-compose
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ CARDANO_DB_SYNC_DIR=db-sync-data

## Hasura Variables
HASURA_PORT=8090
CARDANO_GRAPHQL_VERSION=9.0.1
CARDANO_GRAPHQL_VERSION=9.0.2

## Token Registry Variables
TOKEN_REGISTRY_VERSION=1.6.0
Expand All @@ -47,8 +47,10 @@ CIP_QUERY_PRIORITY=CIP_68,CIP_26
## Background Variables
LOGGER_MIN_SEVERITY=info
METADATA_SERVER_URI=http://token-metadata-registry:${TOKEN_REGISTRY_PORT}
ASSET_BACKFILL_BATCH_SIZE=500000
## Server Variables
API_PORT=3100
POLLING_INTERVAL_ADA_SUPPLY=21600000

## Index Service Variables (requires 'indexes' in COMPOSE_PROFILES)
# Minimum number of blocks to sync before creating indexes (default: 1000)
Expand Down
5 changes: 3 additions & 2 deletions .env.docker-compose-preprod
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ CARDANO_DB_SYNC_DIR=/opt/graph-ql-preprod/db-sync-data

## Hasura Variables
HASURA_PORT=8091
CARDANO_GRAPHQL_VERSION=9.0.1
CARDANO_GRAPHQL_VERSION=9.0.2

## Token Registry Variables
TOKEN_REGISTRY_VERSION=1.6.0
Expand All @@ -46,9 +46,10 @@ CIP_QUERY_PRIORITY=CIP_68,CIP_26
## Background Variables
LOGGER_MIN_SEVERITY=info
METADATA_SERVER_URI=http://token-metadata-registry:${TOKEN_REGISTRY_PORT}

ASSET_BACKFILL_BATCH_SIZE=500000
## Server Variables
API_PORT=3101
POLLING_INTERVAL_ADA_SUPPLY=21600000

## Index Service Variables (requires 'indexes' in COMPOSE_PROFILES)
# Minimum number of blocks to sync before creating indexes (default: 1000)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Check the [releases] for the latest version.
``` console
git clone \
--single-branch \
--branch 9.0.1 \
--branch 9.0.2 \
--recurse-submodules \
https://github.com/cardano-foundation/cardano-graphql.git \
&& cd cardano-graphql
Expand Down
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ services:
- LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY}
- METADATA_SERVER_URI=${METADATA_SERVER_URI}
- POSTGRES_PORT=${POSTGRES_PORT}
- ASSET_BACKFILL_BATCH_SIZE=${ASSET_BACKFILL_BATCH_SIZE}
restart: on-failure
secrets:
- postgres_db
Expand All @@ -198,6 +199,7 @@ services:
- CARDANO_SUBMIT_API_HOST=cardano-submit-api
- LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY}
- NETWORK=${NETWORK}
- POLLING_INTERVAL_ADA_SUPPLY=${POLLING_INTERVAL_ADA_SUPPLY}
- POSTGRES_PORT=${POSTGRES_PORT}
depends_on:
cardano-submit-api:
Expand Down
2 changes: 1 addition & 1 deletion dockerfiles/mithril/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y \
build-essential m4 libssl-dev docker.io jq git pkg-config \
&& apt-get clean

RUN git clone https://github.com/input-output-hk/mithril.git \
RUN git clone https://github.com/IntersectMBO/mithril.git \
&& export PATH="$HOME/.cargo/bin:$PATH" \
&& cd mithril \
&& git checkout $MITHRIL_VERSION \
Expand Down
12 changes: 6 additions & 6 deletions dockerfiles/mithril/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,18 @@ download_mithril_snapshot() {
case $NETWORK in
mainnet)
AGGREGATOR_ENDPOINT=${AGGREGATOR_ENDPOINT:-https://aggregator.release-mainnet.api.mithril.network/aggregator}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-mainnet/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-mainnet/ancillary.vkey)}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/release-mainnet/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/release-mainnet/ancillary.vkey)}
;;
preprod)
AGGREGATOR_ENDPOINT=${AGGREGATOR_ENDPOINT:-https://aggregator.release-preprod.api.mithril.network/aggregator}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-preprod/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-preprod/ancillary.vkey)}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/release-preprod/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/release-preprod/ancillary.vkey)}
;;
preview)
AGGREGATOR_ENDPOINT=${AGGREGATOR_ENDPOINT:-https://aggregator.pre-release-preview.api.mithril.network/aggregator}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/pre-release-preview/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/pre-release-preview/ancillary.vkey)}
GENESIS_VERIFICATION_KEY=${GENESIS_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/pre-release-preview/genesis.vkey)}
ANCILLARY_VERIFICATION_KEY=${ANCILLARY_VERIFICATION_KEY:-$(wget -q -O - https://raw.githubusercontent.com/IntersectMBO/mithril/main/mithril-infra/configuration/pre-release-preview/ancillary.vkey)}
;;
esac
echo "Listing content of /node dir:"
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"engines": {
"node": ">=14.15.0"
},
"version": "9.0.1",
"version": "9.0.2",
"private": true,
"workspaces": [
"packages/*"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ CREATE OR REPLACE VIEW "AdaPots" AS
epoch_no AS "epochNo",
deposits_stake,
deposits_drep,
deposits_proposal
deposits_proposal,
fees,
reserves,
rewards,
Expand Down
6 changes: 3 additions & 3 deletions packages/api-cardano-db-hasura/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cardano-graphql/api-cardano-db-hasura",
"version": "9.0.1",
"version": "9.0.2",
"description": "Module for interfacing with the Cardano DB, populated by cardano-db-sync that utilises Hasura for a powerful query interface",
"main": "dist/index.js",
"typings": "dist/index.d.ts",
Expand Down Expand Up @@ -34,7 +34,7 @@
"schema.graphql"
],
"dependencies": {
"@cardano-graphql/util": "9.0.1",
"@cardano-graphql/util": "9.0.2",
"@cardanosolutions/json-bigint": "^1.0.0",
"@emurgo/cip14-js": "^2.0.0",
"@graphql-tools/delegate": "^6.2.4",
Expand Down Expand Up @@ -65,7 +65,7 @@
"ts-log": "^2.2.3"
},
"devDependencies": {
"@cardano-graphql/util-dev": "9.0.1",
"@cardano-graphql/util-dev": "9.0.2",
"@graphql-codegen/cli": "^1.21.8",
"@graphql-codegen/typescript": "^1.23.0",
"@graphql-codegen/typescript-graphql-files-modules": "^1.18.1",
Expand Down
98 changes: 67 additions & 31 deletions packages/api-cardano-db-hasura/src/HasuraBackgroundClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ import {
const epochInformationNotYetAvailable =
'Epoch information not yet available. This is expected during the initial chain-sync.'

const ASSET_BACKFILL_ADVISORY_LOCK_KEY = 4021991017
const ASSET_BACKFILL_DEFAULT_BATCH_SIZE = 500000

const withHexPrefix = (value: string) =>
`\\x${value !== undefined ? value : ''}`

Expand Down Expand Up @@ -388,41 +391,74 @@ export class HasuraBackgroundClient {
}
}

public async backfillMissingAssets (dbConfig: DbConfig): Promise<string[]> {
public async backfillMissingAssets (dbConfig: DbConfig, batchSize: number = ASSET_BACKFILL_DEFAULT_BATCH_SIZE): Promise<string[]> {
const client = new Client(dbConfig)
await client.connect()
try {
await client.query(`
UPDATE "Asset" a
SET fingerprint = ma.fingerprint
FROM multi_asset ma
WHERE a."assetId" = CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA)
AND a.fingerprint IS NULL
`)
const result = await client.query(`
INSERT INTO "Asset" ("assetId", "assetName", "policyId", "fingerprint", "firstAppearedInSlot")
SELECT
CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA),
ma.name,
ma.policy,
ma.fingerprint,
MIN(b.slot_no)
FROM multi_asset ma
JOIN ma_tx_mint mtm ON mtm.ident = ma.id
JOIN tx ON tx.id = mtm.tx_id
JOIN block b ON b.id = tx.block_id
LEFT JOIN "Asset" a
ON a."assetId" = CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA)
WHERE a."assetId" IS NULL
GROUP BY ma.id, ma.policy, ma.name, ma.fingerprint
ON CONFLICT ("assetId") DO NOTHING
RETURNING encode("assetId", 'hex') AS "assetId"
`)
this.logger.info(
{ module: 'HasuraBackgroundClient', inserted: result.rowCount },
'Backfilled missing assets from multi_asset'
const lockResult = await client.query<{ locked: boolean }>(
'SELECT pg_try_advisory_lock($1) AS locked',
[ASSET_BACKFILL_ADVISORY_LOCK_KEY]
)
return result.rows.map((row: { assetId: string }) => row.assetId)
if (!lockResult.rows[0].locked) {
this.logger.warn(
{ module: 'HasuraBackgroundClient' },
'Asset backfill already in progress on another connection, skipping'
)
return []
}
try {
await client.query(`
UPDATE "Asset" a
SET fingerprint = ma.fingerprint
FROM multi_asset ma
WHERE a."assetId" = CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA)
AND a.fingerprint IS NULL
`)
const maxResult = await client.query<{ maxId: string }>(
'SELECT COALESCE(MAX(id), 0) AS "maxId" FROM multi_asset'
)
const maxId = Number(maxResult.rows[0].maxId)
const insertedAssetIds: string[] = []
for (let cur = 0; cur < maxId; cur += batchSize) {
const result = await client.query(`
INSERT INTO "Asset" ("assetId", "assetName", "policyId", "fingerprint", "firstAppearedInSlot")
SELECT
CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA),
ma.name,
ma.policy,
ma.fingerprint,
MIN(b.slot_no)
FROM multi_asset ma
JOIN ma_tx_mint mtm ON mtm.ident = ma.id
JOIN tx ON tx.id = mtm.tx_id
JOIN block b ON b.id = tx.block_id
LEFT JOIN "Asset" a
ON a."assetId" = CAST(CONCAT(ma.policy, RIGHT(CONCAT(E'\\\\', ma.name), -3)) AS BYTEA)
WHERE a."assetId" IS NULL
AND ma.id > $1
AND ma.id <= $2
GROUP BY ma.id, ma.policy, ma.name, ma.fingerprint
ON CONFLICT ("assetId") DO NOTHING
RETURNING encode("assetId", 'hex') AS "assetId"
`, [cur, cur + batchSize])
if (result.rowCount > 0) {
for (const row of result.rows as { assetId: string }[]) {
insertedAssetIds.push(row.assetId)
}
this.logger.info(
{ module: 'HasuraBackgroundClient', upToId: cur + batchSize, inserted: result.rowCount, total: insertedAssetIds.length },
'Backfilled asset batch from multi_asset'
)
}
}
this.logger.info(
{ module: 'HasuraBackgroundClient', inserted: insertedAssetIds.length },
'Backfilled missing assets from multi_asset'
)
return insertedAssetIds
} finally {
await client.query('SELECT pg_advisory_unlock($1)', [ASSET_BACKFILL_ADVISORY_LOCK_KEY])
}
} finally {
await client.end()
}
Expand Down
13 changes: 2 additions & 11 deletions packages/api-cardano-db-hasura/src/HasuraClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,18 +124,9 @@ export class HasuraClient {
)
})
this.logger.debug({ module: 'HasuraClient' }, 'graphql-engine setup')
await pRetry(async () => {
await this.adaPotsToCalculateSupplyFetcher.initialize()
}, {
factor: 1.1,
forever: true,
maxTimeout: 15000,
onFailedAttempt: util.onFailedAttemptFor(
'Initializing data fetchers',
this.logger
)
this.adaPotsToCalculateSupplyFetcher.initialize().catch(error => {
this.logger.error({ module: 'HasuraClient', err: error }, 'AdaPots fetcher initialization failed')
})
this.logger.debug({ module: 'HasuraClient' }, 'Data fetchers initialized')
this.state = 'initialized'
this.logger.info({ module: 'HasuraClient' }, 'Initialized')
}
Expand Down
7 changes: 6 additions & 1 deletion packages/api-cardano-db-hasura/src/background.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export interface BackgroundConfig {
metadataUpdateInterval?: {
assets: number;
};
assetBackfillBatchSize?: number;
}

async function getConfig (): Promise<BackgroundConfig> {
Expand Down Expand Up @@ -94,6 +95,7 @@ function filterAndTypecastEnvs (env: any) {
const {
COMPOSE_PROFILES,
ASSET_METADATA_UPDATE_INTERVAL,
ASSET_BACKFILL_BATCH_SIZE,
HASURA_CLI_PATH,
HASURA_CLI_EXT_PATH,
HASURA_URI,
Expand All @@ -120,6 +122,9 @@ function filterAndTypecastEnvs (env: any) {
? Number(ASSET_METADATA_UPDATE_INTERVAL)
: undefined
},
assetBackfillBatchSize: ASSET_BACKFILL_BATCH_SIZE
? Number(ASSET_BACKFILL_BATCH_SIZE)
: undefined,
postgres: {
db: POSTGRES_DB,
dbFile: POSTGRES_DB_FILE,
Expand Down Expand Up @@ -201,7 +206,7 @@ function startAssetPolling (
try {
await hasuraBackgroundClient.initialize()
const lastSeenId = await hasuraBackgroundClient.getMaxMultiAssetId(config.db)
const backfilledAssetIds = await hasuraBackgroundClient.backfillMissingAssets(config.db)
const backfilledAssetIds = await hasuraBackgroundClient.backfillMissingAssets(config.db, config.assetBackfillBatchSize)
await worker.initQueue()
startAssetPolling(hasuraBackgroundClient, worker, config.db, lastSeenId, logger)
await metadataClient.initialize()
Expand Down
10 changes: 6 additions & 4 deletions packages/api-cardano-db-hasura/test/ada.query.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import path from 'path'

import { DocumentNode } from 'graphql'
import pRetry from 'p-retry'
import util from '@cardano-graphql/util'
import { TestClient } from '@cardano-graphql/util-dev'
import { testClient } from './util'
Expand All @@ -23,16 +24,17 @@ describe('ada', () => {
})

it('returns ada supply information', async () => {
const result = await client.query({
query: await loadQueryNode('adaSupply')
})
const query = await loadQueryNode('adaSupply')
const result = await pRetry(
() => client.query({ query }),
{ retries: 40, minTimeout: 3000, factor: 1 }
)
const { ada } = result.data
const circulatingSupply = new BigNumber(ada.supply.circulating).toNumber()
const maxSupply = new BigNumber(ada.supply.max).toNumber()
const totalSupply = new BigNumber(ada.supply.total).toNumber()
expect(maxSupply).toEqual(genesis.shelley.maxLovelaceSupply)
expect(maxSupply).toBeGreaterThan(circulatingSupply)
// expect(totalSupply).toBeGreaterThan(circulatingSupply)
expect(totalSupply).toBeLessThan(maxSupply)
})
})
2 changes: 1 addition & 1 deletion packages/cli/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cardano-graphql/cli",
"version": "9.0.1",
"version": "9.0.2",
"description": "Management tool for managing a Cardano GraphQL deployment",
"main": "./dist/index.js",
"bin": {
Expand Down
2 changes: 1 addition & 1 deletion packages/client-ts/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cardano-graphql/client-ts",
"version": "9.0.1",
"version": "9.0.2",
"description": "A client package for Cardano GraphQL, including the GraphQL schema and TypeScript definitions generated from it",
"repository": {
"type": "git",
Expand Down
5 changes: 3 additions & 2 deletions packages/index-service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ A Docker service for creating performance indexes on the cardano-db-sync Postgre
The index service creates additional database indexes to improve query performance. These indexes are **optional** but **recommended for production** because:

- `idx_ma_tx_mint_ident` is required for efficient asset polling (without it polls degrade from <1ms to 500ms+)
- `idx_tx_out_address` significantly speeds up payment address queries
- `idx_tx_out_address_texthash` is required for payment address / UTXO-by-address queries; without it these do a full sequential scan of `tx_out` (hundreds of millions of rows), which can saturate the connection pool under concurrent load
- `idx_asset_fingerprint` speeds up asset fingerprint lookups

Index creation is not enabled by default because:
Expand Down Expand Up @@ -68,7 +68,7 @@ docker compose ps index-service
| Index | Table | Speeds up |
|-------|-------|-----------|
| `idx_ma_tx_mint_ident` | `ma_tx_mint.ident` | New asset polling (critical for background service) |
| `idx_tx_out_address` | `tx_out.address` | Payment address queries |
| `idx_tx_out_address_texthash` | `tx_out.address` | Payment address / UTXO-by-address queries (prevents full-table scans) |
| `idx_asset_fingerprint` | `Asset.fingerprint` | Asset fingerprint lookups |

## Operations
Expand Down Expand Up @@ -117,6 +117,7 @@ docker compose exec postgres psql -U $(cat placeholder-secrets/postgres_user) \

# Drop all custom indexes
DROP INDEX CONCURRENTLY IF EXISTS idx_ma_tx_mint_ident;
DROP INDEX CONCURRENTLY IF EXISTS idx_tx_out_address_texthash;
DROP INDEX CONCURRENTLY IF EXISTS idx_tx_out_address;
DROP INDEX CONCURRENTLY IF EXISTS idx_asset_fingerprint;
```
Expand Down
Loading
Loading