Skip to content

Use @fastify/fastify-postgres. CheckerNetwork/roadmap#220 #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

Goddhi
Copy link

@Goddhi Goddhi commented Mar 10, 2025

Added fastify and @fastify/postgres as dependencies
Updated stats/bin/spark-stats.js to use Fastify's PostgreSQL plugin
Improved connection error handling and graceful shutdown

EDIT @juliangruber:
For CheckerNetwork/roadmap#220

Copy link
Contributor

@pyropy pyropy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your submission @Goddhi 🚀

Good start, but there's some changes that need to be addressed:

  • Install @fastify/postgres package in correct workspace
  • Revert or rethink on how the migrations are done
  • Avoid creating new Fastify instances in order to register databases
  • Figure out how to perform queries from fastify handlers

db/package.json Outdated
Comment on lines 18 to 19
"@fastify/postgres": "^5.2.0",
"@fastify/url-data": "^6.0.3"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"@fastify/postgres": "^5.2.0",
"@fastify/url-data": "^6.0.3"

I think we should remove these packages from db workspace and install @fastify/postgres inside the stats workspace only. You can install workspace specific packages by running npm install @fastify/postgres -w stats

EVALUATE_DB_URL
} = process.env

const app = Fastify({ logger: false })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we shouldn't really create a new Fastify instance in order to run migrations. This should be done without the @fastify/postgres package.

} = process.env

const pgPools = await getPgPools()

const dbFastify = Fastify({ logger: false })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we shouldn't create another Fastify instance here as we already do it inside createApp function. Rather, we should move registering the @fastify/postgres plugin and databases to be executed within createApp funciton.


await dbFastify.register(fastifyPostgres, {
connectionString: DATABASE_URL,
name: 'stats',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is wrong because because we're instantiate only connection to stats database, but we do reference evaluate database bellow.

stats/lib/db.js Outdated
Comment on lines 5 to 6
let fastifyApp = null
let isInitialized = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let fastifyApp = null
let isInitialized = false

I would avoid using globals in this case.

stats/lib/db.js Outdated
if (isInitialized) return

// Create a minimal Fastify instance for database connections
fastifyApp = Fastify({ logger: false })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of creating a new Fastify instance I think it would be better to pass down existing instance as function argument.

Copy link
Member

@bajtos bajtos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to what @pyropy wrote

@Goddhi
Copy link
Author

Goddhi commented Mar 13, 2025

@pyropy I do appreciate your reviews, i have made the following changes as requested, kindly look into thanks.
Moved @fastify/postgres dependencies to correct workspace
make the database connections to a single Fastify instance
removed the db.js which implements multiple Fastify instances. e.t.c.

Copy link
Contributor

@pyropy pyropy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try to avoid having dead code around. Also, let's try to use new way of accessing databases in the API handlers.

stats/lib/app.js Outdated
Comment on lines 40 to 46
const pgPools = {
stats: app.pg.stats,
evaluate: app.pg.evaluate,
async end() {
await app.close()
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const pgPools = {
stats: app.pg.stats,
evaluate: app.pg.evaluate,
async end() {
await app.close()
}
}

I don't see this being used anywhere. If it's not used, let's removed. YAGNI

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is being passed to addRoutes() and addPlatformRoutes() below. However, instead of passing pgPools, the route handlers should access app.pg.stats etc. directly

Comment on lines 7 to 10
const {
DATABASE_URL,
EVALUATE_DB_URL
} = process.env
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const {
DATABASE_URL,
EVALUATE_DB_URL
} = process.env

These constants seem not to be used anywhere. Let's delete them.

console.log('Running migrations for stats database...')
const pgPools = await getPgPools()

// @ts-ignore - PgPoolStats actually does have a query method at runtime
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try to add correct types to migrate functions rather then using @ts-ignore. Using it should be avoided.

// @ts-ignore - PgPoolStats actually does have a query method at runtime
await migrateStatsDB(pgPools.stats)

// @ts-ignore - Similarly for evaluate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, let's please not use @ts-ignore

@pyropy
Copy link
Contributor

pyropy commented Mar 17, 2025

@Goddhi I suggest you link the issue you're working on in the PR description (and not in title). In this case you should've linked this issue CheckerNetwork/roadmap#220

stats/lib/app.js Outdated
Comment on lines 40 to 46
const pgPools = {
stats: app.pg.stats,
evaluate: app.pg.evaluate,
async end() {
await app.close()
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is being passed to addRoutes() and addPlatformRoutes() below. However, instead of passing pgPools, the route handlers should access app.pg.stats etc. directly

@Goddhi
Copy link
Author

Goddhi commented Mar 19, 2025

@pyropy and @juliangruber have made commits that addresses the reviews.
I have updated route handlers to access database connections via request.server.pg which is a property from the @fastify/postgres and as well simplified my code by removing unnecessary abstractions as suggested.

@Goddhi Goddhi changed the title fastify-postgres--implementationUse Use @fastify/fastify-postgres in our REST API services #220 https://github.com/CheckerNetwork/roadmap/issues/220 fastify-progres-implement Mar 19, 2025
@Goddhi Goddhi changed the title https://github.com/CheckerNetwork/roadmap/issues/220 fastify-progres-implement (CheckerNetwork/roadmap#220)[https://github.com/CheckerNetwork/roadmap/issues/220] Mar 19, 2025
Comment on lines 53 to 54
const pgPools = adaptPgPools(request.server.pg);
reply.send(await fetchTopEarningParticipants(pgPools.stats, request.filter)) })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const pgPools = adaptPgPools(request.server.pg);
reply.send(await fetchTopEarningParticipants(pgPools.stats, request.filter)) })
reply.send(await fetchTopEarningParticipants(request.server.pg.stats, request.filter)) })

What do we need the adaptPgPools function for here?

app.register(async app => {
app.addHook('preHandler', filterPreHandlerHook)
app.addHook('onSend', filterOnSendHook)

app.get('/deals/daily', async (/** @type {RequestWithFilter} */ request, reply) => {
const pgPools = adaptPgPools(request.server.pg);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about instead changing the signature of fetchDaily... etc to accept request.server.pg? fetchDailyDealStats(request.server.pg, request.filter) etc

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright, noted, i will do that now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on changing the signature and dropping the adaptPgPools function.

@Goddhi Goddhi changed the title (CheckerNetwork/roadmap#220)[https://github.com/CheckerNetwork/roadmap/issues/220] [CheckerNetwork/roadmap#220](https://github.com/CheckerNetwork/roadmap/issues/220) Mar 19, 2025
@juliangruber juliangruber changed the title [CheckerNetwork/roadmap#220](https://github.com/CheckerNetwork/roadmap/issues/220) Use @fastify/fastify-postgres in our REST API services. CheckerNetwork/roadmap#220 Mar 19, 2025
@juliangruber juliangruber changed the title Use @fastify/fastify-postgres in our REST API services. CheckerNetwork/roadmap#220 Use @fastify/fastify-postgres. CheckerNetwork/roadmap#220 Mar 19, 2025
Copy link
Contributor

@pyropy pyropy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're making great progress @Goddhi! 🚀

app.register(async app => {
app.addHook('preHandler', filterPreHandlerHook)
app.addHook('onSend', filterOnSendHook)

app.get('/deals/daily', async (/** @type {RequestWithFilter} */ request, reply) => {
const pgPools = adaptPgPools(request.server.pg);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on changing the signature and dropping the adaptPgPools function.

@Goddhi
Copy link
Author

Goddhi commented Mar 19, 2025

@juliangruber and @pyropy i have modified the fetcher functions signatures to accept Fastify's pg object directly and removed the adaptPgPools as suggested.

"postgrator": "^8.0.0"
},
"standard": {
"env": [
"mocha"
]
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@@ -6,4 +6,4 @@ import {

const pgPools = await getPgPools()
await migrateStatsDB(pgPools.stats)
await migrateEvaluateDB(pgPools.evaluate)
await migrateEvaluateDB(pgPools.evaluate)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
await migrateEvaluateDB(pgPools.evaluate)
await migrateEvaluateDB(pgPools.evaluate)

stats/lib/app.js Outdated
Comment on lines 38 to 39


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change


const ONE_DAY = 24 * 60 * 60 * 1000

/**
* @param {Queryable} pgPool
* @param {Object} pg
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @param {Object} pg
* @param {Queryable} pg

I believe this still follows the Queryable interface. Object is too unspecific.

Comment on lines 9 to 12

/**
* @param {FastifyPg} pg - Fastify pg object with database connections
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/**
* @param {FastifyPg} pg - Fastify pg object with database connections
*/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@@ -35,7 +41,7 @@ export const fetchRetrievalSuccessRate = async (pg, filter) => {
}

/**
* @param {Object} pg
* @param {any} pg - Fastify pg object with database connections
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @param {any} pg - Fastify pg object with database connections
* @param {FastifyPg} pg - Fastify pg object with database connections

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@@ -399,7 +406,7 @@ export const fetchClientsRSRSummary = async (pg, filter) => {

/**
* Fetches the retrieval stats summary for a single client for given date range.
* @param {Object} pg
* @param {{stats: Queryable, evaluate: Queryable}} pg
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this not also be FastifyPg?

@@ -1,10 +1,16 @@
import { FastifyRequest } from 'fastify'
import { PostgresDb } from '@fastify/postgres'
import { Queryable } from '@filecoin-station/spark-stats-db'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this type imported for?

@juliangruber
Copy link
Member

@Goddhi CI is failing, please take a look

@Goddhi
Copy link
Author

Goddhi commented Mar 20, 2025

@Goddhi CI is failing, please take a look

FIXED

@juliangruber
Copy link
Member

@Goddhi CI is still failing. You can run npm test locally to test this

@Goddhi
Copy link
Author

Goddhi commented Mar 20, 2025

@Goddhi CI is still failing. You can run npm test locally to test this

i have fixed other issues i found after running test, but the only issue left is this
standard: Use JavaScript Standard Style (https://standardjs.com)
/home/goddhi/spark-stats/stats/lib/platform-routes.js:2:3: 'fetchDailyStationCount' is defined but never used. (no-unused-vars)
i also observed this handler function is also not used /stats/lib/platform-routes.js of the main branch, i dont know if the CI will fail if i make commit now.

@juliangruber
Copy link
Member

The function is used here:

reply.send(await fetchDailyStationCount(pgPools.evaluate, request.filter))
🤔

Copy link
Contributor

@pyropy pyropy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great progress 👍🏻

Let's clean the code up a bit; Let's try to pass database instance directly to the fetcher in which case we don't have to change fetcher signatures.

Also I'd advise you to format your code using standard as that's what we use in majority of our repositories. You can do it simply by running npx standard --fix.

Comment on lines 9 to 10
DATABASE_URL,
EVALUATE_DB_URL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add default values for both env variables:

Suggested change
DATABASE_URL,
EVALUATE_DB_URL
DATABASE_URL = 'postgres://localhost:5432/spark_stats',
EVALUATE_DB_URL = 'postgres://localhost:5432/spark_evaluate'

})
app.get('/stations/monthly', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchMonthlyStationCount(pgPools.evaluate, request.filter))
reply.send(await fetchMonthlyStationCount(request.server.pg, request.filter))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
reply.send(await fetchMonthlyStationCount(request.server.pg, request.filter))
reply.send(await fetchMonthlyStationCount(request.server.pg.evaluate, request.filter))

We could also pass correct instance of the database instead of passing the pg object. In that case we don't need to change fetchers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This throws an error reason why went with only the pg object, besides the fetcher functions expects full pg object and uses pg.stats or pg.evaluate interally.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error does it throw?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

* @param {import('./typings.js').DateRangeFilter} filter
*/
export const fetchDailyStationCount = async (pgPool, filter) => {
const { rows } = await pgPool.query(`
export const fetchDailyStationCount = async (pg, filter) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case we pass correct database instance we don't need to change this handler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

})
app.get('/retrieval-result-codes/daily', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchDailyRetrievalResultCodes(pgPools, request.filter))
reply.send(await fetchDailyRetrievalResultCodes(request.server.pg, request.filter))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
reply.send(await fetchDailyRetrievalResultCodes(request.server.pg, request.filter))
reply.send(await fetchDailyRetrievalResultCodes(request.server.pg.stats, request.filter))

We could also pass correct instance of the database instead of passing the pg object. In that case we don't need to change fetchers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@Goddhi
Copy link
Author

Goddhi commented Mar 23, 2025

@pyropy and @juliangruber i have made a commits regarding your suggestions, kindly help review it thanks.

})
app.get('/stations/monthly', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchMonthlyStationCount(pgPools.evaluate, request.filter))
reply.send(await fetchMonthlyStationCount(request.server.pg, request.filter))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error does it throw?

Comment on lines 9 to 12

/**
* @param {FastifyPg} pg - Fastify pg object with database connections
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

* @param {import('./typings.js').DateRangeFilter} filter
*/
export const fetchDailyStationCount = async (pgPool, filter) => {
const { rows } = await pgPool.query(`
export const fetchDailyStationCount = async (pg, filter) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

})
app.get('/retrieval-result-codes/daily', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchDailyRetrievalResultCodes(pgPools, request.filter))
reply.send(await fetchDailyRetrievalResultCodes(request.server.pg, request.filter))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@juliangruber
Copy link
Member

juliangruber commented Mar 24, 2025

@Goddhi tests are still failing, please run npm test to verify locally. See the CI failures here

@Goddhi
Copy link
Author

Goddhi commented Mar 25, 2025

@Goddhi tests are still failing, please run npm test to verify locally. See the CI failures here

Yeah the reason the tests are failing is becuase i need to modify the test files to sync with the changes made in the fetcher and route files, this is what i am currently doing.

@Goddhi
Copy link
Author

Goddhi commented Mar 25, 2025

#341 (comment)
@juliangruber this the errow that it throws request.server.pg.evaluate Argument of type 'PostgresDb' is not assignable to parameter of type 'FastifyPg'. Type 'PostgresDb' is not assignable to type 'Record<string, PostgresDb>'. Property 'pool' is incompatible with index signature. Type 'Pool' is missing the following properties from type 'PostgresDb': pool,

await pgPools.stats.query('DELETE FROM daily_scheduled_rewards')
await pgPools.stats.query('DELETE FROM daily_reward_transfers')
await pgPools.stats.query('DELETE FROM daily_retrieval_result_codes')
await app.pg.evaluate.query('DELETE FROM retrieval_stats')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's preserve pgPools structure please so that we don't have to change every source code line in tests that is working with the DB.

For example:

before(async () => {
  app = await createApp({
    // ..
  })

  pgPools = app.pg
})

// no changes are needed in individual tests

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, noted

@@ -472,12 +472,12 @@ export const fetchAllocatorsRSRSummary = async (pgPools, filter) => {

/**
* Fetches the retrieval stats summary for a single allocator for given date range.
* @param {PgPools} pgPools
* @param {FastifyPg} pg - Fastify pg object with database connections
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the name pgPools to keep the diff smaller?

Suggested change
* @param {FastifyPg} pg - Fastify pg object with database connections
* @param {FastifyPg} pgPools - Fastify pg object with database connections

Same comment applies to all other similar places too.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, noted

"postgrator": "^8.0.0"
},
"standard": {
"env": [
"mocha"
]
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@@ -3,7 +3,6 @@ import {
migrateEvaluateDB,
migrateStatsDB
} from '@filecoin-station/spark-stats-db'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please undo this change to keep the diff clean

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, noted

})
app.get('/stations/monthly', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchMonthlyStationCount(pgPools.evaluate, request.filter))
reply.send(await fetchMonthlyStationCount(request.server.pg, request.filter))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

})
app.get('/retrieval-result-codes/daily', async (/** @type {RequestWithFilter} */ request, reply) => {
reply.send(await fetchDailyRetrievalResultCodes(pgPools, request.filter))
reply.send(await fetchDailyRetrievalResultCodes(request.server.pg, request.filter))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@@ -35,7 +41,7 @@ export const fetchRetrievalSuccessRate = async (pg, filter) => {
}

/**
* @param {Object} pg
* @param {any} pg - Fastify pg object with database connections
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @Goddhi

@Goddhi
Copy link
Author

Goddhi commented Apr 2, 2025

@pyropy Please can you help check what could be causing the CI to fail?
i have fixed the line trailing issues, now the issue seems to be this "AssertionError [ERR_ASSERTION]: INFLUXDB_TOKEN required"
i guess the token needs to be updated from the action secret. Thanks
@juliangruber please kindly help look into this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants