Clickhouse DB Support #293
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Hey! I am happy to see that someone is taking over this :) I will try to throw a review if I have some time |
There was a problem hiding this comment.
i think one part you missed is the rust project types generation part where we inject the database for them to use
pub struct EventContext<TExtensions>
where
TExtensions: Send + Sync,
{
pub database: Arc<PostgresClient>,
pub extensions: Arc<TExtensions>,
}
and then you can use that in context.database we should based on your config change PostgresClient > Clickhouse
this should also be the same when building their indexers to use the clickhouse one if clickhouse enabled
|
Sorry I got busy the last week, I will aim to address this next week so we can get it merged |
|
@jaggad is attempting to deploy a commit to the joshaavecom's projects Team on Vercel. A member of the Team first needs to authorize it. |
| sql.split(';').map(str::trim).filter(|s| !s.is_empty()).collect(); | ||
|
|
||
| for statement in statements { | ||
| self.execute(statement).await?; |
There was a problem hiding this comment.
question: how does it work under the hood, will it block next statements execution or it's more like fire and forget?
There was a problem hiding this comment.
This blocks each statement right now, as far as i know we don't have native clickhouse client support for simple-batch like postgres.
So this takes those "simple batch pipelines" and just runs them as consecutive queries.
It is only used in rindexer startup to create tables / do nothing, so it's very fast atm
documentation/docs/pages/docs/start-building/yaml-config/storage.mdx
Outdated
Show resolved
Hide resolved
4 participants
Clickhouse DB Support
Clickhouse writing is a simple integration to allow an alternate storage engine to
Postgres. All events are indexed in an "generically-indexed" table, where it's a valid cross-chain unique constraint which allows a level of safety that rindexer-postgres without a unique index cannot replicate.No-code
The no-code project will by default use this sort order and so query filters will perform best when this all prior filters are applied (e.g when searching for a tx hash, this will be most efficient when searching
network = ?, block_number = ?, tx_hash = ?.The no-code methodology for constructing tables takes tries to resemble the Postgres method and utilises similar functions and types such as
EthereumSqlTypeWrapperfor simplicity of integration.Rust
Client
The
rindexer codegen typingswill generate valid typings as per normal, and will include aClickhouseClientfor use in the indexer handlers.The
rindexer codegen indexercommand also works and will generate valid Clickhouse bulk inserts. Please note that there are binary serialization protocols, as well as long-running-append protocols that may be more desirable for certain extreme workloads.In most cases, the clickhouse client that is provided in the handler
contextis acceptable.Custom primary keys
The rust project is much more flexible. It is expected that most applications with large-scale data storage and specific query patterns will need the primary key (order by) for the data to be different. Therefore the tables can be defined on the db however needed, and the writes can be done however needed.
In the example above a "unique tx search" may be more efficient to query as follows:
Demo
recording.mp4
Credit to @JosepBove's initial implementation for saving me a few hours of work: #123