Add Prometheus metrics and Grafana dashboard#15
Conversation
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
| # Install runtime dependencies | ||
| RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
| libssl3 \ | ||
| ca-certificates \ | ||
| && apt-get clean && rm -rf /var/lib/apt/lists/* |
There was a problem hiding this comment.
In my machine, not installing usual SSL certificates was making zkboost server crash.
zkboost-1 | /app/zkboost-server: error while loading shared libraries: [libssl.so](http://libssl.so/).3: cannot open shared object file: No such file or directory
| #[derive(Clone, Default)] | ||
| #[derive(Clone)] |
There was a problem hiding this comment.
Default isn't used now since we have metrics. This is resolved with another helper for tests.
| let state = AppState::new(&config, metrics_handle).await?; | ||
|
|
||
| // Record application metrics | ||
| metrics::set_programs_loaded(state.programs.read().await.len()); |
There was a problem hiding this comment.
Mostly trying to catch potential misconfigs in the dashboard. I think we could also go deeper and have a table in Grafana -- but we can think more about it in the near future.
| &program_id.0, | ||
| true, | ||
| start.elapsed(), | ||
| report.total_num_cycles, |
There was a problem hiding this comment.
Note entirely suer the total_num_cycles is super useful. But considering at this layer zkboost is generic (and not only for Ethereum), I think might be good.
I think we can add also metrics in teh relayer whenever that fully exists, since it is another important piece of the flow.
han0110
left a comment
han0110
left a comment
There was a problem hiding this comment.
LGTM!
Tho while trying to spin up it with docker compose, I found when using real zkVM there will be some networking problems:
- zkboost uses
ere-dockerized, which spawns a server container and use client to connect to it - But when zkboost is also started as a container, even given the docker socket such that it can creates server container out of docker, it couldn't connect to it unless we set
network_mode: host - When we set
network_mode: hostto zkboost, the prometheus couldn't scrape the metrics, hence we need to make everything live in host network.
We'd need to figure out a better approach to make these configuration easier to do, one approach is to use ere-dockerized and make it aware we are in docker, and it shuold connect to the container name instead of localhost.
Agree. So we would need to create a network in the docker-compose setup, say, Any preferences on merging or leaving this PR open until this is fully solved? No strong opinion from my side. |
Sounds good!
Ah I think we can resolve it later, feel free to merge. |
2 participants
Considering the zkboost-server will be a pretty important piece of infrastructure, I'm adding some Prometheus metrics, so we can easily understand what's going on, rather than just looking at logs.
Also added a Grafana dashboard for those metrics, which also spins up with docker-compose.
Example:
Enter
https://localhost:3001, login with (admin, admin):(Not that interesting stuff showing half-ish of the dashboard -- prob on first real use we can check how the whole thing looks better)