feat: setup otlp prometheus exporter with correct invocation in node by 0xIcarus · Pull Request #1087 · movementlabsxyz/movement

0xIcarus · 2025-03-07T11:17:13Z

Summary

RFCs: Link to RFC, Link to RFC, or $\emptyset$.
Categories: any of networks, util.
Tracing crate for metrics with prometheus
Refer to issues Integrate Prometheus Exporter and add basic tracing #1126 and Providing metrics support through OTLP Prometheus Exporter #1099

Changelog

Modified existing tracing crate to support prometheus metrics thru OTLP. Created an exporter and an example invocation in the movement full node

Testing

Command used to run the prometheus exporter:

nix develop --command bash -c "just movement-full-node native build.setup.eth-local.celestia-local.telemetry --keep-project"

Before running, make sure prometheus is installed locally:

brew install prometheus

Outstanding issues

Co-authored-by: Mikhail Zabaluev <mikhail.zabaluev@movementlabs.xyz>

Co-authored-by: Andy Golay <andygolay@gmail.com> Co-authored-by: musitdev <philippe.delrieu@free.fr> Co-authored-by: Mikhail Zabaluev <mikhail.zabaluev@movementlabs.xyz> Co-authored-by: Richard Melkonian <35300528+0xmovses@users.noreply.github.com> Co-authored-by: Icarus131 <anirudhprasad131@protonmail.com> Co-authored-by: primata <primata@movementlabs.xyz> Co-authored-by: Icarus131 <anirudhprasad131@gmail.com> Co-authored-by: Richard Melkonian <r.v.melkonian@gmail.com>

Co-authored-by: Andy Golay <andygolay@gmail.com> Co-authored-by: musitdev <philippe.delrieu@free.fr> Co-authored-by: Mikhail Zabaluev <mikhail.zabaluev@movementlabs.xyz> Co-authored-by: Richard Melkonian <35300528+0xmovses@users.noreply.github.com> Co-authored-by: Icarus131 <anirudhprasad131@protonmail.com> Co-authored-by: primata <primata@movementlabs.xyz> Co-authored-by: Icarus131 <anirudhprasad131@gmail.com> Co-authored-by: Richard Melkonian <r.v.melkonian@gmail.com> Co-authored-by: Radu Popa <radupopa21be@gmail.com>

Co-authored-by: Liam Monninger <l.mak.monninger@gmail.com>

…#1075) Co-authored-by: Andy Golay <andygolay@gmail.com> Co-authored-by: musitdev <philippe.delrieu@free.fr> Co-authored-by: Mikhail Zabaluev <mikhail.zabaluev@movementlabs.xyz> Co-authored-by: Richard Melkonian <35300528+0xmovses@users.noreply.github.com> Co-authored-by: Icarus131 <anirudhprasad131@protonmail.com> Co-authored-by: primata <primata@movementlabs.xyz> Co-authored-by: Icarus131 <anirudhprasad131@gmail.com> Co-authored-by: Richard Melkonian <r.v.melkonian@gmail.com> Co-authored-by: Radu Popa <radupopa21be@gmail.com>

…tive process using process-compose

0xmovses · 2025-04-01T17:15:55Z

Ok first, all the merge conflicts need to be resolved and merge main in to this branch, that will resolve the diff here. Thanks!

0xmovses

Am I missing a local step for dev?

when running the cmd, prometheus process yields
bash: line 4: prometheus: command not found

cargo check --all-target gives some errors also:

error: cannot find macro `debug` in this scope
  --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:81:3
   |
81 |         debug!("Block tx execution: {:?}", tx_execution_results);
   |         ^^^^^
   |
help: consider importing one of these items
   |
1  + use aptos_logger::debug;
   |
1  + use tracing::debug;
   |

error[E0425]: cannot find function `aptos_test_root_address` in this scope
   --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:339:4
    |
339 |             aptos_test_root_address(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    |
help: consider importing one of these items
    |
233 +     use aptos_sdk::types::account_config::aptos_test_root_address;
    |
233 +     use aptos_types::account_config::aptos_test_root_address;
    |

error[E0433]: failed to resolve: use of undeclared type `AccountKey`
   --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:340:4
    |
340 |             AccountKey::from_private_key(private_key),
    |             ^^^^^^^^^^ use of undeclared type `AccountKey`
    |
help: consider importing this struct
    |
233 +     use aptos_sdk::types::AccountKey;
    |

error[E0425]: cannot find function `aptos_test_root_address` in this scope
   --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:455:4
    |
455 |             aptos_test_root_address(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    |
help: consider importing one of these items
    |
233 +     use aptos_sdk::types::account_config::aptos_test_root_address;
    |
233 +     use aptos_types::account_config::aptos_test_root_address;
    |

error[E0433]: failed to resolve: use of undeclared type `AccountKey`
   --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:456:4
    |
456 |             AccountKey::from_private_key(private_key),
    |             ^^^^^^^^^^ use of undeclared type `AccountKey`
    |
help: consider importing this struct
    |
233 +     use aptos_sdk::types::AccountKey;
    |

warning: unused import: `aptos_crypto::ValidCryptoMaterialStringExt`
   --> protocol-units/execution/maptos/opt-executor/src/executor/execution.rs:238:6
    |
238 |     use aptos_crypto::ValidCryptoMaterialStringExt;
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: the lint level is defined here
   --> protocol-units/execution/maptos/opt-executor/src/lib.rs:4:8
    |
4   | #[warn(unused_imports)]
    |        ^^^^^^^^^^^^^^

error: could not compile `maptos-opt-executor` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
Some errors have detailed explanations: E0425, E0433.
For more information about an error, try `rustc --explain E0425`.
warning: `maptos-opt-executor` (lib test) generated 1 warning
error: could not compile `maptos-opt-executor` (lib test) due to 5 previous errors; 1 warning emitte

0xmovses · 2025-04-03T16:54:32Z

      run: |
        nix develop --command bash  -c "just movement-full-node native build.setup.eth-local.celestia-local.test-ggp-gas-fee -t=false"

+  core-resource-signer:


Not sure why this test has been added, it runs test-ggp-gas-fee, but we're already running that elsewhere in this file.

0xmovses · 2025-04-04T11:34:40Z

+    ports:
+      - "3000:3000"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning


@nicolaslpf will these values be the same in prod? Or do you imagine we will need some telelmetry.local.yml and a telemetry.production.yml?

…otlp

ganymedio · 2025-04-07T14:44:54Z

Is this the expected output? (Failed to read meta.json):

Also I'm noticing when I run this it tends to cause my laptop to hang up, at least when I try click on tui or hit Ctrl+C to quit, it's not reponsive... not sure if that's just something I'm getting locally, or if others are experiencing it?

0xIcarus · 2025-04-07T15:01:58Z

Is this the expected output? (Failed to read meta.json):

Also I'm noticing when I run this it tends to cause my laptop to hang up, at least when I try click on tui or hit Ctrl+C to quit, it's not reponsive... not sure if that's just something I'm getting locally, or if others are experiencing it?

@andygolay ideally it should launch prometheus and should be accessible at localhost:9091. I have not encountered this error before. I will try reproducing it on my end. As for the tui hanging, its because of how process compose is currently handling prometheus, you can just kill the prometheus process to bring back the tui (can use htop). I am working on a fix for that currently.

ganymedio · 2025-04-07T15:52:29Z

In Prometheus (at http://localhost:9091/targets) I'm getting Error scraping target: Get "http://127.0.0.1:9464/metrics": dial tcp 127.0.0.1:9464: connect: connection refused

…point

0xIcarus · 2025-04-07T16:11:58Z

In Prometheus (at http://localhost:9091/targets) I'm getting Error scraping target: Get "http://127.0.0.1:9464/metrics": dial tcp 127.0.0.1:9464: connect: connection refused

@andygolay, it should work now

ganymedio

I was able to get some metrics

after about 10 - 15 minutes the process-compose stopped being responsive and then I started getting Error scraping target: Get "http://127.0.0.1:9464/metrics": dial tcp 127.0.0.1:9464: connect: connection refused again in the Prometheus targets page.

So it did work, but then crashed after a while. I'll defer to @0xmovses @musitdev @mzabaluev as to whether that's sufficient to merge this PR, or whether the process-compose reliability should be fixed more thoroughly first?

0xIcarus · 2025-04-09T10:14:24Z

I have updated the command to be used to run this in the PR description. When run without the tests, it seems to work perfectly fine, without any of the crashing issues.

ganymedio · 2025-04-09T10:20:37Z

+      env:
+        CELESTIA_LOG_LEVEL: FATAL # adjust the log level while debugging
+      run: |
+        nix develop --command bash  -c "just movement-full-node native build.setup.eth-local.celestia-local.test-ggp-gas-fee -t=false"


@Icarus131 I'm unclear on why this duplicate ggp test is added. Was it perhaps meant to be a CI check for the telemetry?

I wasn't sure how you're testing the docker-compose.telemetry.yml ... how does that work? Just general container tests? Or should there be a test for telemetry in CI?

ganymedio

The setup looks good; I've run Prometheus with the updated command and tried generating a few histograms; the service didn't crash for over 20 minutes while I was testing it.

Nice job @Icarus131

0xmovses · 2025-04-09T12:27:18Z

It's running all ok for me. I did a brew install prometheus but how are you guys getting the UI up? I don't have Docker runnings, assuming that's a prerequisite?

0xIcarus · 2025-04-09T12:36:12Z

It's running all ok for me. I did a brew install prometheus but how are you guys getting the UI up? I don't have Docker runnings, assuming that's a prerequisite?

@0xmovses a prometheus session is spawned at localhost:9091, should be accessible as soon as it starts running in process compose. Once full node also starts running, you will be able to see it active in the /targets route

0xmovses · 2025-04-09T12:37:18Z

Ok, trying agin. Thanks.

0xmovses · 2025-04-09T12:47:55Z

Forget my last message, after some time, it loaded at :9091! 👍

0xmovses

We define ENV vars using this pattern here.

https://github.com/movementlabsxyz/movement/blob/main/protocol-units/execution/maptos/util/src/config/common.rs

Could you use this pattern? Otherwise the implementation looks good.

0xmovses

0xmovses · 2025-04-15T14:06:43Z

Ah one conflict to resolve then this can merge in.

l-monninger · 2025-04-23T02:23:55Z

@@ -0,0 +1,26 @@
+version: "3"
+
+environment:


Previously we had a test for this as well. Is that somewhere in here? Is it added to?

https://github.com/movementlabsxyz/movement/pull/697/files#diff-57bb71e304dbeaa770299637cc3af79f7dc71aac30d1232ff25e8194af69cccb

apenzk and others added 16 commits February 11, 2025 08:04

[protocol-units] : update READMEs (#685)

5228d5d

Co-authored-by: Mikhail Zabaluev <mikhail.zabaluev@movementlabs.xyz>

[Follower ] Backup / restore follower DB (#1051)

d12f243

[Follower] : MOVEMENT_SYNC is not longer mandatory (#1053)

7666f08

[Signing] add testnet in key name environement (#1059)

e85bfbf

[Signing Change Hashicorp Vault delimiter to a - (#1061)

dcaa9ed

feat: add curl in the final container

568ee26

[Node] Add retry on DA connection (#1054)

50612de

[DA] Update light client protocol and heartbeat. (#1064)

fc5109e

Celestia mainnet config (#1038)

7b6f1f7

Co-authored-by: Liam Monninger <l.mak.monninger@gmail.com>

fix(docker): celestia-light-node fixes (#1078)

3a7f256

[Indexer] Add ans processor (#1081)

d05782b

feat: setup otlp prometheus exporter with correct invocation in node

8760186

nicholasflintwillow force-pushed the main branch from 3560d0a to c647978 Compare March 9, 2025 18:09

0xmovses mentioned this pull request Mar 18, 2025

Integrate Prometheus Exporter and add basic tracing #1126

Open

1 task

fix: moved the prometheus instance from docker container to run as na…

f54e931

…tive process using process-compose

ganymedio assigned 0xIcarus Mar 31, 2025

fix: fix prometheus unable to find endpoint to scrape on full node

609a44d

0xIcarus marked this pull request as ready for review April 1, 2025 16:51

0xIcarus requested review from 0xmovses, ganymedio, musitdev and mzabaluev as code owners April 1, 2025 16:51

mzabaluev suggested changes Apr 3, 2025

View reviewed changes

Comment thread .github/workflows/checks-all.yml Outdated

0xIcarus force-pushed the Icarus131/prometheus-otlp branch from 760da1f to bf2124e Compare April 3, 2025 15:19

0xmovses suggested changes Apr 4, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into icarus131/prometheus-…

cb806d9

…otlp

0xIcarus force-pushed the Icarus131/prometheus-otlp branch from bf2124e to cb806d9 Compare April 7, 2025 12:02

fix: prevent guard thread from crashing before proving metrics at end…

9cd83b6

…point

0xIcarus requested review from 0xmovses and ganymedio April 7, 2025 16:13

ganymedio reviewed Apr 8, 2025

View reviewed changes

ganymedio reviewed Apr 9, 2025

View reviewed changes

chore: remove extra ggp test

2fee33d

ganymedio approved these changes Apr 9, 2025

View reviewed changes

0xmovses suggested changes Apr 9, 2025

View reviewed changes

Comment thread protocol-units/bridge/contracts/minter/sources/minter.move Outdated

0xIcarus added 2 commits April 9, 2025 20:39

chore: remove spurious file and use tracing instead of eprintln

f807c01

feat: use env instead of hardcoded port and addr values

4e9dcec

0xIcarus requested review from 0xmovses and mzabaluev April 10, 2025 20:05

0xmovses approved these changes Apr 15, 2025

View reviewed changes

Merge branch 'main' into Icarus131/prometheus-otlp

62eeb54

0xmovses merged commit 236c2be into main Apr 15, 2025
8 checks passed

l-monninger reviewed Apr 23, 2025

View reviewed changes

mzabaluev mentioned this pull request Apr 23, 2025

OpenTelemetry OTLP setup for tracing, take 2 #697

Closed

Conversation

0xIcarus commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changelog

Testing

Outstanding issues

Uh oh!

0xmovses commented Apr 1, 2025

Uh oh!

Uh oh!

0xmovses left a comment

Choose a reason for hiding this comment

Uh oh!

0xmovses Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

0xmovses Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

ganymedio commented Apr 7, 2025

Uh oh!

0xIcarus commented Apr 7, 2025

Uh oh!

ganymedio commented Apr 7, 2025

Uh oh!

0xIcarus commented Apr 7, 2025

Uh oh!

ganymedio left a comment

Choose a reason for hiding this comment

Uh oh!

0xIcarus commented Apr 9, 2025

Uh oh!

ganymedio Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

ganymedio left a comment

Choose a reason for hiding this comment

Uh oh!

0xmovses commented Apr 9, 2025

Uh oh!

0xIcarus commented Apr 9, 2025

Uh oh!

0xmovses commented Apr 9, 2025

Uh oh!

0xmovses commented Apr 9, 2025

Uh oh!

0xmovses left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0xmovses left a comment

Choose a reason for hiding this comment

Uh oh!

0xmovses commented Apr 15, 2025

Uh oh!

Uh oh!

l-monninger Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

0xIcarus commented Mar 7, 2025 •

edited

Loading