feat(coprocessor): add transaction-based performance metrics by goshawk-3 · Pull Request #919 · zama-ai/fhevm

goshawk-3 · 2025-09-29T08:25:19Z

TLDR:

Adds a transaction_id to all major OpenTelemetry spans across all services, enabling the collection of per-operation and per-service timing data for each L1 transaction.
Introduces a new histogram metric (coprocessor_l1_txn_latency_seconds) to track latency of L1 transactions (mint, transfer, etc ..)
Introduces a new histogram metric (coprocessor_zkproof_txn_latency_seconds) to track latency of zkproof transactions

Implementation details:

A new table transactions stores both created_at and completed_at timestamp of a single (L1 or L2) transaction:
Tables (allowed_handles, pbs_computations, verify_proofs and ciphertext_digest) include a new column - transaction_id
Both listeners (gw-listener and host-listener) register the transaction arrival time
Each worker and sender can read the transaction_id from any of PG tables to populate the OTEL spans.
The transaction_sender register the time of completion of a transaction

Copilot

Pull Request Overview

This PR adds transaction-based performance metrics to the coprocessor by tracking transaction latencies across all services. It introduces telemetry capabilities to collect per-operation and per-service timing data for L1 and zkproof transactions.

Key changes:

Added a new transactions table with created_at and completed_at timestamps
Added transaction_id columns to all major database tables (allowed_handles, pbs_computations, verify_proofs, ciphertext_digest)
Introduced two new histogram metrics: coprocessor_l1_txn_latency_seconds and coprocessor_zkproof_txn_latency_seconds

Reviewed Changes

Copilot reviewed 20 out of 127 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
coprocessor/fhevm-engine/zkproof-worker/src/verifier.rs	Updated telemetry tracers to include transaction_id parameter and improved span handling
coprocessor/fhevm-engine/transaction-sender/src/ops/verify_proof.rs	Added transaction_id parameter to proof processing and integrated telemetry
coprocessor/fhevm-engine/transaction-sender/src/ops/allow_handle.rs	Added transaction_id parameter to ACL operations and telemetry integration
coprocessor/fhevm-engine/transaction-sender/src/ops/add_ciphertext.rs	Added transaction_id parameter to ciphertext operations and telemetry integration
coprocessor/fhevm-engine/tfhe-worker/src/tfhe_worker.rs	Added transaction ID tracking to worker operations
coprocessor/fhevm-engine/sns-worker/src/lib.rs	Added transaction_id field to HandleItem and database operations
coprocessor/fhevm-engine/host-listener/src/database/tfhe_event_propagate.rs	Added transaction tracking for TFHE and ACL events
coprocessor/fhevm-engine/gw-listener/src/gw_listener.rs	Added transaction tracking for gateway listener events
coprocessor/fhevm-engine/fhevm-engine-common/src/telemetry.rs	Added comprehensive transaction metrics system with histogram tracking
coprocessor/fhevm-engine/db-migration/migrations/20250929064611_create_transactions_table.sql	Database migration to add transaction tracking infrastructure

Files not reviewed (98)

coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/fhevm-engine-common/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-b7d5ed966527dfc500ce529e0249d96c058a06c18a02ed117ad2f4140fbc470f.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-b801404dd6465cc942d1f953f7aa53eece85e4302cef55f50096fa0b25ab7a50.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/gw-listener/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-27bd6a583f0dcd189cc54e3ba1e3d6ddc03a34ba8a03e20d5c1585d1b6b73dad.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-9fab4c1f3ae480162f9b1ef05988c56eedd7fb9c91f2a37a10fd46b4ee505cfa.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/host-listener/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/scheduler/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-09f4bb2ed7df17aa0305c7fb88e87473e5be704202eb8f2fa571d47a4f98b7af.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-556dbda2415973bd6d94b8455eeae8cb8e4e177cd72376a7acad880eef60e419.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-5631176c0419193e1007788434d8786ea0872b263ad07a2c7c205a8b04d041b8.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-92524ec746c5f2a7dd986cacd005d98d68a8fe112f1e8310c3a78e4e208dcd39.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-a3581b82aa78344b06e4270d0aec5ac76c2d0fa1661c1502600852450d92fe8a.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-ad702e3febf9f7dea99259e19f36b7da23fc42f7363e1f9f29d15f7d94e8fe56.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-b7d5ed966527dfc500ce529e0249d96c058a06c18a02ed117ad2f4140fbc470f.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-daf7efed8ec9936d23a33d89254b8a7b9f8efeaa9ecd266e573ac435330b4d45.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-dc8f55b8896fff4c9a9bd9e3a0ad4eed5e6993ff334807db4ab76651a0697722.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-ef4d3409ed091acbd382ca0959a435c450f2d6c16fabbbdb2ec6dabb1354dbd0.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-f77c171e5206860291e8d884535b3bf1989a40fb31f8df696e46da68ed6b8231.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/sns-worker/.sqlx/query-fd37e7dc679caa21ba22d7724a27409c232cceb719aa41a71e7faf44d2cb8ef9.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-0b85af1e88f24290121400feb960ef80ce040e2b877b259da17188668e6c404a.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-266ee98d9dc072298b3800c73d069bffc31bf34cf379e1295fab32ed2714fa57.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-277708f2a9e2b80977cb5b7eb4bd50820b59d46bef0a3456a83144d538521192.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-27bd6a583f0dcd189cc54e3ba1e3d6ddc03a34ba8a03e20d5c1585d1b6b73dad.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-3719428499a837d1b301d0d58a21d19dd6259f21f499d8a93c0ba55e90603441.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-416ef65e70058585ce4cec14ef80330cd688076d02e375486d7ab07fab628280.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-5061d9ee69dd4a22aa118bd4d1ca26b4a02f9f690def1c3f08c34bf67ec95f8c.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-641036eba016313ea7cf191d71f2b69c1def70ea46139dd02fb510581b6322c2.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-648e618238efe4b918570e78a235947075cc241f3af841e4d34fba3587268570.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-69eea32069c04ab351856d18f4b53d059e01468778eea9588aff3f8aee6171a7.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-78a2670fabfffa3f6fd187cd0288b86b13b401468cabe15089d75a08c5d919c0.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-82afc3c943c43f1f76e150c5d3354bef4482c3da82403ad6c03add0840e71966.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-9fab4c1f3ae480162f9b1ef05988c56eedd7fb9c91f2a37a10fd46b4ee505cfa.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-c0e1b25662f8d9b3ec8813c028e0de5388e5094d8c316b29e6762bc2783939d2.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-d12da013cb1d11f23595b7d4f6e4d79f886ec342b77ab4166b6d1a8febd62f1a.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-eda7c952325475562fbe1d1a5793ac82366742c1618f83dfd6b4da5db9492544.json: Language not supported
coprocessor/fhevm-engine/stress-test-generator/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/test-harness/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/tfhe-worker/.sqlx/query-416ef65e70058585ce4cec14ef80330cd688076d02e375486d7ab07fab628280.json: Language not supported
coprocessor/fhevm-engine/tfhe-worker/.sqlx/query-69eea32069c04ab351856d18f4b53d059e01468778eea9588aff3f8aee6171a7.json: Language not supported
coprocessor/fhevm-engine/tfhe-worker/.sqlx/query-78a2670fabfffa3f6fd187cd0288b86b13b401468cabe15089d75a08c5d919c0.json: Language not supported
coprocessor/fhevm-engine/tfhe-worker/.sqlx/query-d090bb03d3a480f8c37b45498c5cc85742b3509cef16429687f65cc2335b422f.json: Language not supported
coprocessor/fhevm-engine/tfhe-worker/.sqlx/query-e4260b9ba59d978d5504b787a0002c6eec8c6b95b2cb8b5da83f8885fb273c39.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-048212909e0bbe46633e404235d2c5cffb5284903adb757b4fda59b7fbe81d57.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-10c2430d797e96b0b2a7a0f314b33b8b10e9cc8fd0422de69916674216b42f30.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-9c32675069536c1825f8e161677a3d1c443a66514312fa099d0818cbbcfdf400.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-a74057b8122baa26cff2f59a9e0e7403bb1abe1881fd3da8757e192da9f84180.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-b5b633e5812b7396037e2ab0a1db9a1d753b8650ed3367681ba30ed426799502.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-be2b163e885ff2e4df27ae07c51f8c304f534b50565504a96bd63ce63a6179d7.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-eadec222d0154713dc15ea7ba1e113ae7838d935e4462421fd796f5f7986dbbd.json: Language not supported
coprocessor/fhevm-engine/transaction-sender/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-455bd359a58df1cef6d001eeb2e70381328eabdfbd9d5ba39401c634d5403b79.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-45f9a96fb7f0e31ee8f7d316418de59d65d1f9be75c21825f4c07a7f56e5ae4a.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-66fcc6dfb88db7c48ea1cc752e61fc1aefb776aa112b632cd0383144c730e7f8.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-cbf71c3aa66e532d73d0d53c71f0fdc94508cdc26ec474f4d06ee9b64173ea72.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-cf21e76845fc93ff777bc3eb78d577dcdfaab362d08da21a616cdcd8139991a0.json: Language not supported
coprocessor/fhevm-engine/zkproof-worker/.sqlx/query-fd1604ca19ddd4ebb61b085800bf355b6812d8aa8cc254c9e0b27c780462f9e9.json: Language not supported

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

rudy-6-4

Not sure about GC transactions table performance. Could be too slow if called too often.

…transaction latency This introduces histogram metrics to measure L1/L2 transactions latency, by recording the created_at and completed_at timings of a transaction. To support this, a new PG table - transactions has been added.

…attr, host-listener

cla-bot Bot added the cla-signed label Sep 29, 2025

goshawk-3 mentioned this pull request Sep 30, 2025

chore(coprocessor): add a transaction-based performance metric #898

Closed

goshawk-3 changed the title ~~feat(coprocessor): add a set of common APIs to be used for recording …~~ feat(coprocessor): add transaction-based performance metrics Sep 30, 2025

goshawk-3 force-pushed the georgi/metrics/latency branch 6 times, most recently from 7ec969a to ffd3b7e Compare October 1, 2025 10:21

goshawk-3 requested a review from Copilot October 1, 2025 11:07

Copilot AI reviewed Oct 1, 2025

View reviewed changes

Comment thread coprocessor/fhevm-engine/fhevm-engine-common/src/telemetry.rs

Comment thread coprocessor/fhevm-engine/fhevm-engine-common/src/telemetry.rs

goshawk-3 marked this pull request as ready for review October 1, 2025 11:32

goshawk-3 requested review from antoniupop, dartdart26 and rudy-6-4 October 1, 2025 11:32