publishing blockchain data

[iuwqyir]

TL;DR

Added Kafka publisher functionality to stream blockchain data to external systems.

What changed?

• Added a new Publisher module that can stream blocks, transactions, traces, and events to Kafka topics
• Implemented configuration options for the publisher in the root command flags
• Added serialization methods for blockchain data models
• Integrated the publisher with the Committer and ReorgHandler to publish data during normal processing and reorgs
• Added filtering capabilities for transactions and events based on addresses and topics

How to test?

1. Enable the publisher with the --publisher-enabled flag
2. Configure Kafka broker(s) with --publisher-brokers
3. Enable specific data types with their respective flags:
   • --publisher-blocks-enabled
   • --publisher-transactions-enabled
   • --publisher-traces-enabled
   • --publisher-events-enabled
4. Set topic names and filters as needed
5. Verify data is being published to the configured Kafka topics

Why make this change?

This change enables real-time streaming of blockchain data to external systems via Kafka, allowing for more flexible data processing pipelines and integration with other applications. The publisher can be used to build real-time analytics, monitoring systems, or to feed data into other databases or processing frameworks.

[iuwqyir]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• publishing blockchain data #176 👈 (View in Graphite)
• return block data when deleting it #175
• do not pass around pointers to slices and maps #174
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.