Skip to content

Conversation

jadepark-dev
Copy link
Collaborator

@jadepark-dev jadepark-dev commented Sep 26, 2025

This PR adds persistent database store in TON logpoller, improves query interface, and log ingestion pipeline

Primary Changes

  • Added PostgreSQL persistent store with advanced query capabilities including byte-level and bit-level filtering
  • Implemented stream-based event processing pipeline with concurrent transaction loading and log processing
  • Enhanced query interface with chainlink-common integration for standardized pagination and sorting
  • Refactored architecture from synchronous to asynchronous processing with channel-based data flow

Database & Storage

  • Created PostgreSQL schema for filters and logs with proper indexing and constraints
  • Added transaction support with configurable batch insertion and retry logic for timeout handling
  • Implemented cursor-based pagination with address+msgLT composite cursors for efficient large dataset traversal
  • Enhanced in-memory store compatibility to maintain test infrastructure during migration

Query System Improvements

  • Replaced monolithic query builder with modular system supporting field, byte, and bit-level filtering
  • Added TON-specific BOC data filtering with proper header offset handling for PostgreSQL alignment
  • Implemented typed log decoding helpers with TLB parsing and application-level filtering support

Others

  • Implemented memory-efficient log batching with configurable thresholds and database transaction support
  • Extended configuration with database-specific settings while maintaining backward compatibility
  • Enhanced service initialization with proper dependency injection and lifecycle management
  • Added validation and default value application for missing configuration fields

Next Steps

  • e2e behavior validation with core TON2EVM and EVM2TON messaging test
  • add TON schema migration file in core
  • initialize postgreSQL database when relayer is started, replacing memory store

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant