Skip to content

regular node mode #399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: develop
Choose a base branch
from
Draft

regular node mode #399

wants to merge 10 commits into from

Conversation

kobayurii
Copy link
Member

@kobayurii kobayurii commented Apr 28, 2025

This pull request introduces significant changes to enhance the database's handling of state changes by implementing data range partitioning, adding configuration options for archival mode and data ranges, and updating database schema and logic accordingly. These changes improve scalability, allow for more efficient data storage, and provide flexibility in managing historical data.

Configuration Enhancements:

  • Added keep_data_ranges_number and archival_mode fields to GeneralRpcServerConfig and CommonGeneralRpcServerConfig to allow configuration of data ranges and archival mode. Default values and validation logic were also implemented. [1] [2] [3] [4] [5]
  • Updated default_env_configs.rs to include environment variables for KEEP_DATA_RANGES_NUMBER and ARCHIVAL_MODE.

Utility Functions:

  • Added get_data_range_id, get_earliest_available_block_height, and check_block_height_availability utility functions in utils.rs to calculate data range IDs, determine earliest available blocks, and validate block height availability.

Database Schema and Partitioning:

  • Introduced a new procedure create_new_range_tables to dynamically create partitioned tables for state changes based on data range IDs.
  • Dropped the unused block_hash column from state changes tables to simplify the schema.

Database Logic Updates:

  • Updated StateIndexerDbManager to include a method for creating new range tables and removed the block_hash parameter from state change methods.
  • Modified PostgresDBManager to calculate data_range_id dynamically and use it in SQL queries to access partitioned tables. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Documentation and Comments:

  • Added detailed comments to explain the logic behind data range partitioning, including examples of how data_range_id is calculated and used.

@kobayurii kobayurii requested a review from khorolets April 28, 2025 11:39
@kobayurii kobayurii self-assigned this Apr 28, 2025
@kobayurii kobayurii force-pushed the regular_node_mode/rpc_database branch from 4311670 to 9f46355 Compare May 20, 2025 14:12
@kobayurii kobayurii requested a review from khorolets May 26, 2025 07:24
Copy link
Member

@khorolets khorolets left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really good job, I've left a few comments mostly about naming to polish. Overall, looks great!

/// 2. 261 * 5 = 1305
///
/// Output: 1305
pub async fn get_data_range_id(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider get_data_range_suffix, this is not a strong opinion just an idea

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep the name get_data_range_id because the usage of this function is not limited to the database crate.

@kobayurii kobayurii requested a review from khorolets May 26, 2025 10:19
@kobayurii kobayurii force-pushed the regular_node_mode/rpc_database branch 2 times, most recently from e724f0d to 21350a1 Compare May 29, 2025 08:30
@kobayurii kobayurii force-pushed the regular_node_mode/rpc_database branch from 7812fd3 to 9d66715 Compare May 29, 2025 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants