Skip to content

Commit 90cdd1b

Browse files
kobayuriikobayurii
committed
add migration scripts
1 parent b9bfcdd commit 90cdd1b

File tree

8 files changed

+352
-9
lines changed

8 files changed

+352
-9
lines changed

‎database/database_migrations/README.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
> **Note:** If you are starting the project from scratch (with a new, empty database), you do not need to run the migration scripts in this directory. These scripts are only necessary when migrating data from an existing database or upgrading shards. For new deployments, follow the standard database initialization procedures instead.
2+
3+
Database Migration Scripts
4+
5+
This directory contains scripts for migrating database shards and related data.
6+
7+
## Shard Migration Script
8+
9+
The `shard_migration.sh` script is the main migration orchestrator that runs multiple migration scripts in parallel.
10+
11+
### Usage
12+
13+
The script accepts the following command-line arguments:
14+
15+
- `--db_name`: Database name
16+
- `--db_user`: Database username
17+
- `--db_password`: Database password
18+
- `--host`: Database host
19+
- `--port`: Database port
20+
21+
### Examples
22+
23+
#### Basic Example:
24+
```bash
25+
./shard_migration.sh --db_name my_database --db_user postgres --db_password mypassword --host localhost --port 5432
26+
```
27+
28+
#### Local PostgreSQL Database:
29+
```bash
30+
./shard_migration.sh \
31+
--db_name read_rpc_db \
32+
--db_user postgres \
33+
--db_password secretpassword \
34+
--host localhost \
35+
--port 5432
36+
```
37+
38+
#### Remote Database:
39+
```bash
40+
./shard_migration.sh \
41+
--db_name production_db \
42+
--db_user readrpc_user \
43+
--db_password prod_password123 \
44+
--host db.example.com \
45+
--port 5432
46+
```
47+
48+
#### Using Environment Variables:
49+
```bash
50+
# Set environment variables first
51+
export DB_NAME="my_database"
52+
export DB_USER="postgres"
53+
export PGPASSWORD="mypassword"
54+
export DB_HOST="localhost"
55+
export DB_PORT="5432"
56+
57+
# Then run the script (it will use the environment variables)
58+
./shard_migration.sh
59+
```
60+
61+
### Prerequisites
62+
63+
1. **Make the script executable:**
64+
```bash
65+
chmod +x shard_migration.sh
66+
```
67+
68+
2. **Make all migration scripts executable:**
69+
```bash
70+
chmod +x migrate_*.sh
71+
```
72+
73+
3. **Ensure all required migration scripts exist:**
74+
- `migrate_access_keys.sh`
75+
- `migrate_accounts.sh`
76+
- `migrate_contracts.sh`
77+
- `migrate_state_changes.sh`
78+
79+
### How it Works
80+
81+
The `shard_migration.sh` script:
82+
83+
1. Parses command-line arguments and sets environment variables
84+
2. Creates a log file named `migration_${DB_NAME}.log`
85+
3. Runs four migration scripts in parallel using the `&` operator
86+
4. Waits for all migrations to complete using the `wait` command
87+
5. Logs start and completion times
88+
89+
### Output
90+
91+
- Migration progress and results are logged to `migration_${DB_NAME}.log`
92+
- Console output shows start and completion timestamps
93+
- Each individual migration script may produce its own output
94+
95+
### Notes
96+
97+
- All arguments are required for the script to function properly
98+
- The script runs migrations in parallel to improve performance
99+
- Make sure you have proper database permissions before running the migration
100+
- Review the individual migration scripts to understand what data will be migrated

‎database/database_migrations/migrate_access_keys.sh‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#!/bin/bash
2+
3+
# Function to migrate a single partition
4+
migrate_partition() {
5+
local partition=$1
6+
# shellcheck disable=SC2155
7+
local start_time=$(date +"%T")
8+
9+
echo "[INFO] Starting migration for partition state_changes_access_key_$partition at $start_time"
10+
echo "[INFO] Starting migration for partition state_changes_access_key_$partition at $start_time" >> "$LOG_FILE"
11+
12+
psql -U "$DB_USER" -d "$DB_NAME" -h "$DB_HOST" -p "$DB_PORT" -c "
13+
WITH ordered_data AS (
14+
SELECT
15+
account_id,
16+
data_key,
17+
data_value,
18+
block_height AS block_height_from,
19+
LAG(block_height) OVER (PARTITION BY account_id, data_key ORDER BY block_height DESC) AS block_height_to
20+
FROM state_changes_access_key_$partition
21+
)
22+
INSERT INTO state_changes_access_key_compact_$partition (account_id, data_key, data_value, block_height_from, block_height_to)
23+
SELECT
24+
account_id,
25+
data_key,
26+
data_value,
27+
block_height_from::bigint,
28+
block_height_to::bigint
29+
FROM ordered_data
30+
WHERE data_value IS NOT NULL
31+
ON CONFLICT (account_id, data_key, block_height_from) DO NOTHING;
32+
" 2>&1 | tee -a "$LOG_FILE"
33+
34+
# shellcheck disable=SC2155
35+
local end_time=$(date +"%T")
36+
echo "[INFO] Finished migration for partition state_changes_access_key_$partition at $end_time"
37+
echo "[INFO] Finished migration for partition state_changes_access_key_$partition at $end_time" >> "$LOG_FILE"
38+
}
39+
40+
# Run migrations in parallel for partitions 0 to 99
41+
for i in $(seq 0 99); do
42+
migrate_partition "$i" &
43+
done
44+
45+
# Wait for all background jobs to finish
46+
wait

‎database/database_migrations/migrate_accounts.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/bin/bash
2+
3+
# Function to migrate a single partition
4+
migrate_partition() {
5+
local partition=$1
6+
# shellcheck disable=SC2155
7+
local start_time=$(date +"%T")
8+
9+
echo "[INFO] Starting migration for partition state_changes_account_$partition at $start_time"
10+
echo "[INFO] Starting migration for partition state_changes_account_$partition at $start_time" >> "$LOG_FILE"
11+
12+
psql -U "$DB_USER" -d "$DB_NAME" -h "$DB_HOST" -p "$DB_PORT" -c "
13+
WITH ordered_data AS (
14+
SELECT
15+
account_id,
16+
data_value,
17+
block_height AS block_height_from,
18+
LAG(block_height) OVER (PARTITION BY account_id ORDER BY block_height DESC) AS block_height_to
19+
FROM state_changes_account_$partition
20+
)
21+
INSERT INTO state_changes_account_compact_$partition (account_id, data_value, block_height_from, block_height_to)
22+
SELECT
23+
account_id,
24+
data_value,
25+
block_height_from::bigint,
26+
block_height_to::bigint
27+
FROM ordered_data
28+
WHERE data_value IS NOT NULL
29+
ON CONFLICT (account_id, block_height_from) DO NOTHING;
30+
" 2>&1 | tee -a "$LOG_FILE"
31+
32+
# shellcheck disable=SC2155
33+
local end_time=$(date +"%T")
34+
echo "[INFO] Finished migration for partition state_changes_account_$partition at $end_time"
35+
echo "[INFO] Finished migration for partition state_changes_account_$partition at $end_time" >> "$LOG_FILE"
36+
}
37+
38+
# Run migrations in parallel for partitions 0 to 99
39+
for i in $(seq 0 99); do
40+
migrate_partition "$i" &
41+
done
42+
43+
# Wait for all background jobs to finish
44+
wait

‎database/database_migrations/migrate_contracts.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/bin/bash
2+
3+
# Function to migrate a single partition
4+
migrate_partition() {
5+
local partition=$1
6+
# shellcheck disable=SC2155
7+
local start_time=$(date +"%T")
8+
9+
echo "[INFO] Starting migration for partition state_changes_contract_$partition at $start_time"
10+
echo "[INFO] Starting migration for partition state_changes_contract_$partition at $start_time" >> "$LOG_FILE"
11+
12+
psql -U "$DB_USER" -d "$DB_NAME" -h "$DB_HOST" -p "$DB_PORT" -c "
13+
WITH ordered_data AS (
14+
SELECT
15+
account_id,
16+
data_value,
17+
block_height AS block_height_from,
18+
LAG(block_height) OVER (PARTITION BY account_id ORDER BY block_height DESC) AS block_height_to
19+
FROM state_changes_contract_$partition
20+
)
21+
INSERT INTO state_changes_contract_compact$partition (account_id, data_value, block_height_from, block_height_to)
22+
SELECT
23+
account_id,
24+
data_value,
25+
block_height_from::bigint,
26+
block_height_to::bigint
27+
FROM ordered_data
28+
WHERE data_value IS NOT NULL
29+
ON CONFLICT (account_id, block_height_from) DO NOTHING;
30+
" 2>&1 | tee -a "$LOG_FILE"
31+
32+
# shellcheck disable=SC2155
33+
local end_time=$(date +"%T")
34+
echo "[INFO] Finished migration for partition state_changes_contract_$partition at $end_time"
35+
echo "[INFO] Finished migration for partition state_changes_contract_$partition at $end_time" >> "$LOG_FILE"
36+
}
37+
38+
# Run migrations in parallel for partitions 0 to 99
39+
for i in $(seq 0 99); do
40+
migrate_partition "$i" &
41+
done
42+
43+
# Wait for all background jobs to finish
44+
wait

‎database/database_migrations/migrate_state_changes.sh‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#!/bin/bash
2+
3+
# Function to migrate a single partition
4+
migrate_partition() {
5+
local partition=$1
6+
# shellcheck disable=SC2155
7+
local start_time=$(date +"%T")
8+
9+
echo "[INFO] Starting migration for partition state_changes_data_$partition at $start_time"
10+
echo "[INFO] Starting migration for partition state_changes_data_$partition at $start_time" >> "$LOG_FILE"
11+
12+
psql -U "$DB_USER" -d "$DB_NAME" -h "$DB_HOST" -p "$DB_PORT" -c "
13+
WITH ordered_data AS (
14+
SELECT
15+
account_id,
16+
data_key,
17+
data_value,
18+
block_height AS block_height_from,
19+
LAG(block_height) OVER (PARTITION BY account_id, data_key ORDER BY block_height DESC) AS block_height_to
20+
FROM state_changes_data_$partition
21+
)
22+
INSERT INTO state_changes_data_compact_$partition (account_id, data_key, data_value, block_height_from, block_height_to)
23+
SELECT
24+
account_id,
25+
data_key,
26+
data_value,
27+
block_height_from::bigint,
28+
block_height_to::bigint
29+
FROM ordered_data
30+
WHERE data_value IS NOT NULL
31+
ON CONFLICT (account_id, data_key, block_height_from) DO NOTHING;
32+
" 2>&1 | tee -a "$LOG_FILE"
33+
34+
# shellcheck disable=SC2155
35+
local end_time=$(date +"%T")
36+
echo "[INFO] Finished migration for partition state_changes_data_$partition at $end_time"
37+
echo "[INFO] Finished migration for partition state_changes_data_$partition at $end_time" >> "$LOG_FILE"
38+
}
39+
40+
# Run migrations in parallel for partitions 0 to 99
41+
for i in $(seq 0 99); do
42+
migrate_partition "$i" &
43+
done
44+
45+
# Wait for all background jobs to finish
46+
wait

‎database/database_migrations/shard_migration.sh‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/bin/bash
2+
3+
# Parse arguments or use environment variables
4+
while [[ $# -gt 0 ]]; do
5+
key="$1"
6+
case $key in
7+
--db_name)
8+
export DB_NAME="$2"
9+
shift 2
10+
;;
11+
--db_user)
12+
export DB_USER="$2"
13+
shift 2
14+
;;
15+
--db_password)
16+
export PGPASSWORD="$2"
17+
shift 2
18+
;;
19+
--host)
20+
export DB_HOST="$2"
21+
shift 2
22+
;;
23+
--port)
24+
export DB_PORT="$2"
25+
shift 2
26+
;;
27+
*)
28+
echo "Unknown option: $1"
29+
exit 1
30+
;;
31+
esac
32+
done
33+
34+
# Set defaults from environment if not set by args
35+
: "${DB_NAME:=${DB_NAME}}"
36+
: "${DB_USER:=${DB_USER}}"
37+
: "${PGPASSWORD:=${PGPASSWORD}}"
38+
: "${DB_HOST:=${DB_HOST}}"
39+
: "${DB_PORT:=${DB_PORT}}"
40+
41+
# Check required variables
42+
if [[ -z "$DB_NAME" || -z "$DB_USER" || -z "$PGPASSWORD" || -z "$DB_HOST" || -z "$DB_PORT" ]]; then
43+
echo "All arguments are required: --db_name, --db_user, --db_password, --host, --port (or set corresponding env vars)"
44+
exit 1
45+
fi
46+
47+
48+
# Set log file
49+
export LOG_FILE="migration_${DB_NAME}.log"
50+
# Remove old log file if it exists
51+
rm -f "$LOG_FILE"
52+
touch "$LOG_FILE"
53+
54+
echo "Starting migration at $(date)" | tee -a "$LOG_FILE"
55+
56+
./migrate_access_keys.sh &
57+
./migrate_accounts.sh &
58+
./migrate_contracts.sh &
59+
./migrate_state_changes.sh &
60+
61+
wait
62+
63+
echo "Migration completed at $(date)" | tee -a "$LOG_FILE"

‎database/src/postgres/migrations/shard_db/20250218132509_create_state_changes_compact.up.sql‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ CREATE TABLE IF NOT EXISTS state_changes_data_compact (
22
account_id text NOT NULL,
33
data_key text NOT NULL,
44
data_value bytea NOT NULL,
5-
block_height_from numeric(20,0) NOT NULL,
6-
block_height_to numeric(20,0) NULL,
5+
block_height_from bigint NOT NULL,
6+
block_height_to bigint NULL,
77
PRIMARY KEY (account_id, data_key, block_height_from)
88
) PARTITION BY HASH (account_id);
99

@@ -22,8 +22,8 @@ CREATE TABLE IF NOT EXISTS state_changes_access_key_compact (
2222
account_id text NOT NULL,
2323
data_key text NOT NULL,
2424
data_value bytea NOT NULL,
25-
block_height_from numeric(20,0) NOT NULL,
26-
block_height_to numeric(20,0) NULL,
25+
block_height_from bigint NOT NULL,
26+
block_height_to bigint NULL,
2727
PRIMARY KEY (account_id, data_key, block_height_from)
2828
) PARTITION BY HASH (account_id);
2929

@@ -41,8 +41,8 @@ END $$;
4141
CREATE TABLE IF NOT EXISTS state_changes_contract_compact (
4242
account_id text NOT NULL,
4343
data_value bytea NOT NULL,
44-
block_height_from numeric(20,0) NOT NULL,
45-
block_height_to numeric(20,0) NULL,
44+
block_height_from bigint NOT NULL,
45+
block_height_to bigint NULL,
4646
PRIMARY KEY (account_id, block_height_from)
4747
) PARTITION BY HASH (account_id);
4848

@@ -59,8 +59,8 @@ END $$;
5959
CREATE TABLE IF NOT EXISTS state_changes_account_compact (
6060
account_id text NOT NULL,
6161
data_value bytea NULL,
62-
block_height_from numeric(20,0) NOT NULL,
63-
block_height_to numeric(20,0) NULL,
62+
block_height_from bigint NOT NULL,
63+
block_height_to bigint NULL,
6464
PRIMARY KEY (account_id, block_height_from)
6565
) PARTITION BY HASH (account_id);
6666

‎database/src/postgres/migrations/shard_db/20250218132510_create_get_text_partition_function.down.sql‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
DROP FUNCTION IF EXISTS get_text_partition(TEXT, INT);
1+
DROP FUNCTION IF EXISTS get_text_partition(TEXT, INT);

0 commit comments

Comments
 (0)