Harfork Pipeline optimization: Verifications

[dkijania]

Hardfork Verification Script Performance Analysis

Executive Summary

Total Execution Time: 25.9 minutes (1553 seconds)

Script: ./scripts/hardfork/mina-verify-packaged-fork-config

Log: ~/Downloads/hardfork-package-generation-new_build_119_verify-packaged-artifacts.log

---

Time Breakdown by Category

Category	Time	Percentage	Description
Genesis Generation	9.1 min	35.0%	Creating legacy and new genesis ledgers
Daemon Operations	9.0 min	34.7%	Starting daemon 3 times to export ledgers
RocksDB Scanning	5.0 min	19.4%	Scanning 9 RocksDB instances for verification
Verification	1.4 min	5.4%	Tarball extraction and curl downloads
Setup	1.0 min	3.7%	Script initialization

---

Detailed Operation Breakdown

1. Setup Phase (0.96 min)

• Script initialization and environment setup
• Git operations, submodule setup

2. Genesis Ledger Generation (9.07 min - 35%)

• Legacy genesis: 4.54 min (272.6s)
  • Generates 3 ledgers (genesis + 2 epoch)
  • Each ledger takes ~8-27 seconds to create and tar
• New genesis: 4.53 min (271.6s)
  • Same process with new format
  • Similar per-ledger timings

3. Daemon Extraction (8.98 min - 35%)

Three sequential daemon startups to export ledgers:

• Daemon implement miner #1 (Download from S3): 3.13 min
  • Starts daemon, downloads ledgers from S3
  • Waits for daemon ready (~1 minute)
  • Exports 3 ledgers (staged, staking, next)
• Daemon implement blockchain accumulator #2 (Reference): 2.28 min
  • Shorter because ledgers already generated locally
• Daemon add some pedersen modules, change mining loop #3 (Packaged): 3.57 min
  • Longest due to loading from /var/lib/coda

4. RocksDB Verification (5.02 min - 19%)

For each of 3 ledgers, scans 3 copies (packaged, web, generated):

• Ledger 1 (epoch): 1.96 min (3 scans × ~30-45s)
• Ledger 2 (epoch): 1.87 min (3 scans × ~25-45s)
• Ledger 3 (genesis): 1.19 min (3 scans × ~22-25s)

5. Download & Extract (1.39 min - 5%)

• Curl downloads from S3: ~83 seconds total
• Tarball extraction: ~10 seconds total

---

Critical Performance Bottlenecks

🔴 Top 3 Issues:

1. Genesis ledgers generated twice (9.1 min)

   • Legacy format: 4.5 min
   • New format: 4.5 min
   • Both create the same 3 ledgers

2. Sequential daemon startups (9.0 min)

   • 3 separate daemon instances started sequentially
   • Each waits ~1 minute for daemon ready
   • Total waiting: ~3 minutes

3. Sequential RocksDB scans (5.0 min)

   • 9 scans executed sequentially
   • Each scan is independent and could run in parallel

---

Optimization Proposals

🚀 HIGH IMPACT (Potential savings: 10-15 minutes)

1. Parallelize RocksDB Scans (Save: ~4 minutes)

Current: 9 sequential scans taking 5 minutes
Proposed: Run 3 scans in parallel per ledger

# Instead of:
for kind in {packaged,web,generated}; do
    $MINA_ROCKSDB_SCANNER --db-path "$tardir/$kind" --output-file "$WORKDIR/$kind.scan"
done

# Do:
$MINA_ROCKSDB_SCANNER --db-path "$tardir/packaged" --output-file "$WORKDIR/packaged.scan" &
$MINA_ROCKSDB_SCANNER --db-path "$tardir/web" --output-file "$WORKDIR/web.scan" &
$MINA_ROCKSDB_SCANNER --db-path "$tardir/generated" --output-file "$WORKDIR/generated.scan" &
wait

Expected time: 5 min → 1.5 min (save ~3.5 min)

2. Skip Legacy Genesis Generation (Save: ~4.5 minutes)

Current: Generates both legacy and new format
Proposed: Only generate new format, skip legacy if not needed for validation

The script generates legacy ledgers but only uses them to validate hashes. If the fork config is already validated, this step can be skipped or cached.

# Add flag to skip legacy generation
if [ -z "${SKIP_LEGACY_VALIDATION}" ]; then
    "$MINA_LEGACY_GENESIS_EXE" --config-file "$FORK_CONFIG" ...
fi

Expected time: 9.1 min → 4.5 min (save ~4.5 min)

3. Parallelize Curl Downloads (Save: ~1 minute)

Current: 3 sequential downloads (34s + 11s + 25s = 70s)
Proposed: Download all 3 tarballs in parallel before processing

# Download all first
for tarname in "${tarnames[@]}"; do
    curl "$base_s3_url/$tarname.tar.gz" -o "/tmp/$tarname.tar.gz" &
done
wait

# Then extract
for tarname in "${tarnames[@]}"; do
    tar -xzf "/tmp/$tarname.tar.gz" -C "$tardir/web"
done

Expected time: 70s → 34s (save ~36s)

4. Reuse Single Daemon Instance (Save: ~6 minutes)

Current: Starts 3 separate daemon instances
Proposed: Start daemon once, export all needed ledgers by switching configs

This is complex but could save significant time by eliminating 2 daemon startups.

Expected time: 9 min → 3 min (save ~6 min)

---

🔶 MEDIUM IMPACT (Potential savings: 2-4 minutes)

5. Cache Generated Ledgers

Check if ledgers already exist with correct hash before regenerating:

if [ -f "$WORKDIR/ledgers/genesis_ledger_$HASH.tar.gz" ]; then
    echo "Using cached ledger"
else
    "$MINA_GENESIS_EXE" --config-file "$FORK_CONFIG" ...
fi

6. Optimize Daemon Startup

• Use --insecure-rest-server to skip unnecessary security checks
• Reduce --log-level to Error instead of Info
• Skip unnecessary initialization steps

mina daemon --libp2p-keypair "$WORKDIR/keys/p2p" \
    --config-file "$config_file" \
    --seed \
    --insecure-rest-server \
    --log-level Error \
    --genesis-ledger-dir "$ledger_dir"

---

🟢 LOW IMPACT (Potential savings: <1 minute)

7. Stream curl → tar

Instead of downloading then extracting, pipe directly:

curl "$base_s3_url/$tarname.tar.gz" | tar -xz -C "$tardir/web"

This is already done in the script, so no change needed.

8. Use tmpfs for Working Directory

If RAM is available, use tmpfs for intermediate files:

WORKDIR=/dev/shm/verification

This could speed up I/O operations slightly.

---

Implementation Priority

Phase 1: Quick Wins (2-3 hours implementation)

1. ✅ Parallelize RocksDB scans (save ~3.5 min)
2. ✅ Parallelize curl downloads (save ~1 min)
3. ✅ Skip legacy generation with flag (save ~4.5 min)

Total savings: ~9 minutes (26 min → 17 min)

Phase 2: Moderate Effort (1-2 days implementation)

4. ✅ Cache generated ledgers (save ~2 min)
5. ✅ Optimize daemon startup (save ~1 min)

Total savings: ~12 minutes (26 min → 14 min)

Phase 3: Complex Refactor (3-5 days implementation)

6. ✅ Reuse single daemon instance (save ~6 min)

Total savings: ~18 minutes (26 min → 8 min)

---

Recommended Action Plan

Immediate (This Week)

Implement Phase 1 optimizations:

• Parallelize RocksDB scans
• Add SKIP_LEGACY_VALIDATION flag
• Parallelize downloads

Result: 26 min → 17 min (35% reduction)

Short-term (Next Sprint)

Implement Phase 2 optimizations:

• Add ledger caching
• Optimize daemon flags

Result: 26 min → 14 min (46% reduction)

Long-term (Future Consideration)

If verification time is still critical:

• Refactor to reuse daemon instance
• Consider splitting verification into parallel jobs

Result: 26 min → 8 min (69% reduction)

---

Risk Assessment

Optimization	Risk Level	Notes
Parallel RocksDB	Low	Independent operations, safe to parallelize
Skip legacy	Medium	Ensure validation coverage is maintained
Parallel downloads	Low	S3 supports concurrent connections
Reuse daemon	High	Complex state management, needs thorough testing
Cache ledgers	Medium	Must validate hash matches
Daemon flags	Low	May reduce debug info but safe

---

Appendix: Raw Timing Data

Script initialization:                  57.6s
Legacy genesis generation:             272.6s
New genesis generation:                271.6s
Daemon #1 (download):                  187.9s
Daemon #2 (reference):                 136.7s
Daemon #3 (packaged):                  213.9s
Verification + RocksDB:                386.5s
Total:                                1552.9s (25.9 min)

Questions for Clarification

1. Is legacy genesis validation strictly required, or can it be made optional?
2. What are the resource constraints (CPU cores, RAM) on the CI system?
3. Are there any requirements for maintaining specific log output format?
4. Can we use additional build artifacts (e.g., pre-generated ledgers) in CI cache?

[glyh]

The points marked with high impact are low hanging fruit, probably take 10min each to implement.As a start we could try to do that.