feat: implement partition-based verification filtering (PE-8163)

[djwhitt]

Summary

This PR implements partition-based verification filtering to reduce verification workload while ensuring network-wide coverage of all data.

Changes

Configuration

• Add VERIFICATION_PARTITION_COUNT (default: 64) - number of partitions to divide ID space
• Add VERIFICATION_PARTITION_THRESHOLD (default: 70) - priority threshold for partition filtering

Partition Assignment

• Each node is assigned to 1 of 64 partitions at startup
• Assignment based on:
  • AR_IO_WALLET address (if configured) - deterministic
  • Random bytes (if no wallet) - random distribution
• Partition assignment is logged at startup

Verification Filtering

• IDs with priority < 70 are filtered by partition
• High-priority data (priority ≥ 70) bypasses filtering and is verified by all nodes
• Filtering happens in the data verification worker after retrieving verifiable IDs
• Skipped IDs have their retry count incremented to ensure eventual verification

Implementation Details

• Created src/lib/verification-partition.ts with partition calculation functions
• Updated data verification worker to apply partition filtering
• No database changes required - filtering happens at application level

Benefits

• Reduces verification workload by ~98.4% (63/64) for low-priority data
• Ensures high-priority ArNS data is still verified by all nodes
• Deterministic partitioning for nodes with wallets
• Fair distribution through retry count increments

Testing

• Verify partition assignment at startup
• Confirm high-priority data bypasses filtering
• Check retry count increments for skipped IDs
• Test with different partition counts
• Verify deterministic behavior with wallets

Related

• Jira: PE-8163
• Builds on PR feat: implement demand-driven opt-out background verification for ArNS data (PE-8163) #447 (demand-driven verification for ArNS data)

🤖 Generated with Claude Code