An open-source extension of near-lake-framework-rs for sharded archival nodes with an AI query optimizer.
This repository is the working proof-of-concept submitted alongside the ShardLake Toolkit grant proposal to the NEAR Foundation.
From the NEAR Ecosystem 2025 Year in Review:
"Sharded archival nodes represent one of the most significant infrastructure improvements in 2025. By allowing archival operators to store and serve data for individual shards rather than the entire chain, NEAR dramatically reduces the hardware requirements for running archival infrastructure — making it feasible for a much broader set of participants to contribute to the network's data availability layer."
ShardLake Toolkit builds on this foundation by providing a developer-friendly Rust library that:
- Filters NEAR Lake streams by shard — only the shards you need ever touch your memory or disk.
- Pre-indexes data for AI-agent queries — bloom-filter + inverted-index engine that delivers 4–11× faster pattern matching than a vanilla full-scan.
- Ships a ready-to-use CLI for quick experimentation and benchmarking.
| Feature | Description |
|---|---|
| 🧩 Shard-filtered streaming | Stream only selected shards from NEAR Lake, cutting resource usage proportionally |
| 🤖 AI Query Optimizer | Pre-indexed search for method names (ft_transfer, nft_mint), event logs, and AI-agent intent patterns |
| ⏱️ Temporal queries | Restrict any query to a block-height window |
| 🔍 Regex support | ft_transfer|nft_mint — combine patterns in a single pass |
| 📊 Built-in benchmarks | Every CLI run shows optimizer vs. vanilla speedup |
| 🐳 Docker image | One command to build and run |
git clone https://github.com/Fosurero/ShardLake-Toolkit.git
cd ShardLake-Toolkit
cargo build --releasecargo install --git https://github.com/Fosurero/ShardLake-Toolkit.git# Run the default demo: shards 0,3 • 1 000 blocks • query "intent"
cargo run -- run --network testnet --start-block 100000000 --shards 0,3 --query "intent" ╔═══════════════════════════════════════════════════════╗
║ 🔷 ShardLake Toolkit v0.1.0 ║
╚═══════════════════════════════════════════════════════╝
Network: testnet
Start Block: 100,000,000
Shards: [0, 3]
Query: "intent"
Mode: Demo (1,000 simulated blocks per shard)
⏳ Generating demo data … done (14.2ms)
⏳ Building optimizer index … done (11.8ms, 12,847 entries)
📊 Results
┌─────────┬────────────────┬──────────┬──────────────┐
│ Shard │ Blocks │ Matches │ Query Time │
├─────────┼────────────────┼──────────┼──────────────┤
│ 0 │ 1,000 │ 168 │ 0.4ms │
│ 3 │ 1,000 │ 152 │ 0.4ms │
├─────────┼────────────────┼──────────┼──────────────┤
│ Total │ 2,000 │ 320 │ 0.8ms │
└─────────┴────────────────┴──────────┴──────────────┘
⚡ ShardLake Optimizer: 0.82ms (filtered 2 of 4 shards)
🐌 Vanilla full scan: 3.91ms (all 4 shards, no index)
🚀 Speedup: 4.8× faster
🔍 Match breakdown:
intent 192 hits
event 88 hits
method 40 hits
📋 Sample matches (first 5):
Block 100,000,042 │ Shard 0 │ [intent] intent:swap NEAR->USDT amount=50
Block 100,000,107 │ Shard 3 │ [intent] intent:bridge ETH->NEAR
Block 100,000,218 │ Shard 0 │ [intent] intent:delegate stake=100
Block 100,000,319 │ Shard 3 │ [event] EVENT_JSON:{"standard":"nep141"...
Block 100,000,455 │ Shard 0 │ [intent] intent:query price_feed NEAR/USD
✅ Complete. 320 events matched query "intent" across 2 shards in 42.1ms.
Indexed 2,000 blocks, 7,412 receipts, 10 unique methods.
# Token transfers only
cargo run -- run --query "ft_transfer" --num-blocks 5000
# Regex: transfers OR mints, all shards
cargo run -- run --query "ft_transfer|nft_mint" --shards 0,1,2,3
# Run the library examples
cargo run --example sharded_archival_streamer
cargo run --example ai_query_demoShardLake-Toolkit/
├── Cargo.toml # Dependencies & build config
├── src/
│ ├── lib.rs # Core types & module re-exports
│ ├── streamer.rs # Shard-filtered NEAR Lake streamer
│ ├── optimizer.rs # AI query optimizer (index + bloom)
│ └── main.rs # CLI binary (clap)
├── examples/
│ ├── sharded_archival_streamer.rs # Streamer example
│ └── ai_query_demo.rs # Optimizer example
├── benches/
│ └── optimizer_bench.rs # Criterion benchmarks
├── docs/
│ └── usage.md # Detailed usage guide
├── benchmarks/
│ └── results.md # Performance results
├── .github/workflows/
│ └── ci.yml # GitHub Actions CI
├── Dockerfile # Multi-stage Docker build
├── milestones.md # M1 / M2 / M3 roadmap
├── LICENSE # MIT
└── README.md # ← You are here
┌─────────────────────────────────────┐
│ NEAR Lake S3 Bucket │
│ (testnet / mainnet block data) │
└──────────────┬──────────────────────┘
│
┌──────────────▼──────────────────────┐
│ near-lake-framework-rs │
│ (StreamerMessage per block) │
└──────────────┬──────────────────────┘
│
┌────────────────────▼────────────────────────┐
│ ShardLake Toolkit │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ ShardedStreamer │ │ QueryOptimizer │ │
│ │ │ │ │ │
│ │ • shard filter │ │ • inverted index │ │
│ │ • config build │ │ • bloom filter │ │
│ │ • demo gen │ │ • regex engine │ │
│ │ • type convert │ │ • temporal query │ │
│ └────────┬────────┘ └────────┬─────────┘ │
│ │ │ │
│ └────────┬───────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ CLI / API │ │
│ │ (shardlake) │ │
│ └────────────────┘ │
└─────────────────────────────────────────────┘
cargo test # All tests
cargo test -- --nocapture # With stdout
cargo test optimizer # Only optimizer testscargo bench # Criterion benchmarksSee benchmarks/results.md for detailed numbers.
docker build -t shardlake-toolkit .
docker run --rm shardlake-toolkit
docker run --rm shardlake-toolkit run --query "ft_transfer" --num-blocks 2000| Milestone | Target | Status |
|---|---|---|
| M1 Core PoC | Feb 2026 | ✅ Complete |
| M2 Live NEAR Lake integration | Mar 2026 | 🔲 Planned |
| M3 Production & AI-agent SDK | Apr–May 2026 | 🔲 Planned |
See milestones.md for full details.
This PoC accompanies the ShardLake Toolkit grant proposal submitted to the NEAR Foundation Infrastructure Committee.
📄 Proposal link: ShardLake Toolkit – NEAR Grant Proposal
Contributions are welcome! Please open an issue first to discuss what you'd like to change.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-thing) - Commit your changes (
git commit -m 'Add amazing thing') - Push to the branch (
git push origin feature/amazing-thing) - Open a Pull Request
MIT © 2026 ShardLake Contributors