Zeam Lean API Implementation Plan

[bomanaps]

TL;DR

We need to implement 3 lean API endpoints for the visualization tool. The plan is to have:

• New Package: pkgs/api/ with swappable storage interface
• Phase 1: RAM storage (in-memory HashMap)
• Key Requirement: Easy to swap storage backends later

Decision Needed: Finalize technical details

Overview

Implementation plan for lean API endpoints to support the new visualization tool:

• GET /lean/v0/blocks/{block_id}
• GET /lean/v0/headers/{block_id}
• GET /lean/v0/node/version

Team ( Guillaume ) Requirements

• Swappable Storage Backend: Easy to swap between RAM, disk, or other storage
• Start with RAM Store: In-memory implementation for first version
• New Subsystem: Built from ground up with clean architecture

Current Architecture Analysis

What We Have

• HTTP Server Pattern: Existing metrics_server.zig using std.http.Server
• Block Metadata: ProtoArray stores block headers and fork choice data
• Chain State: BeamChain manages fork choice and state transitions
• Threading: Background HTTP server pattern already established

Current Gaps ( I can Confirm to this)

• No Full Block Storage: Only metadata (ProtoBlocks) persisted, full SignedBeamBlock discarded after processing
• No Database Layer: Everything in-memory, LevelDB is empty placeholder
• No Block ID Resolution: Need to support slots, roots, and keywords (head, finalized, genesis)

Proposed Architecture: Swappable Storage Design

1. Package Structure with Storage Abstraction

pkgs/api/
├── src/
│   ├── lib.zig              # Main LeanApiServer
│   ├── storage/
│   │   ├── interface.zig    # Storage trait/interface
│   │   ├── ram_store.zig    # In-memory implementation
│   │   └── future_stores.zig# Placeholder for disk/db stores
│   ├── lean/
│   │   ├── blocks.zig       # GET /lean/v0/blocks/{block_id}
│   │   ├── headers.zig      # GET /lean/v0/headers/{block_id}
│   │   └── node.zig         # GET /lean/v0/node/version
│   ├── types.zig            # API response types
│   └── utils.zig            # Block ID parsing & resolution

2. Storage Interface Design Team Requirement

Storage Trait Pattern

// storage/interface.zig
pub const BlockStorage = struct {
    ptr: *anyopaque,
    vtable: *const VTable,

    pub const VTable = struct {
        store_block: *const fn(ptr: *anyopaque, root: Root, block: SignedBeamBlock) anyerror!void,
        get_block: *const fn(ptr: *anyopaque, root: Root) ?SignedBeamBlock,
        get_block_by_slot: *const fn(ptr: *anyopaque, slot: Slot) ?SignedBeamBlock,
        has_block: *const fn(ptr: *anyopaque, root: Root) bool,
        get_head: *const fn(ptr: *anyopaque) ?SignedBeamBlock,
        get_finalized: *const fn(ptr: *anyopaque) ?SignedBeamBlock,
        get_genesis: *const fn(ptr: *anyopaque) ?SignedBeamBlock,
    };
};

RAM Store Implementation (Phase 1)

// storage/ram_store.zig  
pub const RamBlockStore = struct {
    blocks: HashMap(Root, SignedBeamBlock),
    slotToRoot: HashMap(Slot, Root),
    head_root: ?Root = null,
    finalized_root: ?Root = null,
    genesis_root: ?Root = null,
    
    pub fn getStorage(self: *Self) BlockStorage {
        return BlockStorage{
            .ptr = self,
            .vtable = &vtable,
        };
    }
};

Future Store Placeholders (Easy Swapping)

// storage/future_stores.zig
pub const DiskBlockStore = struct { /* TODO */ };
pub const LevelDBStore = struct { /* TODO */ };  
pub const SqliteStore = struct { /* TODO */ };

3. Block ID Resolution

const BlockId = union(enum) {
    slot: Slot,                    // "123"
    root: Root,                    // "0x1234..."
    keyword: enum { head, finalized, genesis },  // "head"
};

4. Integration Points

// In BeamNode initialization
var beam_node = try BeamNode.init(allocator, .{
    // ... existing options
    .apiPort = beamcmd.apiPort,  // New CLI flag
});

// Start API server alongside metrics
try beam_node.startApiServer();

Concrete Implementation Plan

Phase 1: Storage Foundation

# Create package structure
mkdir -p pkgs/api/src/{storage,lean}

# Priority order:
1. storage/interface.zig     # Storage trait definition
2. storage/ram_store.zig     # RAM implementation  
3. Chain integration point   # Hook into BeamChain.onBlock()
4. Basic testing            # Verify blocks are stored/retrieved

Phase 2: HTTP Server & Routing

# Following metrics_server.zig pattern:
1. lib.zig                  # Main LeanApiServer struct
2. utils.zig               # Block ID parsing logic
3. types.zig               # API response structures
4. Basic HTTP routing      # URL pattern matching

Phase 3: Endpoint Implementation

# One endpoint at a time:
1. lean/node.zig           # Simple version endpoint (test HTTP)
2. lean/headers.zig        # Block headers (ProtoBlock data)  
3. lean/blocks.zig         # Full blocks (storage integration)

Phase 4: Node Integration

# BeamNode integration:
1. CLI flag addition       # --api-port 9669
2. Node startup hook       # startApiServer()
3. Build system            # Add api module to build.zig
4. End-to-end testing      # Full flow verification

Team Decisions Made

** Storage Strategy**: Swappable Interface + RAM Store

• Use trait pattern for easy backend swapping
• Start with in-memory HashMap implementation
• Prepare for future disk/database backends

** Architecture Pattern**: New Subsystem

• Built from ground up following existing patterns
• Clean separation from chain logic
• Modular design for easy maintenance

Remaining Decisions Needed

** Technical Decisions**

1. API Port: Default 9669? (metrics uses 9667)
2. Storage Integration: Hook into chain.onBlock() or separate?
3. Error Responses: JSON error format alignment with beacon API?
4. Block Retention: Store all blocks or implement LRU eviction?

Example API Responses

Block Header Response

{
  "data": {
    "slot": "123",
    "proposer_index": "1", 
    "parent_root": "0x...",
    "state_root": "0x...",
    "body_root": "0x..."
  }
}

Node Version Response

{
  "data": {
    "version": "zeam/v0.1.0",
    "implementation": "zeam",
    "version_number": "0.1.0"
  }
}

Next Steps

1. Discussion on storage strategy choice
2. Coordinate with Jun on API requirements

---

Action Items

1. Review this updated plan - Does the swappable storage approach meet requirements?
2. Set timeline - Target completion date for visualization tool integration?
3. Technical decisions - Finalize API port, error formats, etc.

Success Metrics

• Swappable storage backends (easy to change from RAM → disk)
• Clean separation from existing chain logic
• All 3 lean endpoints working
• Visualization tool integration successful

This plan addresses Guillaume requirement for swappable storage while providing a concrete roadmap for the new subsystem implementation.

[bomanaps]

@gballet @g11tech created this thread to discuss how we are going to go about this issue #146

[GrapeBaBa]

For package organization structure, I have some ideas, not for API packages, but for the entire zeam project. The current project's package, module organization looks more like a mix of Go and Rust styles without following Zig's way. For large Zig projects, from looking at the Zig standard library and the source code of Tigerbeetle and Bun, it was found that Zig prefers flat project organization forms, rarely nesting multi-layer directories, and at the same time, they cannot see files of such types.zig, lib.zig, util.zig. There is also no large amount of addImport code in build.zig. I think at the right time we can reconstruct the organizational form of the project in the Zig standard library.

[GrapeBaBa]

1. Shouldn't it make more sense that the storage module should be an separate module with the same level of API?
2. Should we consider the https://github.com/karlseguin/http.zig or https://github.com/zigzap/zap instead of std.http.Server for simplify API development?

[bomanaps]

This is good at least we can test storage logic freely and it will be cleaner too, but we also need to know how we are going to handle the additional package and do we want to add storage module to build.zig? plus the make sure API and Storage versions stays compatible, do we also want to add external dependencies to the project from what I have noticed so far we are trying to avoid that as much as possible, this is part of the things we still need to consider

[bomanaps]

For the storage suggestion I agree because the benefits outweighs the cost, the HTTP framework am not sure, we have 3 endpoint to implement I don't know how complex the API will become in future

[bomanaps]

Dropping the Lean API documenet for reference https://hackmd.io/@reamlabs/SJksQfkoeg