Background

We need to update the SDS protocol to support more efficient message retrieval from Store nodes as per this update to the SDS specification: vacp2p/rfc-index#130

Problem

Currently, causal history entries only contain SDS message IDs. When a participant needs to retrieve missing messages based on causal dependencies, they have no way to efficiently query Store nodes since:

1. Store nodes work with Waku message hashes, not SDS message IDs
2. Time-based queries are inefficient and unreliable
3. We lack a mapping between SDS message IDs and queryable identifiers

Proposed Solution

Update the causal history structure to include optional retrieval hints that can be used to efficiently query external stores:

message Message {
  string message_id = 2;
  string channel_id = 3;
  optional int32 lamport_timestamp = 10;
  repeated HistoryEntry causal_history = 11;  // Updated structure
  optional bytes bloom_filter = 12;
  optional bytes content = 20;
}

message HistoryEntry {
  string message_id = 1; // SDS message ID
  optional bytes retrieval_hint = 2; // Optional hint for efficient retrieval (e.g., Waku message hash)
}

Motivation

1. Efficient Store Queries: Applications can provide Waku message hashes as retrieval hints for direct Store node queries
2. Separation of Concerns: SDS message IDs remain independent of transport layer concerns
3. Backward Compatibility: Optional field ensures existing implementations continue to work

Implementation Tasks

1. Update Protocol Definitions

• Update message.nim to include new HistoryEntry type
• Modify Message struct to use seq[HistoryEntry] instead of seq[MessageID] for causalHistory
• Update protobuf serialization/deserialization in protobuf.nim

2. Update Core Logic

• Modify wrapOutgoingMessage to populate retrieval hints when creating causal history
• Update dependency checking logic to work with new HistoryEntry structure

3. API Changes

• Update markDependenciesMet to handle both message IDs and retrieval hints
• Add new callback for applications to provide retrieval hints for historical messages
• Update callback signatures to use seq[HistoryEntry]

4. Testing Requirements

• Update tests to cover both with/without retrieval hint scenarios
• Test causal dependency resolution with and without retrieval hints
• Verify protobuf serialization handles optional retrieval hints correctly
• Ensure bloom filter ACK mechanisms work with new structure
• Update libsds and sds-go bindings to work with new HistoryEntry type