Build Centralized Tasking System

[pyropy]

Context

The new Walrus and Arweave checkers don't rely on a centralized tasking system. Instead, they independently select the blobs and transactions they want to check. Sampling blobs and transactions is performed by querying public APIs, with each checker making two requests per minute. Currently, our network consists of approximately 40,000 active nodes, meaning that deploying the new checker modules across the entire network would generate over 1,000 requests per second.

To avoid driving too much traffic to the public APIs' we would need create a centralised tasking service.

Description

Design and implement a tasking service that generates and distributes measurement tasks to checker nodes across different subnets (Arweave, Walrus). This system will act as an intermediary between public network APIs and checker nodes, significantly reducing the load on these APIs by eliminating redundant calls from individual nodes.

Acceptance Criteria

• Periodically query public network APIs to gather necessary information for generating valid measurement tasks
• Implement task assignment and round algorithms based on current implementations of spark-api
• Create new rounds with pool of ready-to-execute measurement tasks for all supported networks
• Provide an API for checker nodes to read current round details including tasks within the round

This component is critical for enabling increased measurement coverage while avoiding putting pressure on the public APIs, and will serve as the foundation for scaling the subnet measurement infrastructure in the future.

[juliangruber]

This system will act as an intermediary between public network APIs and checker nodes, significantly reducing the load on these APIs by eliminating redundant calls from individual nodes.

How certain are we that the load created by checkers is an issue?

This component is critical for enabling increased measurement coverage while respecting the limitations of public APIs, and will serve as the foundation for scaling the subnet measurement infrastructure in the future.

What limitations are in place? Do you have links for this?

[pyropy]

How certain are we that the load created by checkers is an issue?

At this point it's just an assumption, but as we've previously discussed doing 2 requests per minute from 40k active nodes results in over 1k requests per second.

What limitations are in place? Do you have links for this?

No but it's good that you've spotted this. I've used Claude to rephrase original task description 👉🏻 👈🏻 I'm going to edit the task description.