Refactor to continuous background preimage generation and L1 finality caching

[yoshidan]

Description

This PR significantly refactors the architecture to switch from on-demand preimage generation to a continuous, always-on background generation model. This improves response latency and ensures preimages are readily available for finalized blocks.

Key Changes

• Continuous Preimage Collection: Introduced PreimageCollector, which runs in the background to continuously fetch L1 finality updates and generate preimages for the range between the agreed L2 block and the finalized L2 block.

• Decoupled Serving: The HTTP server now serves already generated preimages from the repository (FilePreimageRepository), decoupling the generation logic from the request path.
  L1 Finality Caching: Implementing FinalizedL1Repository to persistently store the latest finalized L1 headers.

• API Updates:

  • get_preimage: Updated to return pre-generated data from storage.
  • get_finalized_l1: New endpoint to retrieve finalized L1 headers by hash.
  • get_latest_metadata & list_metadata: New endpoints for better visibility into the available preimages.

• Concurrency & Maintenance:

  • Preimage collection supports parallel execution (max_concurrency).
  • Added PreimagePurger to automatically clean up expired preimages and headers based on TTL.

[siburu]

@yoshidan I still see the warning: This branch has conflicts that must be resolved.
It is necessary to fix some conflicts, right?

[yoshidan]

@siburu I fixed the conflict now.

[ikehara]

Some findings were reported by AI. I reviewed them and commented on the actual issues.

[ikehara]

Are there any metrics we can check to see if it’s working properly?
I’m concerned that, without metrics, we won’t be able to monitor the progress of preimage generation.

[yoshidan]

Are there any metrics we can check to see if it’s working properly?
I’m concerned that, without metrics, we won’t be able to monitor the progress of preimage generation.

No dedicated metrics exist, but we can get the latest preimage metadata through the "/get_latest_metadata" API.
We can perform synthetic checks to see if the value stays the same for too long.