forge caches RPC responses automatically, they are saved to ~/.foundry/cache/rpc/{chain}/{blockNumber}, so there is probably just some CI config we need to save/load the cache to improve CI performance
We have code in the monorepo that handles this caching based on the pinned block: https://github.com/ethereum-optimism/optimism/blob/b6a82c2de9c9372d3012e73e702e9671579425b7/.circleci/config.yml#L869-L915