Skip to content

Reproducible environment for load testing of retrievals #1019

Open
@nonsense

Description

@nonsense

We should be able to spin up reproducible environments to perform load tests on specific instance types, so that we have better visibility on the state of retrieval performance and whether we make things better/worse with time.

This should be made possible on AWS.

This involves:

  • producing prebuilt AMIs that we can very quick start on AWS (via packer for example)
  • these AMIs should contain the latest set of proof params, so that we don't download every time hundreds of GiB of proof parameters to start a network
  • AMIs should automatically start a network that a load test could hit. This means all Lotus and Boost services should spin up when the machine is up: lotus daemon, lotus-miner, boostd, booster-http and booster-bitswap, and endpoints to access them should be reachable
  • dev network should be pre-populated with a number of 32GiB sectors that already have deals in them that we could fetch. Deals must be of realistic size -- i.e. anywhere from 300MiB up to 30GiB.
  • ideally it should be possible to spin up an AWS instance with such a predefined environment within 5-10min.
  • local monitoring stack should also be provisioned on the AMI and publicly accessible

Nice to have:

  • ideally a user should be able to make a deal with a random 4GiB file without hitting any of the proofs, i.e. network should be configured with fake-proofs.
  • ideally it should be possible to create a deal and be able to retrieve it within 2-3min.

Relies on:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions