Antithesis engagement with correctness testing

[serathius]

What would you like to be added?

For some time etcd maintainers have been in contact with https://antithesis.com/ a company offering a deterministic execution environment perfect for testing distributed systems like etcd. This could work as a complementary to existing robustness test that validate etcd correctness under stress and failure, but struggle with reproducibility. Why not combine both approaches and have 100% reproducible correctness tests?

Linux Foundation has offered to sponsor a 3 month engagement to see how much we can achieve by combining efforts community and awesome software build by Antithesis. We are on finals steps of legal contract signing, with planned start on February 14.

I'm looking for people interested in helping integrate robustness tests with Antithesis. The first goal would be to setup etcd

Materials:

• https://www.antithesis.com/docs/test_templates/test_composer_reference/
• https://www.antithesis.com/docs/test_templates/first_test/
• https://github.com/guergabo/quickstarts
• https://github.com/antithesishq/etcd-test-composer

/cc @MadhavJivrajani @fuweid @siyuanfoundation @henrybear327 @AwesomePatrol @jmhbnz @wenjiaswe

Why is this needed?

Provide 100% reproducible robustness tests

[aladesawe]

I'm interested

[fuweid]

+1 to help!

[siyuanfoundation]

+1 to help!

[serathius]

Thanks for interest, please read the linked Antithesis documentation and see https://github.com/antithesishq/etcd-test-composer/tree/main/test-template/main as example of integrating linearization. Hopefully we will get confirmation that we can start by the end of the week.

[serathius]

Got an update from Antithesis team, agreement is on final stage and we expect the engagement to start next week.

First thing that we need to do on etcd side is to agree on location where the work should be done.
I would like to propose to keep the code within the main etcd repository to allow us to iterate fast on integrating it with robustness libraries. The antithesis specific code should be separated in it's own directory tests/anthithesis.

cc @ahrtr @jmhbnz @fuweid @ivanvc for opinion.

[ayushpatil2122]

I'm also interested

[CraigAlfieri]

Hi etcd Maintainers! Craig Alfieri from Antithesis (www.Antithesis.com) here 👋🏻. We're looking forward to kicking off the Project with the team. We have finalized the paperwork with Linux Foundation as of yesterady and are ready to proceed forward at the team's earliest convenience.

Our team has instantiated the environment. Our Professional Services Team is ready to meet with the group to address any immediate questions / help guide the team through getting started.

Per Marek's request we Open Sourced the etcd Test Composer Templates (https://github.com/antithesishq/etcd-test-composer/tree/main/test-template/main), and are ready when the team is ready to proceed.

Looking forward to working on the robustness testing of etcd with the group.
Best,
-Antithesis Team.

[serathius]

Due o upcoming KubeCon conference I expect people are busy. I think can start the work on April 7th.

[serathius]

Requested a admin access for maintainers to setup a secret for Antithesis credentials. kubernetes/org#5523

[serathius]

@abdurrehman107 The first step for the Antithesis integration would be to move parts of demo https://github.com/antithesishq/etcd-test-composer/ into the etcd repository. First good milestone would be to be able to run the test locally starting from setting up a local cluster in docker compose as described in https://github.com/antithesishq/etcd-test-composer/tree/main?tab=readme-ov-file#testing-locally

On etcd side we would like to create a directory tests/antithesis and under it we would like to put the config/docker-compose.yaml file from the etcd-test-composer repo. In the docker compose file we would want to make some adjustments like switching the images from bitnami to official etcd image.

[serathius]

Got the basic setup in #19739

[serathius]

Step 2 would be integrate a github workflow with Antithesis. Let's start with a daily workflow that triggers a run in antithesis based on https://github.com/antithesishq/etcd-test-composer/tree/main/.github/workflows. Maybe instead of two separate workflows one that builds and runs the test would be better for us.

To access secrets we need to configure environment in github action
name: Antithesis
secrets:

• ANTITHESIS_CONTAINER_REGISTRY_TOKEN
• ANTITHESIS_WEBHOOK_PASSWORD
• ANTITHESIS_WEBHOOK_USERNAME

cc @henrybear327 could you take a look?