Simplify and Accelerate Integration Tests

[paul-rogers]

Motivation

Today Druid integration tests are slow to run in Travis, and painful to use as a developer. We wish to fix this issue.

This proposal starts the discussion. A prototype of the redesign is in progress. However, the integration tests are complex: let's get more eyes on the issue to ensure we have a workable plan to modernize the tests.

Proposed changes

Proposed is a restructuring of the integration tests to allow faster Travis runs and far easier debugging. Key goals include:

• Speed-up the Druid test image build by avoiding download of dependencies. (Instead, any such dependencies are managed by Maven and reside in the local build cache.)
• Use official images for dependencies to avoid the need to download, install, and manage those dependencies.
• Ensure that it is easy to manually build the image, launch a cluster, and run a test against the cluster.
• Convert tests to JUnit so that they will easily run in your favorite IDE, just like other Druid tests.
• Use the actual Druid build from distribution so we know what is tested.
• Leverage, don't fight, Maven.
• Run the integration tests easily on a typical development machine.

This project is mostly a matter of managing many details. Rather than spell out the gory details here, please see the project documentation in the prototype branch.

Rationale

The current integration tests are characterized by a number of quirks resulting from their evolution.

• The tests run in Maven before the distribution project, forcing the integration tests to do their own build. This means that what we test is not the software we "ship."
• The tests are in a single Maven project. That single project can start the cluster, run tests, and shut down the cluster. Since we have multiple test groups, this means that we need a separate Travis run for each test group.
• The test image contains Druid plus dependencies such as MySQL, ZooKeeper and Kafka. Each build pulls these dependencies down from the public repository, resulting in very slow image builds (and unkind load on the upstream repositories.)
• The plumbing (scripts, Maven tasks, etc.) show their age: they are very complex and quite difficult to understand and extend.
• Tests are based on the TestNG framework and are hard to run within an IDE.
• The directory mounted into the container is placed in the user's home directory, which makes it hard to manage and outside of the Maven build tree. When run locally, tests share the same directory, leading to non-deterministic results.

Each of these issues contains the seed of its resolution:

• Move tests to run in Maven after distribution so that the tests run against the artifacts produced by the main Maven build. This eliminates the need for a second build within the tests.
• Split the tests into multiple Maven subprojects. Each can start a Docker cluster as that test needs, run the test, and shuts down the cluster. Maven will step through the subprojects (former test groups) one by one in a single run.
• MySQL, ZK and Kafka all provide "official" images. We can use those to avoid pulling down the software on each build. We can use Maven to pull down things like the MySQL driver, MariaDB driver and Kafka protobuf provider. Both Docker and Maven provide caches, so that we hit the upstream repositories only when the dependencies change, not once per test run.
• By splitting tests into multiple projects, we can simplify the "plumbing": each test project performs the setup that it needs, without complex scripts and and if-statements.
• TestNG usage is replaced by the JUnit library which has built-in IDE support. A bit of shim code ensures that most test code can remain unchanged.
• The shared directory mounted into the containers now resides in each project's target folder, so that Maven will clean up the directory the same way it cleans all other build artifacts.

Migration

The Druid integration tests represent a large amount of code. One-shot conversion will not be possible. Instead, we must work step-by-step.

• The prototype branch has worked out how to convert a single test group using the new structure. (The current code is a work in progress.)
• We propose to check in a final version of that code as a new Druid project. The project would not yet be built in Travis, but can be used for local testing by developers.
• Once we feel the new code is stable, we can replace the existing Travis tests group-by-group. A single new Travis job will eventually include all of the current test groups, resulting in a faster build cycle.
• If we don't have capacity to convert all tests, then some groups may continue to run in the old system for the time being.

This approach does mean some short-term redundancy, but provides a safe roll-out path.

Future work

This proposal is an outline of an approach, along with a prototype to demonstrate the idea. Additional work includes:

• Complete the foundational work. Mostly a matter of final clean-ups, verifying that tests flow smoothly, etc.
• Conversion of the remaining tests.
• Ensure the tests work in the Kubernetes and Quickstart environments which some tests seem to support.
• Support multiple environments: MariaDB vs. MySQL drivers, Hadoop vs. S3 vs. Azure vs. GCS, etc.

[FrankChen021]

To be frank, the Integration test is the most painful thing that I don't want to touch in Druid. It takes very long time for developers to get the test result whenever they add or modify the tests. I really like your idea that we can just run these tests from IDE.

The proposal is challengeable because to replace current integration tests into new ones contains lots of work. I agree that we can migrate tests group by group but don't think we can migrate all tests even under one group. I've no good idea on it.

[gianm]

I agree that the integration tests are a pain. It's really great to hear an idea for improving them. I wish you the best of luck in this effort.

As to migration: IMO it would make a big difference to just write new tests in a better framework. We can migrate old tests as time permits. Even if that takes a long time, it's still good to have new tests run on a more solid system.

[gianm]

I read over the basic idea and took a look at the branch. It seems like a good approach.

[kfaraz]

Thanks for looking into this, @paul-rogers !!

Maven will step through the subprojects (former test groups) one by one in a single run.

Does this mean all the (new) integration-tests will run in a single Travis job?
Having separate Travis jobs for each test-group helped with parallelism and also allowed a user to only rerun failed tests. How would retry of a single failed integration test work in the new setup?

[gianm]

Having separate Travis jobs for each test-group helped with parallelism and also allowed a user to only rerun failed tests.

Personally, I'd hope that newer integration tests are written in such a way that they are not as flaky as the current ones. When I've looked into the flakiness of specific tests, most of what I've seen is explained by race conditions or logic errors in the tests themselves. It's certainly possible to write integration tests that don't have issues like this.

[kfaraz]

That's true, we should hope to get rid of the flakiness in the new setup.

But won't having all the integration tests in one big Travis job also hurt overall build time?
I agree that the number of jobs is a little too many right now. But some split would still be good.
We could probably run some of the test groups together, based on the underlying functionality that they test?

[gianm]

But won't having all the integration tests in one big Travis job also hurt overall build time?

IMO, it makes sense to split them in Travis if runtime is long, to allow them to run in parallel. Curious what @paul-rogers thinks too, as it's his proposal.

[paul-rogers]

@kfaraz, On the question about running tests in parallel: one reason we want to do that today is that every test builds Druid, builds the Docker image, and downloads a bunch of stuff. Even for something like the High Availability tests which then run in a few seconds. So, today, we're parallelizing wasted work. With the change, we'd build everything once, then run. Each run is:

• Start Docker Compose cluster using test-specific configs.
• Run the ITs for that group.
• Shut down the cluster.

If we find that is slow, then we can split the tests into two sets and do builds in both then run both.

There is still room for parallelism. The groups will run in sequence. However, it looks like we run with multiple configurations: with MySQL driver or with MariaDB driver. Secure or not secure, etc. Each of these variations must still run separately because we need to invoke the same maven modules, but with different profiles.

On Travis, one way to speed up more is if we could cache the Docker container and build artifacts (all the Druid jars). This means:

1. Build Druid and the Docker image. Place in a PR-specific repo.
2. Run all IT tests in parallel. Each pulls jars and the image from the above repo.

The above should work for config variations also, since the Druid jars and the test image are consistent; only the test code and "shared folder" would be twiddled to change configs.

I don't know Travis (and repos) well enough to know if this is possible, but it is something to consider.

[paul-rogers]

@gianm, from the very small sample size I've seen, it is clear that the tests are retrying blindly. The check for the custom node role will retry forever. But, if the custom node role is down, it will stay down; it's not coming back on its own. If we can ensure that tests start only after the target services are alive, we'll save quite a bit of flailing. How do we do that? Easy, just check ZK. The services seem to register to ZK as the last step on startup, so if they're not good by then, they never will be.

This comment does suggest a good place to start is to find the "usual suspects" flaky tests and convert those, paying attention to why they are flaky and fixing that. If the tests are flaky, then we can assume that a client using the same API will be flaky and we'd want to fix that.

[FrankChen021]

On Travis, one way to speed up more is if we could cache the Docker container and build artifacts (all the Druid jars). This means:

Build Druid and the Docker image. Place in a PR-specific repo.
Run all IT tests in parallel. Each pulls jars and the image from the above repo.
The above should work for config variations also, since the Druid jars and the test image are consistent; only the test code and "shared folder" would be twiddled to change configs.

I don't know Travis (and repos) well enough to know if this is possible, but it is something to consider.

This is very good suggestion. By separating the image building from the tests, we can re-run the tests again without re-build the images if there's any possible flaky test. I think it can save a lot of time.

And the proposal is feasible. After building the image, we can execute a 'docker push' command to push the image to the docker hub. I've done similar work on one of my repo by using GitHub Actions -- for each push on master branch, an image is automatically built and pushed to the docker hub. Here is the GitHub Action file I'm using.

For Travis, I think it's the same. The only problem is how to keep some secret information such as password out of the travis configuration file. GitHub Action allows us to configure these secret per repository and it will inject these information to the configuration automatically. I don't know how to do it on Travis but I think there must be a way.

[paul-rogers]

There is a PR for this work: #12368. With this, it is now possible to:

1. Build Druid in the normal way.
2. Build the test image: about 1 minute.
3. Run a test: depending on task, 1 minute to several minutes.

It is also possible to speed up the process for debugging.

2a. After the first image build, you can use a script to rebuild in just a few seconds.
3a. The converted ITs can be run from your favorite IDE as a JUnit test.

The PR is pretty big, but most of it is configs and refactored code. Please review and provide suggestions.