HBASE-30087: Create GitHub Actions for testing Read-Replica feature

[kgeisz]

HBASE-30087: Create GitHub Actions for testing Read-Replica feature

Introduction

• Apache HBase Read Replica Cluster Feature
• Read Replica Test Plan Document
• HBase Website Documentation for Read Replica Cluster

HBase's Read-Replica feature was merged into master in PR #8364. This pull request introduces integration tests for Read-Replica via GitHub Actions and Docker containers. GitHub Actions and Docker containers have been useful for testing this feature because it works around the META_TABLE_NAME issue mentioned in HBASE-29691 and PR #7730.

In a Read-Replica setup, clusters share the same storage location but need to have different META_TABLE_NAMEs. The table names are distinguished using the hbase.meta.table.suffix configuration property. However, this still leads to problems when using MiniHBaseCluster to test a multi-cluster Read-Replica setup because these clusters run in the same JVM and end up sharing the same static META_TABLE_NAME variable.

How It Works

• The .github/workflows/read-replica-integration-tests.yml file is the main driver of the integration tests. This file defines the workflow and steps performed during the integration test.
• The dev-support/integration-test/read-replica directory contains various files for building a Docker image, running containers with HBase procecces, and running integration test scripts written in Python. It also contains files and directories used by HBase, such as conf directories.
• The hbase_docker_client.py file is the most important Python file. Every test script uses this file to communicate with each hbase-docker container. It does so by using docker exec to run commands in a container's HBase shell.

The tests are triggered when a someone submits a pull request with the read-replica label. Once the label is added, the test begins by checking out the HBase repo. This repo contains the pull request's code, so any changes to read-replica-integration-tests.yml and dev-support/integration-test/read-replica can be tested immediately. After, some environment setup is done.

Next, the test checks out the submitted pull request's HBase repo again, except the repo's directory is placed in dev-support/integration-test/read-replica. Here, the repo can be used to build a Docker image for running HBase. After the image has been built, two Docker containers are started, each of which are running HBase in a Read-Replica setup. One container starts as the active cluster (read-write mode), while the other container starts as a replica cluster (read-only mode). These clusters share a data-store directory that contain the hbase.rootdir. It is a mounted volume between each container and the local filesystem, and it is created by the Python scripts before container startup and given 777 permissions in order to avoid HBase startup failures. There are other volumes as well for easy access, such as each container's conf directory and a utils directory for bulkloading data.

Once the containers are up and running, a series of Python scripts are run as integration tests. They test expected behavior for a Read-Replica cluster setup, such as verifying valid/invalid startup, blocking writes on replica clusters, being able to refresh meta and HFiles on replica clusters to make them consistent with the active cluster, read-only flag flipping (hbase.global.readonly.enabled), verifying bug fixes, etc. The containers are shut down when the test is complete. If a failure occurs at any point, the workflow is stopped and the HBase logs for each container are dumped.

Other Information

• The test is re-triggered when a new commit is pushed. If the test is already running when a new commit is pushed, then that test is stopped and a new one is started using the latest commit. The test is not run for draft PRs.
• docker-compose.yml defines and configures two hbase-docker containers capable of running in a read-replica setup.
• build_images.sh uses Dockerfile to build an hbase-docker image. This script assumes there is an hbase repo in the same directory.
• .env defines environment variables used by docker-compose.yml, build_images.sh, and Python scripts.
• proto_compiler.py copies ActiveClusterSuffix.proto from the hbase repo and compiles it. The generated output is used for verifying the active.cluster.suffix.id file.
• utils is a directory containing scripts for bulkloading data into HBase
• conf1 and conf2 are the active and replica cluster's configuration directories, respectively. They each contain their own hbase-site.xml, log4j2.properties, and zoo.cfg files. The directories are defined as mounted volumes in docker-compose.yml. This is especially useful for changing the value of hbase.global.readonly.enabled in order to change a cluster's read-only mode.

Example Test Runs

• Successful test run.
  • This test run is from an example pull request that makes changes to resolve HBASE-30301. This test run builds a Docker image that has the bug fix, resulting in all tests passing.
  • https://github.com/kgeisz/hbase/actions/runs/30566311529/job/90954447739
• Failed test run.
  • The test run from this example PR does not include changes for HBASE-30301. As a result, the Docker image that was built did not contain the bug fix, and one of the integration tests failed. The HBase logs were dumped after the failure.
  • https://github.com/kgeisz/hbase/actions/runs/30374441518/job/90326363151