Skip to content

README.md

Latest commit

 

History

History
292 lines (229 loc) · 9.95 KB

README.md

File metadata and controls

292 lines (229 loc) · 9.95 KB
Raw

Back to Spanner Tools

Cloud Spanner CDC Load Test

Disclaimer: This tool is provided "as-is" for development, testing, and debugging purposes. It is not intended for critical production workloads or strict SLA environments. Use at your own risk.

A Java-based load generator for Cloud Spanner, designed to test Change Data Capture (CDC) and general database performance. It supports various strategies to simulate different types of load.

Strategies

Strategy Description
RANDOM Standard Insert/Update/Delete on random IDs.
SEQUENTIAL Strict sequence of Insert -> 10 Updates -> Delete on a unique ID. High churn per ID.
HOTSPOT High contention on a single shared row (Read-Write Transaction).
ATOMICITY Simulates money transfer between two shared rows (Read-Write Transaction).
SATURATION Large batch inserts (~1MB per op) to saturate network/CPU.
INTEGRITY Rapid Insert/Delete cycles on a fixed pool of keys (Resurrection testing).
READ_HEAVY 90% Read (point lookup), 10% Insert. Maintains local cache of 10k recent keys.
MIXED Cycles through all the above strategies across workers.

Workflow Overview

  1. Infrastructure Setup: Provision Spanner instance(s), database, and change stream options using Terraform (or manual DDL).
  2. Build Application: Compile the load generator JAR.
  3. Run Load Test: Run locally or deploy to Cloud Run Jobs for distributed load generation.

1. Infrastructure Setup

Option A: Using Terraform (Recommended)

A lightweight Terraform module is provided in the terraform/ directory to spin up Spanner instances, databases, tables, and change streams with backups disabled.

cd terraform
cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars. Here is a complete, clean example with all available options:

project_id          = "your-project-id"
region              = "us-central1"
instance_config     = "regional-us-central1"
instance_prefix     = "spanner-loadtest"
processing_units    = 1000
database_id         = "loadtest-db"
deletion_protection = false
change_streams      = ["OLD_AND_NEW_VALUES"]

Change Stream Scenarios

Configure change_streams to control which instance(s) get provisioned:

  • 1 Instance with standard Change Stream (Default): change_streams = ["OLD_AND_NEW_VALUES"]
  • 1 Instance WITHOUT Change Stream: change_streams = ["NONE"]
  • 2 Instances (Baseline vs CDC): change_streams = ["NONE", "OLD_AND_NEW_VALUES"]
  • All Options: change_streams = ["NONE", "OLD_AND_NEW_VALUES", "NEW_VALUES", "NEW_ROW", "NEW_ROW_AND_OLD_VALUES"]

Deploy the infrastructure:

terraform init
terraform apply

The output will display the provisioned instance IDs, database IDs, and ready-to-run load test commands.

Option B: Manual Setup (Existing Database)

If you prefer to use an existing database or create schema manually, execute the following DDL in Cloud Console (Spanner Studio) or via gcloud:

CREATE TABLE LoadTestTable (
    Id STRING(36) NOT NULL,
    Data STRING(MAX),
    Counter INT64,
    IsActive BOOL,
    ExampleTimestamp TIMESTAMP OPTIONS (allow_commit_timestamp=true)
) PRIMARY KEY (Id);

-- Optional: Create Change Stream (select desired value_capture_type, or omit entirely)
CREATE CHANGE STREAM LoadTestStream FOR LoadTestTable
OPTIONS (
  value_capture_type = 'OLD_AND_NEW_VALUES', -- Options: 'OLD_AND_NEW_VALUES', 'NEW_VALUES', 'NEW_ROW', 'NEW_ROW_AND_OLD_VALUES'
  retention_period = '1d'
);

2. Build

Prerequisites

  • Java 17+
  • Maven 3.8+
  • Google Cloud SDK (gcloud)
mvn clean package -DskipTests

3. Local Usage

Command-Line Options

Option Long Option Required Default Description
-p --project Yes — Google Cloud Project ID
-i --instance Yes — Cloud Spanner Instance ID
-d --database Yes — Cloud Spanner Database ID
-c --concurrency No 10 Number of concurrent worker threads
-s --strategy No RANDOM Load strategy (RANDOM, SEQUENTIAL, HOTSPOT, ATOMICITY, SATURATION, INTEGRITY, READ_HEAVY, MIXED). See Strategies.
--duration No 60 Duration of test in seconds
-h --help No — Show help message and exit
-V --version No — Print version information and exit

Run Locally

Set your environment variables:

export PROJECT_ID="your-project-id"
export SPANNER_INSTANCE="your-instance-id"
export SPANNER_DATABASE="your-database-id"

Run with custom parameters:

java -jar target/spanner-cdc-loadtest-1.0-SNAPSHOT.jar \
  -p "${PROJECT_ID}" \
  -i "${SPANNER_INSTANCE}" \
  -d "${SPANNER_DATABASE}" \
  -c 10 \
  -s MIXED \
  --duration 60

Run with defaults (10 threads, RANDOM strategy, 60s duration):

java -jar target/spanner-cdc-loadtest-1.0-SNAPSHOT.jar \
  -p "${PROJECT_ID}" \
  -i "${SPANNER_INSTANCE}" \
  -d "${SPANNER_DATABASE}"

4. Cloud Run Jobs Deployment

Run massive load tests serverlessly using Cloud Run Jobs.

1. Variables

Set your environment variables:

export PROJECT_ID="your-project-id"
export REGION="us-central1"
# Use Artifact Registry (pkg.dev), NOT gcr.io
export REPO_NAME="spanner-loadtest-repo"
export IMAGE_NAME="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/spanner-loadtest:latest"
export SPANNER_INSTANCE="your-instance-id"
export SPANNER_DATABASE="your-database-id"

2. Initial Setup (One-time)

Create the Artifact Registry repository:

gcloud artifacts repositories create ${REPO_NAME} \
    --repository-format=docker \
    --location=${REGION} \
    --description="Docker repository for Spanner Load Test"

3. Build and Push

Build the container image using Cloud Build:

gcloud builds submit --tag ${IMAGE_NAME}

4. Create the Job Template

Create the Cloud Run Job. This registers the job definition, compute resources, and default CLI arguments for the load generator.

Note

gcloud run jobs create only registers the job template; it does not start any containers or run the load test yet. The --args provided here become the default configuration used by executions unless overridden. Note the --args syntax requires a single comma-separated string without spaces.

gcloud run jobs create spanner-loadtest \
  --image ${IMAGE_NAME} \
  --region ${REGION} \
  --cpu=4 \
  --memory=8Gi \
  --task-timeout=1h \
  --args="-p,${PROJECT_ID},-i,${SPANNER_INSTANCE},-d,${SPANNER_DATABASE},-c,5,-s,MIXED,--duration,60"

Note: Ensure --task-timeout is greater than your load test --duration.

5. Execute the Load Test

Trigger an execution run of the job.

Option A: Run with Default Arguments (Recommended)

By default, executions automatically inherit the target project, instance, database, threads, and strategy configured in step 4. No --args are needed:

gcloud run jobs execute spanner-loadtest \
  --region ${REGION} \
  --tasks 4

Option B: Temporary Argument Overrides (Optional)

If you want to run a one-off test with a different strategy (e.g. HOTSPOT) or duration without updating the base job template, pass --args to execute:

# Example: Run with 200 Total Threads (50 threads per task * 4 tasks) with HOTSPOT strategy
gcloud run jobs execute spanner-loadtest \
  --region ${REGION} \
  --tasks 4 \
  --task-timeout=30m \
  --args="-p,${PROJECT_ID},-i,${SPANNER_INSTANCE},-d,${SPANNER_DATABASE},-c,50,-s,HOTSPOT,--duration,300"

6. High-Scale / Distributed Execution (launch_jobs.sh)

For massive distributed load testing across multiple parallel executions, use the included launch_jobs.sh script. It fires multiple asynchronous Cloud Run Job executions concurrently:

# Trigger 8 executions of 10 tasks each = 80 concurrent containers
./launch_jobs.sh

You can customize the execution scale via environment variables:

NUM_EXECUTIONS=4 TASKS_PER_EXECUTION=25 REGION=us-central1 ./launch_jobs.sh

7. Update Configuration (Resources)

To change the CPU or Memory limits for an existing job:

gcloud run jobs update spanner-loadtest \
  --cpu=8 \
  --memory=16Gi \
  --region=${REGION}

Important: These resources (8 CPU, 16Gi Memory) are allocated PER TASK. If you execute this job with --tasks 10, you will provision 10 separate containers, each with 8 vCPUs (Total: 80 vCPUs). Rates are billed per vCPU-second and GB-second for each task.

Troubleshooting

  • Error: None of [grpclb] specified: The shaded JAR is missing ServicesResourceTransformer. Ensure it's in pom.xml and rebuild.
  • Error: denied: gcr.io repo does not exist: Use Artifact Registry (pkg.dev) as shown above.

5. End-to-End Cloud Run Quickstart

Variables

export PROJECT_ID="your-project-id"
export REGION="us-central1"
export REPO_NAME="spanner-loadtest-repo"
export IMAGE_NAME="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/spanner-loadtest:latest"

export SPANNER_INSTANCE="your-instance-id"
export SPANNER_DATABASE="your-database-id"

Initial setup

gcloud artifacts repositories create ${REPO_NAME} \
    --repository-format=docker \
    --location=${REGION} \
    --description="Docker repository for Spanner Load Test"

gcloud builds submit --tag ${IMAGE_NAME}

gcloud run jobs create spanner-loadtest \
  --image ${IMAGE_NAME} \
  --region ${REGION} \
  --cpu=4 \
  --memory=8Gi \
  --task-timeout=1h \
  --args="-p,${PROJECT_ID},-i,${SPANNER_INSTANCE},-d,${SPANNER_DATABASE},-c,4,-s,MIXED,--duration,600"

gcloud run jobs execute spanner-loadtest --region ${REGION} --tasks 50

Rebuild after changes

gcloud builds submit --tag ${IMAGE_NAME}
gcloud run jobs execute spanner-loadtest --region ${REGION} --tasks 2