|
| 1 | +# CLI Live Integration Tests |
| 2 | + |
| 3 | +Live integration tests run the real CLI binary against Confluent Cloud. They create, read, update, and delete real resources to verify end-to-end CLI behavior. |
| 4 | + |
| 5 | +## Prerequisites |
| 6 | + |
| 7 | +1. **CLI binary** — Build with `make build-for-live-test` |
| 8 | +2. **Confluent Cloud credentials** — Set the following environment variables: |
| 9 | + |
| 10 | +| Variable | Required | Description | |
| 11 | +|---|---|---| |
| 12 | +| `CONFLUENT_CLOUD_EMAIL` | Yes | Confluent Cloud login email | |
| 13 | +| `CONFLUENT_CLOUD_PASSWORD` | Yes | Confluent Cloud login password | |
| 14 | +| `CLI_LIVE_TEST_CLOUD` | No | Cloud provider: `aws` (default), `gcp`, `azure` | |
| 15 | +| `CLI_LIVE_TEST_REGION` | No | Cloud region (default: `us-east-1`) | |
| 16 | +| `LIVE_TEST_ENVIRONMENT_ID` | Kafka topics only | Pre-existing environment ID for topic tests | |
| 17 | +| `KAFKA_STANDARD_AWS_CLUSTER_ID` | Kafka topics only | Pre-existing cluster ID for topic tests | |
| 18 | + |
| 19 | +## Running Tests |
| 20 | + |
| 21 | +### All tests |
| 22 | +```bash |
| 23 | +make live-test |
| 24 | +``` |
| 25 | + |
| 26 | +### By group |
| 27 | +```bash |
| 28 | +make live-test-core # environment, service account, API key |
| 29 | +make live-test-essential # core + kafka |
| 30 | +make live-test CLI_LIVE_TEST_GROUPS="kafka" # kafka only |
| 31 | +``` |
| 32 | + |
| 33 | +### Multi-cloud |
| 34 | +```bash |
| 35 | +CLI_LIVE_TEST_CLOUD=gcp CLI_LIVE_TEST_REGION=us-east1 make live-test-essential |
| 36 | +``` |
| 37 | + |
| 38 | +### Single test |
| 39 | +```bash |
| 40 | +CLI_LIVE_TEST=1 go test ./test/live/ -v -run TestLive/TestKafkaClusterCRUDLive \ |
| 41 | + -tags="live_test,kafka" -timeout 30m |
| 42 | +``` |
| 43 | + |
| 44 | +## Test Groups |
| 45 | + |
| 46 | +Tests are organized into groups via Go build tags: |
| 47 | + |
| 48 | +| Group | Tag | Tests | |
| 49 | +|---|---|---| |
| 50 | +| Core | `core` | Environment, Service Account, API Key CRUD | |
| 51 | +| Kafka | `kafka` | Kafka Cluster CRUD, Kafka Topic CRUD | |
| 52 | +| All | `all` | Everything | |
| 53 | + |
| 54 | +The `essential` group in Semaphore/Makefile maps to `core,kafka`. |
| 55 | + |
| 56 | +## Concurrency Model |
| 57 | + |
| 58 | +- Each test method calls `s.setupTestContext(t)` which creates an **isolated HOME directory** and logs in. This means each test has its own CLI config — no shared state. |
| 59 | +- Tests opt in to concurrency by calling `t.Parallel()` at the start. All current tests do this. |
| 60 | +- The `-parallel 10` flag in the Makefile controls max concurrent tests. |
| 61 | +- Tests that need sequential execution (e.g., tests modifying shared external state) should simply omit the `t.Parallel()` call. |
| 62 | + |
| 63 | +## Writing a New Test |
| 64 | + |
| 65 | +### 1. Create a test file |
| 66 | + |
| 67 | +```go |
| 68 | +//go:build live_test && (all || mygroup) |
| 69 | + |
| 70 | +package live |
| 71 | + |
| 72 | +func (s *CLILiveTestSuite) TestMyResourceCRUDLive() { |
| 73 | + t := s.T() |
| 74 | + t.Parallel() |
| 75 | + state := s.setupTestContext(t) |
| 76 | + |
| 77 | + // ... test body ... |
| 78 | +} |
| 79 | +``` |
| 80 | + |
| 81 | +The test method name **must** end with `Live` to match the `-run=".*Live$"` filter. |
| 82 | + |
| 83 | +### 2. Define test steps |
| 84 | + |
| 85 | +Use `CLILiveTest` structs for each CLI command: |
| 86 | + |
| 87 | +```go |
| 88 | +steps := []CLILiveTest{ |
| 89 | + { |
| 90 | + Name: "Create resource", |
| 91 | + Args: "resource create my-name -o json", |
| 92 | + JSONFieldsExist: []string{"id"}, |
| 93 | + CaptureID: "resource_id", // captures JSON "id" field into state |
| 94 | + }, |
| 95 | + { |
| 96 | + Name: "Describe resource", |
| 97 | + Args: "resource describe {{.resource_id}} -o json", |
| 98 | + UseStateVars: true, // enables {{.key}} template substitution |
| 99 | + JSONFields: map[string]string{"name": "my-name"}, |
| 100 | + }, |
| 101 | +} |
| 102 | +``` |
| 103 | + |
| 104 | +### 3. Register cleanup |
| 105 | + |
| 106 | +Always register cleanup **before** creating resources (LIFO execution order): |
| 107 | + |
| 108 | +```go |
| 109 | +s.registerCleanup(t, "resource delete {{.resource_id}} --force", state) |
| 110 | +``` |
| 111 | + |
| 112 | +### 4. Run steps |
| 113 | + |
| 114 | +```go |
| 115 | +for _, step := range steps { |
| 116 | + t.Run(step.Name, func(t *testing.T) { |
| 117 | + s.runLiveCommand(t, step, state) |
| 118 | + }) |
| 119 | +} |
| 120 | +``` |
| 121 | + |
| 122 | +### CLILiveTest Field Reference |
| 123 | + |
| 124 | +| Field | Type | Description | |
| 125 | +|---|---|---| |
| 126 | +| `Name` | `string` | Step name shown in output | |
| 127 | +| `Args` | `string` | CLI arguments (supports `{{.key}}` when `UseStateVars` is true) | |
| 128 | +| `ExitCode` | `int` | Expected exit code (default 0) | |
| 129 | +| `Input` | `string` | Stdin content | |
| 130 | +| `Contains` | `[]string` | Strings that must appear in output | |
| 131 | +| `NotContains` | `[]string` | Strings that must NOT appear in output | |
| 132 | +| `Regex` | `[]string` | Regex patterns output must match | |
| 133 | +| `JSONFields` | `map[string]string` | JSON fields to check (empty value = any non-empty value) | |
| 134 | +| `JSONFieldsExist` | `[]string` | JSON fields that must exist (any value) | |
| 135 | +| `WantFunc` | `func(t, output, state)` | Custom assertion function | |
| 136 | +| `CaptureID` | `string` | State key to store extracted JSON "id" field | |
| 137 | +| `UseStateVars` | `bool` | Enable `{{.key}}` template substitution in Args | |
| 138 | + |
| 139 | +### Async Operations |
| 140 | + |
| 141 | +For operations that take time (e.g., cluster provisioning), use `waitForCondition`: |
| 142 | + |
| 143 | +```go |
| 144 | +s.waitForCondition(t, |
| 145 | + "kafka cluster describe {{.cluster_id}} -o json", |
| 146 | + state, |
| 147 | + func(output string) bool { |
| 148 | + return strings.EqualFold(extractJSONField(t, output, "status"), "UP") |
| 149 | + }, |
| 150 | + 30*time.Second, // poll interval |
| 151 | + 10*time.Minute, // timeout |
| 152 | +) |
| 153 | +``` |
| 154 | + |
| 155 | +## Adding a New Test Group |
| 156 | + |
| 157 | +1. Create test file(s) with build tag: `//go:build live_test && (all || mygroup)` |
| 158 | +2. Add a Makefile target: |
| 159 | + ```makefile |
| 160 | + .PHONY: live-test-mygroup |
| 161 | + live-test-mygroup: |
| 162 | + @$(MAKE) live-test CLI_LIVE_TEST_GROUPS="mygroup" |
| 163 | + ``` |
| 164 | +3. Update the Semaphore promotion parameters if the group should be selectable in CI. |
| 165 | + |
| 166 | +## CI (Semaphore) |
| 167 | + |
| 168 | +Live tests are triggered via the "Run live integration tests" promotion in `.semaphore/semaphore.yml`. Parameters: |
| 169 | + |
| 170 | +- **CLI_LIVE_TEST_GROUPS** — Test group to run (default: `essential`) |
| 171 | +- **CLI_LIVE_TEST_CLOUD** — Cloud provider (default: `aws`) |
| 172 | +- **CLI_LIVE_TEST_REGION** — Cloud region (default: `us-east-1`) |
| 173 | + |
| 174 | +Credentials are loaded from Vault secrets in the Semaphore pipeline. |
0 commit comments