- Developer Guide
So you want to contribute code to the OpenSearch Go Client? Excellent! We're glad you're here. Here's what you need to do:
Fork opensearch-project/opensearch-go and clone locally, e.g. git clone https://github.com/[your username]/opensearch-go.git.
OpenSearch Go Client builds using Go 1.24 at a minimum.
Docker is required for building some OpenSearch artifacts and executing integration tests.
To build the project on Windows, use WSL2, the compatibility layer for running Linux applications.
Install make
sudo apt install make
Go has a simple tool for running tests, and we simplified it further by creating this make command:
make test-unit
Individual unit tests can be run with the following command:
cd folder-path/to/test;
go test -v -run TestName;
In order to test opensearch-go client, you need a running OpenSearch cluster. You can use Docker to accomplish this. The Docker Compose file supports the ability to run integration tests for the project in local environments. If you have not installed docker-compose, you can install it from this link.
Ensure that Docker is installed on your local machine. You can check by running docker --version. Next, navigate to your local opensearch-go repository. Run the following command to build and start the OpenSearch docker container.
make cluster.build cluster.start
This command will start the OpenSearch container using the docker-compose.yaml configuration file. During the build process, the necessary dependencies and files will be downloaded, which may take some time depending on your internet connection and system resources.
Once the container is built and running, you can open a web browser and navigate to localhost:9200 to access the OpenSearch docker container.
The cluster targets work with any of three container providers. When you do not specify one, the Makefile auto-detects a provider by CLI presence in this order:
- Colima (
colimaonPATH) - Rancher Desktop (
rdctlonPATH) - Docker / Docker Desktop (
dockeronPATH)
Override the choice explicitly with CONTAINER_PROVIDER:
CONTAINER_PROVIDER=docker make cluster.start
Selecting a provider does four things:
- Pins the docker context commands talk to (
colimafor Colima,rancher-desktopfor Rancher; the Docker provider leaves your active context alone). IfDOCKER_CONTEXTis already set in your environment, it is respected and not overridden. - Resolves the CLI runtime
$(CTR)(defaults todockerfor every provider). For advanced containerd use you can forceCONTAINER_RUNTIME=nerdctl, though docker-context pinning does not apply to nerdctl. - Ensures the backing VM/daemon is running (
make cluster.provider.ensure, run automatically bycluster.start), starting it if needed. - Sets
vm.max_map_count=262144through the provider's VM (colima ssh/rdctl shell) or, for the Docker provider, a privileged helper container.
Run make cluster.runtime to see the detected provider, docker context, and runtime.
In order to differentiate unit tests from integration tests, Go has a built-in mechanism for allowing you to logically separate your tests with build tags. The build tag needs to be placed as close to the top of the file as possible, and must have a blank line beneath it. Hence, create all integration tests with build tag 'integration'.
- Run below command to start containers. By default, it will launch latest OpenSearch cluster.
make cluster.build cluster.start - Run all integration tests.
make test-integ race=true - Stop and clean containers.
make cluster.stop cluster.clean
By default, make cluster.start launches a 3-node cluster where every node has the same resources and roles (cluster_manager,data,ingest). The targets below let you customize the cluster for testing weighted round-robin routing, role-based request routing, and other behavior that only surfaces with non-uniform nodes.
Scale the running cluster to a different number of nodes without rebuilding:
make cluster.scale.1 # Single-node cluster
make cluster.scale.2 # 2-node cluster
make cluster.scale.3 # Full 3-node cluster (default)
Override files let you change CPU limits and node roles independently. Each target writes a Docker Compose override file under .ci/opensearch/ that is automatically merged with docker-compose.yml on the next cluster.build or cluster.start. The override files are not checked into source control.
The override file paths are:
| Override | File |
|---|---|
| CPU limits | .ci/opensearch/docker-compose.cpu-override.yml |
| Node roles | .ci/opensearch/docker-compose.roles-override.yml |
These are standard Docker Compose files. You can hand-edit them for custom configurations (e.g., different CPU ratios or role combinations not covered by the Make targets), or remove individual files to selectively reset one dimension while keeping the other:
# Remove only the CPU override, keep roles
rm .ci/opensearch/docker-compose.cpu-override.yml
# Remove only the roles override, keep CPU limits
rm .ci/opensearch/docker-compose.roles-override.yml
Set per-node CPU limits so the client's weighted round-robin allocates proportional traffic. The deploy.resources.limits.cpus value is reported by each node via GET /_nodes/os as allocated_processors, which the client uses to compute connection weights.
# Balanced weights [1,1,2]
make cluster.heterogeneous.cpu.1 # node1=2, node2=2, node3=4 CPUs
# Skewed weights [1,2,4]
make cluster.heterogeneous.cpu.2 # node1=1, node2=2, node3=4 CPUs
Assign different roles to each node so the client's role-based routing policy can direct requests to the correct nodes (e.g., bulk requests to ingest-capable nodes, search requests to data nodes):
make cluster.heterogeneous.roles # node1=cluster_manager+ingest, node2=data+ingest, node3=data
CPU and role overrides are independent files and can be combined. Set both, then rebuild:
make cluster.heterogeneous.cpu.1 cluster.heterogeneous.roles
make cluster.stop cluster.clean cluster.build cluster.start
Verify the resulting configuration:
curl -sk 'https://admin:myStrongPassword123%21@localhost:9200/_nodes/http,os?pretty' \
| jq '.nodes[] | {name, roles: .roles, processors: .os.allocated_processors}'
Remove all override files and return to the default homogeneous 3-node cluster:
make cluster.homogeneous
make cluster.stop cluster.clean cluster.build cluster.start
The cluster supports any published OpenSearch Docker image version. Always clean before switching versions to avoid stale data or cached images:
make cluster.stop
OPENSEARCH_VERSION=3.6.0 make cluster.clean cluster.build cluster.start
make test-integ
Set SECURE_INTEGRATION=false to disable TLS and basic auth:
SECURE_INTEGRATION=false OPENSEARCH_VERSION=3.6.0 make cluster.clean cluster.build cluster.start
Use make cluster.status to display cluster health, node info, Docker container state, and index/shard details. It auto-detects whether the cluster is running in secure or insecure mode.
make cluster.status
The cluster.latency.* targets inject artificial network delay into running Docker containers using Linux tc (traffic control). This lets you test the client's RTT-bucketed connection scoring and congestion-aware routing under realistic network topologies without leaving your development machine.
All latency targets use tc qdisc with netem delay and require the cluster to be running. They operate on the Docker container network interfaces directly.
Four predefined profiles cover common deployment topologies:
# Asymmetric -- simulates 3 availability zones at different distances
make cluster.latency.asymmetric # node1=0ms, node2=50ms ±5ms, node3=150ms ±15ms
# Symmetric -- single data center, all nodes equidistant
make cluster.latency.symmetric # all nodes 1ms ±100us
# Bimodal -- 2 local nodes + 1 remote node
make cluster.latency.bimodal # node1=1ms, node2=1ms, node3=20ms
# Graduated -- wide spread across 3 tiers
make cluster.latency.graduated # node1=1ms, node2=10ms, node3=100ms
# Show current tc qdisc rules on each node
make cluster.latency.show
# Remove all artificial latency
make cluster.latency.clear
The cmd/demo binary exercises the client with affinity routing enabled, printing live per-node request distribution and latency statistics. It connects to a running cluster and generates configurable workloads.
make demo.build # Build bin/demo
make demo.run # Build and run with default affinity-visible settings
demo.run connects to all three local nodes with request routing, node discovery, and periodic stats output. The binary accepts flags for concurrency, workload type, fanout, and report output -- run bin/demo -help for the full list.
The verify.* targets automate end-to-end validation of the client's affinity routing behavior. Each target applies a latency profile, runs a workload through the demo binary, and writes a JSON report to reports/verify/.
Run a single latency profile with the default search workload:
make verify.symmetric # Equal distribution under symmetric latency
make verify.asymmetric # Tier equalization under asymmetric latency (0/50/150ms)
make verify.bimodal # 2-local + 1-remote equalization (1/1/20ms)
make verify.graduated # Wide tier-spread equalization (1/10/100ms)
# Compare fanout=1 vs fanout=2 vs fanout=3 under bimodal latency
make verify.fanout
# Compare search, read-write, bulk-write, and write-only across all latency profiles
make verify.workloads
# Verify CPU-proportional distribution with heterogeneous CPUs [1,2,4]
make verify.hetero.cpu
Run every combination of latency profile, workload, fanout, and hardware configuration:
make verify.matrix
Reports are written to reports/verify/. The full matrix takes several minutes to complete.
# Print a tabular summary of all reports (name, total, RPS, P50, P99, connections)
make verify.summary
# Validate that distribution expectations are met
make verify.check
When a GitHub Actions check fails, searching the raw logs for keywords like "fail" or "error" produces many false positives from test names that legitimately contain those words. The gh.fail.* Makefile targets use the gh CLI to fetch only the failed steps and extract Go's structured failure markers (--- FAIL:, FAIL\t, panic:), giving clean output with no noise.
All targets auto-detect the most recent failed run on the current branch. Requires the GitHub CLI (gh) to be installed and authenticated.
Show which jobs failed and a deduplicated list of failed test names:
make gh.fail.summary
Example output:
Run 22722089811 -- failed jobs:
✗ integ-test-compat (true, 2.19.5)
✗ integ-test-compat (false, 3.6.0)
Failed tests:
--- FAIL: TestShardExactRouting_FullPipeline_Integration (2.56s)
--- FAIL: TestShardExactRouting_FullPipeline_Integration/pipeline/routing=0 (0.01s)
--- FAIL: TestShardExactRouting_FullPipeline_Integration/pipeline/routing=order-12345 (0.01s)
Show all --- FAIL:, FAIL, and panic: lines with the job/step/timestamp prefix stripped:
make gh.fail
Show 5 lines of context before each --- FAIL: or panic: line. This surfaces the assertion error messages (e.g., Error:, Messages:, Error Trace:) that explain why the test failed:
make gh.fail.context
Dump the complete log output from all failed steps (raw gh run view --log-failed):
make gh.fail.full
All targets accept a GH_RUN_ID override to inspect any run, not just the most recent failure:
make gh.fail.summary GH_RUN_ID=22705106053
To find run IDs, list recent checks:
make gh.checks # All checks on the current branch
make gh.checks.failed # Only failed checks
The cmd/osgen tool generates typed path builder structs (internal/path/) and API consumer files (opensearchapi/, plugins/) from the published OpenSearch API specification. It reads x-operation-group, x-version-added, x-version-deprecated, x-version-removed, and x-error-responses extensions from the spec to produce version-aware Go source.
The opensearchapi/ package is the code-generated v5 API surface, produced by cmd/osgen from the spec. See opensearchapi/README.md for usage and UPGRADING.md for migration guidance.
PRs that edit
*_gen.gowill be rejected. These files are generated. To change them, send a PR againstcmd/osgen(the generator) or against the OpenSearch API specification (the input).
Spec-extension status. Some
x-*extensions osgen reads (notablyx-error-responses) are still in flight upstream. Until they merge, the build pulls from the localopensearch-openapi.yamlcheckout in this repo rather than the published spec. Once upstream catches up, the local file goes away.
The x-error-responses extension on a spec operation declares the categories of partial failure that operation can produce. For each category named on an operation, osgen emits:
- A typed Go error (e.g.
*PartialBulkError,*PartialSearchError,*ShardFailureError,*MultiSearchItemError) decoded from the response body when that category fires. - A bit on
errmask.ErrorMask(PascalCase, e.g.errmask.BulkItems) plus the corresponding env-var token (bulk_items) so callers can suppress or surface it viaConfig.ErrorsorOPENSEARCH_GO_ERROR_MASK. - A per-Resp helper method on the operation's typed response (e.g.
BulkResp.BulkItemFailures(),SearchResp.SearchShardFailures()). These exist as engine machinery for the dispatch and are not the recommended call-site pattern; user docs point callers at afor/switchoveropensearchapi.Errors(err)instead. - A
PartialFailures(mask)aggregator on the same Resp.
Operations that declare two or more categories also get a per-op error container (e.g. *MSearchErrors) implementing Unwrap() []error, used when more than one category fires on a single response.
The user-facing partial-failure model and best-practices guidance live in opensearchapi/README.md and guides/usage-error_handling.md. They deliberately omit x-error-responses terminology because callers don't need to read the spec to use the resulting errors. The spec-driven mechanics are documented here and in cmd/osgen/README.md.
To regenerate (downloads the spec automatically if not cached):
make gen
To force a fresh spec download, remove the cached file first:
rm -f opensearch-openapi.yaml
make gen
By default, make gen includes all operations and fields across every OpenSearch version. Three Makefile variables scope the generated output to a version window:
| Variable | Default | Effect |
|---|---|---|
GEN_MIN_VERSION |
epoch |
Exclude operations removed before this floor |
GEN_MAX_VERSION |
latest |
Exclude operations added after this ceiling |
GEN_REMOVE_DEPRECATED |
epoch |
Treat operations deprecated at or before this as removed |
Override on the command line:
# Generate a client targeting only OpenSearch 2.x+, dropping deprecated APIs
make gen GEN_MIN_VERSION=2.0 GEN_REMOVE_DEPRECATED=2.0
# Generate a client for a specific version range
make gen GEN_MIN_VERSION=2.0 GEN_MAX_VERSION=2.99
# Strict less-than ceiling (excludes 3.0.0 itself)
make gen GEN_MAX_VERSION="<3.0.0"
Version flags accept an optional operator prefix (>=, >, <=, <). When omitted, --min-version defaults to >= and --max-version defaults to <=. The magic values epoch (no floor) and latest (no ceiling) mean "include everything".
When items are excluded by version filtering, breadcrumb comments are left in the generated code explaining the reason (e.g., // cat.masterPath: deprecated in OpenSearch 2.0.0 (treated as removed).). Breadcrumb visibility is configurable per category via -version-breadcrumb-* flags.
See cmd/osgen/README.md for the full flag reference and subcommand details.
To keep all the code in a certain uniform format, it was decided to use some writing rules. If you wrote something wrong, it's okay, you can simply run the script to check the necessary files, and optionally format the content. But keep in mind that all these checks are repeated on the pipeline, so it's better to check locally.
To check the markdown files, run the following command:
make lint.markdown
To check all go files, run the following command:
make linters
To get the repository test coverage, run the following command:
For the results to be display in your terminal:
make coverage
For the results to be display in your browser:
make coverage-html
You can import the OpenSearch project into GoLand as follows:
- Select File | Open
- In the subsequent dialog navigate to the ~/go/src/opensearch-go and click Open
After you have opened your project, you need to specify the location of the Go SDK. You can either specify a local path to the SDK or download it. To set the Go SDK, navigate to Go | GOROOT and set accordingly.
To improve your vim experience with Go, you might want to check out fatih/vim-go. For example it correctly formats the file and validates it on save.