[RFC] Adopt OpenAPI Overlay Specification for Distribution-Specific API Filtering

[Hailong-am]

[RFC] Adopt OpenAPI Overlay Specification for Distribution-Specific API Filtering

Is your feature request related to a problem? Please describe

The OpenSearch API specification currently embeds vendor-specific distribution annotations (x-distributions-excluded, x-distributions-included) directly into the spec source files. This was introduced in #510 and debated in #522, which has remained open and unresolved for 2 years.

Problems with the current approach:

1. Spec pollution -- 88 occurrences of x-distributions-excluded are scattered across security.yaml (82), indices.yaml (5), and cat.yaml (1). All target amazon-managed / amazon-serverless. No other vendor has contributed annotations.

2. Not a standard -- x-distributions-excluded is a custom extension. No OpenAPI ecosystem tool understands it natively. The repo maintains a custom OpenApiVersionExtractor to process it.

3. Only supports exclusion -- Cannot express vendor-specific additions (e.g., AOS's /_ultrawarm/* and /_cold/* endpoints that don't exist in OSS).

4. Scales poorly -- If Aiven, Oracle OCI, Instaclustr, and other managed providers each add their exclusions inline, the spec becomes unreadable.

5. Increases contributor friction -- Spec contributors must understand vendor deployment details to avoid breaking x-distributions-* annotations.

Describe the solution you'd like

Adopt the OpenAPI Overlay Specification (v1.1.0, Jan 2026) as the mechanism for expressing distribution-specific API surface differences. Overlay is an official companion specification from the OpenAPI Initiative, released Oct 2024 -- two months after the #522 discussion concluded without resolution.

What is OpenAPI Overlay?

The Overlay Specification defines a document format for information that augments an existing OpenAPI description yet remains separate from it. An Overlay contains an ordered list of actions (using RFC 9535 JSONPath targeting) that update, remove, or copy elements in a target OpenAPI document.

Key properties:

• Official OAI standard (not a custom extension)
• Supported by multiple production tools (Speakeasy, Bump.sh, openapi-overlay-js)
• Uses JSONPath (RFC 9535) for targeting -- resilient to spec reordering
• Supports both removal AND addition of API elements
• Each overlay is a separate file with independent version control

Proposed Architecture

opensearch-api-specification/
  spec/                                  <- CLEAN, vendor-neutral (no x-distributions-*)
    namespaces/
      security.yaml
      indices.yaml
      cat.yaml
      ...
  overlays/                              <- NEW: distribution-specific overlays
    amazon-managed.overlay.yaml          <- Maintained by AWS contributors
    amazon-serverless.overlay.yaml       <- Maintained by AWS contributors
    README.md                            <- How to add your distribution's overlay
  tools/
    src/merger/
      OpenApiVersionExtractor.ts         <- Simplified (distribution logic removed)

Example: overlays/amazon-managed.overlay.yaml

overlay: 1.0.0
info:
  title: Amazon Managed OpenSearch Service - API surface overlay
  version: 2026.08.1
  description: |
    Operations not available on Amazon Managed OpenSearch Service domains.
    Apply this overlay to produce an AOS-specific OpenAPI spec.
extends: ../opensearch-openapi.yaml
actions:
  # Security plugin management APIs -- not exposed on managed
  - target: $.paths['/_plugins/_security/authinfo'].get
    remove: true
  - target: $.paths['/_plugins/_security/authinfo'].post
    remove: true
  - target: $.paths['/_plugins/_security/dashboardsinfo'].get
    remove: true
  - target: $.paths['/_plugins/_security/dashboardsinfo'].post
    remove: true
  - target: $.paths['/_plugins/_security/health'].get
    remove: true
  - target: $.paths['/_plugins/_security/health'].post
    remove: true
  - target: $.paths['/_plugins/_security/tenantinfo'].get
    remove: true
  - target: $.paths['/_plugins/_security/tenantinfo'].post
    remove: true
  - target: $.paths['/_plugins/_security/whoami'].get
    remove: true
  - target: $.paths['/_plugins/_security/whoamiprotected'].get
    remove: true

  # Cluster-level index operations -- blocked on managed
  - target: $.paths['/_cache/clear'].post
    remove: true
  - target: $.paths['/_forcemerge'].post
    remove: true
  - target: $.paths['/_segments'].get
    remove: true
  - target: $.paths['/_settings'].get
    remove: true
  - target: $.paths['/{index}/_segments'].get
    remove: true

  # Cat
  - target: $.paths['/_cat/nodeattrs'].get
    remove: true

Example: overlays/amazon-managed.overlay.yaml (showing additive capability)

overlay: 1.0.0
info:
  title: Amazon Managed OpenSearch Service - API surface overlay
  version: 2026.08.1
  description: |
    API modifications for Amazon Managed OpenSearch Service (AOS).
    Includes both excluded OSS APIs and AOS-specific additions.
extends: ../opensearch-openapi.yaml
actions:
  # --- Exclusions (same as above, abbreviated) ---
  - target: $.paths['/_plugins/_security/authinfo']
    remove: true
  # ... (other exclusions)

  # --- AOS-specific additions (not in OSS) ---
  # UltraWarm tier management
  - target: $.paths
    description: AOS UltraWarm tier APIs
    update:
      /_ultrawarm/migration/{index}/_warm_to_cold:
        post:
          operationId: ultrawarm.warm_to_cold.0
          description: Migrates an index from warm storage to cold storage.
          # ... full operation definition
      /_cold/migration/{index}/_cold_to_warm:
        post:
          operationId: cold.cold_to_warm.0
          description: Migrates an index from cold storage back to warm storage.
          # ... full operation definition

Usage

Producing a vendor-filtered spec becomes a single command:

# Using Speakeasy CLI
speakeasy overlay apply -s opensearch-openapi.yaml -o overlays/amazon-managed.overlay.yaml \
  > opensearch-openapi-amazon-managed.yaml

# Or using openapi-overlay-js (integrates into existing Node.js tooling)
npx openapi-overlay apply opensearch-openapi.yaml overlays/amazon-managed.overlay.yaml

# Or integrated into existing npm scripts
npm run build:spec -- --overlay=amazon-managed

Migration Plan

Phase 1: Add Overlays (non-breaking, backward-compatible)

1. Programmatically generate overlay files from existing x-distributions-excluded annotations
2. Add overlays/ directory to the repo
3. Add CI job: apply overlay + validate the filtered spec matches what OpenApiVersionExtractor currently produces
4. Document the overlay approach in CONTRIBUTING.md

Phase 2: Remove In-Spec Annotations

1. Remove all x-distributions-excluded / x-distributions-included from spec YAML files
2. Simplify OpenApiVersionExtractor.ts -- remove #exclude_per_distribution() and #remove_keys_not_matching_distribution()
3. CI produces filtered specs by applying overlays at build time
4. Continue publishing both clean and vendor-filtered specs in GitHub releases

Phase 3: Ecosystem Integration

1. Add CLI flag to spec merger: npm run build -- --overlay=<name>
2. Publish per-distribution specs in releases: opensearch-openapi-3.0.yaml, opensearch-openapi-3.0-amazon-managed.yaml, etc.
3. Document how other vendors (Aiven, Instaclustr, etc.) can contribute their own overlay files
4. Add CODEOWNERS rules so each vendor approves changes to their overlay

Comparison: Current vs. Proposed

Dimension	Current (x-distributions-*)	Proposed (Overlay)
Spec neutrality	Vendor annotations in source	Source is clean; overlays are separate
Standard	Custom x- extension	Official OAI companion spec (v1.1.0)
Scalability	Every vendor adds inline	Each vendor owns one file
Bidirectional	Exclusion only	Both removal AND addition
Tooling	Custom TypeScript extractor	Standard ecosystem + custom
Auditability	Scattered across many YAML files	One file per vendor
Merge conflicts	High (annotations in same lines as spec)	Low (overlay targets by path name)
Contributor friction	Must know vendor details	Only describe OSS surface

Addressing Concerns from #522

@dblock: "The spec is changing fast"

Two years later, the spec is substantially more complete. Overlay JSONPath targets reference paths by name ($.paths['/_plugins/_security/authinfo']), not by position. Spec reordering or new API additions do not break existing overlays. Only actual path renames require overlay updates -- and CI catches those.

@dbwiddis: "Apps need to query filtered specs"

Overlays produce concrete, filtered OpenAPI documents at build time. The published release artifacts include per-vendor specs. Services host their filtered spec at a well-known URL. This satisfies the app/front-end need without runtime complexity.

@reta: "Cloud vendor's responsibility to maintain"

Each overlay file is owned by its respective vendor contributors. With CODEOWNERS, changes to overlays/amazon-*.overlay.yaml require AWS contributor approval. The base spec is fully vendor-neutral.

@reta: "Simplifies spec completion"

Removing x-distributions-* means contributors describe only the OSS API surface. No vendor knowledge required. The cognitive load for new contributors drops significantly.

Alternatives Considered

Alternative	Why Not
Keep x-distributions-excluded as-is	Stalemate for 2 years; non-standard; doesn't support additions; won't scale
Vendor forks of the repo	Divergence, merge hell, duplicated work
Separate YAML file (not Overlay format)	Requires custom tooling; reinvents what Overlay standardizes
Runtime /_spec endpoint filtering	Complex; not all distributions can serve dynamic specs

Tooling Availability

Tool	Type	Status
Speakeasy CLI	Go binary	Production, overlay apply command
openapi-overlay-js	Node.js library	Production, direct integration into tools/
Bump.sh	SaaS + CLI	Production
Redocly CLI	Node.js	Overlay support via decorators

The repo's tooling is TypeScript/Node.js -- openapi-overlay-js integrates natively.

Additional Context

• OpenAPI Overlay Spec v1.0.0 released Oct 2024 (announcement)
• v1.1.0 released Jan 2026 -- adds copy action
• The spec uses RFC 9535 JSONPath -- the same standard used by Kubernetes, NIST, and others
• This proposal directly resolves #522 and can be referenced from it
• The approach is already documented as a best practice: OpenAPI Overlays for Adding Tool-Specific Content Without Polluting the Spec

Call for Feedback

I'd like input on:

1. Should overlays live in this repo (under overlays/) or in vendor-specific repos?
2. Should Phase 1 include overlays for other providers (Aiven, Instaclustr) or start with AWS only?
3. Preference for overlay tooling: integrate openapi-overlay-js into existing tools/, or use Speakeasy CLI in CI?
4. Timeline: is this ready for a PR, or does it need further design discussion?

/cc @dblock @reta @dbwiddis