Skip to content

yaml-reference.md

Latest commit

 

History

History
331 lines (272 loc) · 13.5 KB

yaml-reference.md

File metadata and controls

331 lines (272 loc) · 13.5 KB
Raw

YAML reference

Two files drive a native run: the test definition (-f) describes what to do; the config (-c) describes how much load to apply.

Both are validated against JSON Schemas before execution — errors point at the offending field path, not a raw parser dump. The schemas live in schema/ and can drive editor autocomplete via a modeline:

# yaml-language-server: $schema=https://raw.githubusercontent.com/Perfscale/perfscale/main/schema/test.schema.json

Test definition (-f test.yaml)

A single steps array. Each virtual user (VU) executes the whole list in a loop until the configured duration expires. An optional top-level import: inherits a base document — see Composing documents.

steps:
  - name: login                     # optional label used in log output
    use: std/http@v1                # required — action ID
    with:                           # action parameters (see below)
      method: POST
      url: https://api.example.com/login
      body:
        user: demo
    check:                          # optional inline assertions
      status: 200
      duration_ms_lt: 500
    outputs: login                  # optional — store output for later steps

Step fields

Field Required Description
use yes Action ID: std/http@v1, std/graphql@v1, std/tcp@v1, std/udp@v1, std/ws@v1, std/ws-connect@v1, std/ws-send@v1, std/ws-recv@v1, std/ws-ping@v1, std/ws-close@v1, std/grpc@v1, std/grpc-connect@v1, std/grpc-call@v1, std/grpc-stream-open@v1, std/grpc-stream-send@v1, std/grpc-stream-recv@v1, std/grpc-stream-close@v1, std/db-connect@v1, std/db-query@v1, std/db-tx-begin@v1, std/db-tx-commit@v1, std/db-tx-rollback@v1, std/db-close@v1, std/check@v1, std/sleep@v1, std/log@v1, std/file-read@v1, std/file-write@v1, std/child_process@v1, std/kill_process@v1, std/thresholds@v1 (short aliases http, graphql, tcp, udp, ws, ws-connect, ws-send, ws-recv, ws-ping, ws-close, grpc, grpc-connect, grpc-call, grpc-stream-open, grpc-stream-send, grpc-stream-recv, grpc-stream-close, db-connect, db-query, db-tx-begin, db-tx-commit, db-tx-rollback, db-close, check, sleep, log, file-read, file-write, child_process, kill_process, thresholds also work). uses: is accepted as an alias for use:
name no Human-readable label shown in log lines
with no Action parameters — see Actions
check no Assertions on this step's output — same keys as std/check@v1
outputs no Variable name to store the step output under
severity no std/thresholds@v1 only: what a violated gate becomes — fail (default; the run exits non-zero), warn, or info
message no std/thresholds@v1 only: custom label appended to the violation summary (interpolated)

Variables (${{ ... }})

Steps pass data to later steps through GitHub-Actions-style placeholders. A step stores its output under the name given by outputs:; any string value in a later step's with: or check: can then reference it:

Expression Resolves to
${{ name }} The whole stored output, stringified
${{ name.field }} One field of a stored object (e.g. .status, .body, .duration_ms)
${{ name.a.b }} Nested path — one JSON level per ., e.g. ${{ resp.headers.x-request-id }}
${{ __last__ }} / ${{ __last__.field }} The immediately preceding step's output — always available, no outputs: needed
${{ config.<name>.<field> }} Output of a config-file before: setup step (see Config)
${{ vars.<key> }} A static value from the config file's variables: block

For std/http@v1 the stored output is { "status": <int>, "body": <string>, "duration_ms": <float>, "headers": <object> } — header names lowercase, so a session/token header from response 1 can flow into request 2:

steps:
  - use: std/http@v1
    with: { url: "https://api.example.com/first" }
    outputs: r1
  - use: std/http@v1
    with:
      url: "https://api.example.com/second"
      headers:
        x-session: "${{ r1.headers.x-session }}"
steps:
  - name: login
    use: std/http@v1
    with:
      method: POST
      url: "https://api.example.com/token"
      body: { user: demo }
    outputs: auth                          # ← stored as `auth`

  - name: fetch profile
    use: std/http@v1
    with:
      url: "https://api.example.com/me"
      headers:
        authorization: "Bearer ${{ auth.body }}"   # nested values work
    check:
      body_contains: "${{ auth.body }}"            # check values too

  - use: std/log@v1
    with:
      message: "login took ${{ auth.duration_ms }}ms → ${{ auth.status }}"

Rules and edge cases:

  • Placeholders work in string values at any depth of with/check — nested objects (headers), array elements, bodies. Keys are never interpolated.
  • Whitespace inside the braces is ignored: ${{auth.status}} ≡ ${{ auth.status }}.
  • A missing variable or field resolves to an empty string — the run does not fail. Gate on the value with check: when absence should be an error.
  • Path segments descend one JSON object level per . — header names with dots in them cannot be addressed (rare; everything else works).
  • Placeholders are resolved per virtual user, per iteration — each VU sees the outputs of its own step chain, never another VU's.
  • Steps without any ${{ are executed as-is: the engine skips the interpolation pass entirely, so plain steps pay zero overhead for this feature.
  • YAML quoting: both plain (Bearer ${{ auth.body }}) and quoted ("${{ auth.body }}") scalars work; quote when the value starts with a character YAML treats specially ({, [, *, …).

Config (-c config.yaml)

vus: 10          # virtual users, default 1
duration: 5m     # "30s", "1m", "5m30s", "1h" — default "1m"

report:          # optional — forward the summary after the run
  url: http://localhost:7999
Field Default Description
vus 1 Concurrent virtual users
duration 1m Wall-clock run length; bare numbers are seconds
report.url — A perfscale serve base URL; the CLI --report flag overrides it
before [] One-time setup steps — see Setup and variables
after [] One-time teardown steps — see Teardown
variables {} Static values exposed to steps as ${{ vars.* }}
allow_process_actions false Let steps spawn/signal OS processes (std/child_process@v1, std/kill_process@v1). Fail-closed: a step list from an untrusted source cannot touch processes until you opt in
import — Base document to inherit from — see Composing documents

Composing documents: import

Both test definitions and configs accept a top-level import: naming a base document. The base loads first (it may import its own base, recursively), then the current document deep-merges on top: objects merge key-by-key, scalars and arrays (including steps:) are replaced by the importing side.

# team config — inherits the org-wide base, overrides one variable
import: ../shared/_base.yaml
variables:
  region: us

Three source forms:

# relative filesystem path (resolved against the importing file's directory)
import: ../shared/_base.yaml

# raw HTTP(S) URL — pin the ref in the path
import: "https://raw.githubusercontent.com/org/repo/v1.2.0/perf/config/_base.yaml"

# any git host (SSH or HTTPS remotes, self-hosted included)
import:
  git: git@gitlab.example.com:group/repo.git
  ref: v1.2.0          # tag, branch, or commit SHA
  file: perf/config/_base.yaml

Remote imports (URL and git) are fail-closed: they run only when the caller passes --allow-remote-import. The permission belongs to the caller because import resolution happens before the allow_file_actions / allow_process_actions gates — a remote base could otherwise grant itself those rights and pull in a std/child_process@v1 step. A document can never opt itself into the network.

Origins stay confined: a document fetched from a URL resolves relative imports against its own URL; a document from a git repo may only import files inside that same clone (../ escapes are rejected). A remote document can never read your local filesystem. Import cycles fail with the chain printed.

Git imports clone with --depth 1 through your system git (SSH keys and credential helpers apply) and cache under ~/.cache/perfscale/imports/. Tags and commit SHAs are immutable — cached forever. Branches revalidate against the remote after a short TTL, so ref: main follows the branch; --refresh-imports forces a refetch. The full guide lives in docs/core/imports.md.

Setup and variables

before: steps run once, in order, before any VU starts — for one-time setup like fetching a token or building a connection profile. Each before step is a normal step (use/with/outputs); its outputs name is exposed to every test step under the config namespace. variables: holds static values, exposed under vars.

vus: 50
variables:
  region: eu-west
before:
  - uses: std/http@v1
    with:
      method: POST
      url: "https://api.example.com/token"
      body: { user: demo, region: "${{ vars.region }}" }
    outputs: auth            # ← exposed to test steps as config.auth
# test.yaml
steps:
  - uses: std/http@v1
    with:
      url: "https://api.example.com/me"
      headers:
        authorization: "Bearer ${{ config.auth.body }}"   # from before step
        x-region: "${{ vars.region }}"                      # from variables
  • before runs regardless of --quiet (its failures always print). If any setup step fails, the run aborts before spawning VUs — a broken setup would make every iteration fail identically.
  • before steps see ${{ vars.* }} and earlier setup outputs (under their own outputs name). Test steps see config.* and vars.* but not each other's.
  • Interpolation always yields a string, so a numeric config value like ${{ config.fix_config.port }} reaches the action as "1111". Actions that take numbers accept the string form.

For the full lifecycle (including background processes started in before:), see Setup and teardown.

Teardown (after:)

after: steps run once after the load stops — on a normal finish, on a failed run, on a failed before:, and on Ctrl-C/SIGTERM alike. They see the same ${{ config.* }} and ${{ vars.* }} as test steps. Unlike before:, a failing teardown step is logged but does not abort the remaining ones (best-effort cleanup). The typical after: step is a std/kill_process@v1 for a server before: started:

allow_process_actions: true

before:
  - name: web
    uses: std/child_process@v1
    with:
      command: python3
      args: ["-m", "http.server", "8080"]
      port: 8080
      waitUntil: { port_open: 8080, timeout: 10s }
    outputs: web

after:
  - name: stop web
    uses: std/kill_process@v1
    with: { name: web, signal: TERM }

Processes still alive after the after: steps are stopped automatically, so the explicit kill is about a clean, timely stop rather than leak prevention. See std/child_process@v1 for the full parameter list (restart policy, output capture, waitUntil), examples/with-processes.config.yaml for a runnable setup, and Setup and teardown for the whole lifecycle (interrupts, auto-kill, data flow).

after: is also the home of run-level SLO gates: std/thresholds@v1 evaluates k6-style expressions (p95<500, rate<0.05, count==0) against the metrics the whole run collected, once, and fails the run (non-zero exit) when a severity: fail gate is violated:

after:
  - name: slo gate
    use: std/thresholds@v1
    with:
      db_query_duration: ["p95<500", "max<2000"]
      db_query_failed: ["rate<0.05"]
      db_errors: ["count==0"]
    severity: fail            # fail (default) | warn | info
    message: "checkout SLO"   # optional, interpolated

See std/thresholds@v1 for the expression grammar, metric kinds, and the thresholds field in the run summary JSON.

With --locust, the same config maps to locust's --users/--spawn-rate/--run-time. With --k6, load config lives in the script's own options block and the config file is ignored.

Validating without running: perfscale lint

Check files ahead of time — in CI, pre-commit hooks, or while writing them:

perfscale lint test.yaml config.yaml

Beyond schema validation, lint flags unknown and typo'd field names with did-you-mean suggestions (chek → check, vsu → vus, std/htp@v1 → std/http@v1), including per-action with: parameters. See CLI commands → lint.

Validation errors

perfscale validates before running. Examples of what you'll see:

error: schema validation failed:
  /steps/0 — every step must name an action: `use: std/http@v1` (or the `uses:` alias)

error: invalid YAML: found unexpected end of stream

Regenerate the schemas after changing the Rust types:

cargo run -p perfscale-core --example gen_schema

(CI's shipped_schemas_match_generated_ones test fails if schema/ goes stale.)