Automax

Automax is a Python, YAML-driven SSH automation engine for running resumable job/task/step/substep workflows against remote systems.

Operational definitions are external to the Python sources:

• job YAML
• inventory YAML
• variables YAML
• secrets YAML
• optional external plugins

Documentation site

The GitHub Pages site is the full public manual, not just a README mirror. It includes quickstarts, concepts, guides, CLI reference, security notes, state and resume documentation, artifacts and the generated builtin plugin reference.

Build it locally with:

NO_MKDOCS_2_WARNING=1 mkdocs build --strict

Start from docs/index.md or browse the published site configured in mkdocs.yml.

Core model

Job
  Task[]
    Step[]
      Substep[]

A step opens one fresh SSH connection per target and reuses it for all its substeps. Runtime context is not kept in a shell session: values are passed through the Automax context, registered outputs and state store.

YAML flow control

Job YAML supports engine-level flow-control substeps in addition to plugin substeps:

• if / list-style if / else and switch / case / default for branching;
• for / in / do for loops over lists, including values returned by plugins;
• retry / do for retrying a nested block;
• set / let for flow values;
• echo for operator-visible messages;
• assert for deliberate boolean checks;
• sleep for explicit pauses without shell commands;
• noop for explicit no-op branches or placeholders;
• fail for deliberate failures;
• block for grouping nested substeps under one when or logical id;
• try / rescue / always for recovery and teardown;
• break / continue inside loops.

Example:

- id: grade_branch
  if:
    - when: "{{ x < 50 }}"
      then:
        - id: grade_f
          echo: F
    - when: "{{ x < 80 }}"
      then:
        - id: grade_b
          echo: B
    - else:
        - id: grade_a
          echo: A

Use register or set / let values in later conditions through outputs.*, vars.* or direct variable names in the same target execution path.

Install

pip install -e .

Optional database drivers:

pip install -e '.[postgres]'
pip install -e '.[mysql]'
pip install -e '.[oracle]'
pip install -e '.[database]'

Quick smoke

Install development extras before running the test suite. Installing only .[docs] is enough for MkDocs, but it intentionally does not install pytest.

python -m pip install -e '.[dev]'
python -m compileall -q src tests scripts
python -m pytest -q
scripts/release-check.sh

python -m automax validate \
  --job examples/jobs/local-smoke.yaml \
  --inventory examples/inventory/local.yaml

For a local-only plan:

python -m automax plan \
  --job examples/jobs/local-smoke.yaml \
  --inventory examples/inventory/local.yaml

For machine-readable output:

python -m automax plan \
  --job examples/jobs/local-smoke.yaml \
  --inventory examples/inventory/local.yaml \
  --format=json

python -m automax schema export --kind job --format=json --output /tmp/automax-job.schema.json

Run a job:

automax run \
  --job /srv/automax/jobs/deploy.yaml \
  --inventory /srv/automax/inventory/prod.yaml \
  --vars /srv/automax/vars/prod.yaml \
  --secrets /srv/automax/secrets/prod.yaml \
  --state-dir /var/lib/automax/runs

When selected remote substeps use sudo and the target account requires a password, keep sudo authentication explicit instead of configuring NOPASSWD on the target:

export AUTOMAX_SUDO_PASSWORD='...'
automax run --job job.yaml --inventory inventory.yaml --sudo-password-env AUTOMAX_SUDO_PASSWORD
automax capabilities install --job job.yaml --inventory inventory.yaml --sudo-password-env AUTOMAX_SUDO_PASSWORD

Inspect an existing run:

automax runs show <run-id> --state-dir /var/lib/automax/runs
automax runs show <run-id> --state-dir /var/lib/automax/runs --failed
automax runs show <run-id> --state-dir /var/lib/automax/runs --server web01

Resume a failed run:

automax resume <run-id> --state-dir /var/lib/automax/runs
# If rerun substeps need sudo authentication:
automax resume <run-id> --state-dir /var/lib/automax/runs --sudo-password-env AUTOMAX_SUDO_PASSWORD

Resume from an explicit checkpoint:

automax resume <run-id> \
  --state-dir /var/lib/automax/runs \
  --from task.install:step.packages:substep.install_nginx

Export run evidence after a successful or failed run:

automax runs export <run-id> --state-dir /var/lib/automax/runs --output evidence.md
automax runs events <run-id> --state-dir /var/lib/automax/runs --output events.jsonl
automax runs bundle <run-id> --state-dir /var/lib/automax/runs --output automax-evidence.zip

Collect runtime evidence from a real SSH lab host:

AUTOMAX_SSH_HOST=web01.example.com \
AUTOMAX_SSH_USER=deploy \
scripts/runtime-evidence.sh
scripts/runtime-validation-packs.sh --pack debian-like

Stored runs also write an append-only JSONL event stream at .automax/runs/<run-id>/events.jsonl and expose it with automax runs events.

Task-level and step-level restart points are also accepted:

automax run --job job.yaml --inventory inventory.yaml --from task.install
automax run --job job.yaml --inventory inventory.yaml --from task.install:step.packages

Job inspection and recovery

Before running or resuming a job, operators can generate one masked review pack for the exact job, inventory, vars, secrets, target filters and tag filters under review:

automax review --job job.yaml --inventory inventory.yaml --output review.md
automax review --job job.yaml --inventory inventory.yaml --secrets secrets.yaml --format=json

The review pack combines strict validation, selected inventory, masked secret status, masked rendered vars, capability requirements, check-mode preview, diff/manual preview coverage and ready-to-copy operator commands. Review reports that use --secrets are printed to stdout only; saved reports are reserved for inputs that do not load a secrets file. The individual inspection commands remain available when a narrower view is needed:

automax inventory show --job job.yaml --inventory inventory.yaml --limit web
automax secrets check --job job.yaml --inventory inventory.yaml --secrets secrets.yaml
automax vars render --job job.yaml --inventory inventory.yaml --secrets secrets.yaml
automax plan --check --job job.yaml --inventory inventory.yaml
automax plan --diff --job job.yaml --inventory inventory.yaml
automax commands render --job job.yaml --inventory inventory.yaml --limit web01

Manual command rendering is designed for failed-step recovery: copy the command, fix the host manually, then restart from the next checkpoint with automax resume --from or automax run --from. See docs/guides/job-inspection-and-recovery.md for the full operator workflow.

Approval gates let operators bind a reviewed, secret-free plan to a JSON approval file before execution:

automax approval create --job job.yaml --inventory inventory.yaml --approved-by change-1234 --output approval.json
automax approval verify --job job.yaml --inventory inventory.yaml --approval approval.json
automax run --job job.yaml --inventory inventory.yaml --secrets secrets.yaml --approval approval.json

External files

Job, inventory, variable and secret files can live anywhere. They do not need to be inside this repository.

--job        /path/to/job.yaml
--inventory  /path/to/inventory.yaml
--vars       /path/to/vars.yaml
--secrets    /path/to/secrets.yaml
--state-dir  /path/to/run-state

Builtin plugins

The public DSL exposes canonical plugin names only. Run:

automax plugins list
automax plugins describe fs.file.template
automax plugins coverage --format=markdown --output plugin-coverage.md
automax plugins coverage --strict
automax plugins audit

plugins coverage renders a builtin/external plugin matrix with category, remote-session, check-mode, dry-run, manual preview, diff preview and tool requirement coverage. --strict turns metadata and preview coverage failures into a release gate. External plugin SDK commands create, validate, package, verify and index plugin sources or package ZIPs before they are loaded with --plugin-path:

automax plugins init company.echo --output ./plugins
automax plugins check ./plugins
automax plugins package ./plugins --output dist/company-plugins.zip
automax plugins verify-package dist/company-plugins.zip
automax plugins index dist/company-plugins.zip --output dist/automax-plugins-index.json
automax plugins install dist/company-plugins.zip --dest .automax/plugins
automax plugins installed --dest .automax/plugins
automax plugins describe company.echo --plugin-path ./plugins
automax plugins describe company.echo --plugin-path dist/company-plugins.zip
automax plugins describe company.echo --plugin-path .automax/plugins

Public families:

commands:      command.local.run, command.remote.run
flow:          if/then/else, switch/case/default, retry/do, for/in/do, block, set/let, echo, assert, sleep, noop, fail, try/rescue/always, break/continue
filesystem:    fs.*
data:          data.archive.*, data.compression.*, data.download.*, data.transfer.*, data.backup.*, data.restore.*
database:      database.<engine>.check, database.<engine>.query
containers:    container.image.*, container.container.*, container.compose.*
orchestration: kubernetes.*, helm.*
identity:      identity.user.*, identity.group.*
network:       network.connectivity.*, network.dns.*, network.firewall.*, network.http.*, network.link.*, network.route.*
os/packages:   os.*, os.package.*
security:      security.*
storage:       storage.*
system:        system.host.*, system.service.*, system.systemd.*, system.kernel.*, system.process.*, system.cron.*, system.journal.*, system.log.*
devices:       device.udev.*
notifications: notify.mail.send

See docs/plugins/ for detailed examples of every builtin plugin. Use scripts/release-check.sh before tagging a release to verify generated plugin docs, runbook indexes, policy/risk preflight, SQLite database smoke, plugin coverage gates, tests, documentation and packages together.

Variables and secrets

Variables are external and can be overridden by CLI:

automax run --job job.yaml --inventory inventory.yaml --vars vars.yaml --var app_version=2.1.0

Secrets support local sources and common external secret managers through controller-side providers.

secrets:
  ssh_user:
    provider: env
    name: AUTOMAX_SSH_USER

  ssh_key_file:
    provider: file
    path: ~/.ssh/id_ed25519.path

  deploy_token:
    provider: vault
    path: secret/prod/app
    field: deploy_token

  db_password:
    provider: aws_secrets_manager
    secret_id: prod/db
    json_key: password
    region: eu-west-1

Dynamic inventory providers are available when targets come from a generated file, local command or HTTP inventory service. See the GitHub Pages manual for details.

Strategy, tags and failure policy

Strategies are scoped at job/task/step level:

strategy:
  mode: parallel
  max_parallel: 5

strategy:
  mode: rolling
  batch_size: 2
  pause_between_batches: 10

Tags are filtered from the CLI:

automax run --job job.yaml --inventory inventory.yaml --tags install --skip-tags dangerous

Failure policy is declarative:

failurePolicy:
  onFailure: stop_job      # stop_job, stop_task, stop_host, continue
  onUnreachable: stop_host
  maxFailedHosts: 1

Advanced error policy can downgrade expected non-zero command results to warnings after stdout/stderr normalization:

errorPolicy:
  acceptedRc: [1, 2, 3]
  expected:
    - stream: combined
      pattern: "PRVF-5436.*NTP"
      reason: "Expected Oracle RAC precheck diagnostic"
  fail:
    - stream: combined
      pattern: "ORA-[0-9]+"
  unmatched: fail
  acceptedStatus: warning

State store

Each run gets a generated run-id. The state store is local SQLite by default:

.automax/runs/<run-id>/state.sqlite

It records run paths, checkpoints, target statuses, outputs and events. This is required for audit and restart from task/step/substep.

SSH smoke

Run the extended real SSH smoke against an operator-provided host:

AUTOMAX_SSH_HOST=192.0.2.10 \
AUTOMAX_SSH_USER=deploy \
AUTOMAX_SSH_KEY_FILE=~/.ssh/id_ed25519 \
AUTOMAX_SSH_HOST_KEY_POLICY=reject \
./scripts/ssh-smoke.sh

The default smoke is non-destructive and covers remote commands, filesystem, archive, transfer, flow control, read-only checks and artifact capture. Optional package manager, systemd and user/group/process checks are enabled with explicit environment variables. See docs/guides/ssh-smoke.md.

External plugins can be loaded with:

automax run --job job.yaml --inventory inventory.yaml --plugin-path /opt/automax/plugins

Documentation site

Build the documentation locally:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[docs]'
NO_MKDOCS_2_WARNING=1 mkdocs build --strict

Alternatively, when the package is not installed in editable mode, install the documentation requirements explicitly before running MkDocs:

python -m pip install -r requirements-docs.txt
NO_MKDOCS_2_WARNING=1 mkdocs build --strict

GitHub Pages publishing is handled by .github/workflows/docs.yml. Configure the repository Pages source as GitHub Actions.

Development tooling

Automax uses Ruff and pre-commit for local development checks:

python -m pip install -e '.[dev]'
python -m ruff check src tests scripts
pre-commit install

Operator tooling

Create an external job workspace:

automax init ./company-automation

Validate job definitions strictly before running them:

automax validate --strict --job jobs/local-smoke.yaml --inventory inventory/local.yaml

Check the controller environment:

automax doctor

Python compatibility guardrails

Automax supports Python 3.9 and newer. Development and CI run an explicit compatibility guard to catch runtime-only Python 3.10+ constructs before they reach the Python 3.9 matrix job:

python scripts/check-python39-compat.py

Release packages can be validated with scripts/package-release-smoke.sh --output-dir dist/package-release-smoke.

A sample external plugin collection is available in examples/external-plugins/.

Generate a release readiness summary with scripts/release-readiness-report.sh --output-dir dist/release-readiness.