Skip to content

runbook.md

Latest commit

 

History

History
208 lines (139 loc) · 4.79 KB

runbook.md

File metadata and controls

208 lines (139 loc) · 4.79 KB
Raw

Runbook

Operational procedures for projects managed by quartzctl.

Commands assume you are in the target project root with quartz.yaml present. Use --config <path> when operating from another directory.

Procedure 1 - Preflight A Project

quartz check
quartz render --out ./out/quartz.generated.yaml

Review the rendered file for expected name, dns.domain, provider settings, stage ids, and stage paths.

Procedure 2 - Run Or Resume A Full Install

quartz install
quartz install --yes

If the run failed at a known stage:

quartz install --resume-from <stage> --yes

If OpenTofu needs deferred actions for a specific install:

quartz install --allow-deferral --yes

Procedure 2A - Confirm Application Delivery Inputs

Before a full install, render and confirm any app-owned delivery settings:

quartz render --out ./out/quartz.generated.yaml

Check that each application includes:

  • repo_url pointing at the app repository
  • path pointing at the app-owned manifest entrypoint (defaults to deploy)
  • settings.post_deploy with the expected enablement / gate policy

Quartz treats gitops.apps as optional and deprecated for application delivery. If it is omitted, that is expected.

Procedure 3 - Plan Or Apply A Single Stage

quartz tofu plan --stage <stage> --init
quartz tofu apply --stage <stage> --init

Use this for targeted validation before a full quartz install. The --init flag initializes the stage backend first.

Procedure 4 - Inspect Outputs

quartz tofu output --stage <stage> --init

Use outputs to confirm cross-stage values before debugging downstream failures.

Procedure 5 - Repair Missing State With Import

When a resource exists in the cloud but is missing from OpenTofu state:

quartz tofu import --stage <stage> ADDRESS ID --init

Flag form is also supported:

quartz tofu import --stage <stage> --address ADDRESS --id ID --init

Then run:

quartz tofu plan --stage <stage>

Procedure 6 - Release A Stale Lock

When OpenTofu reports a stale lock:

quartz tofu force-unlock --stage <stage> LOCK_ID --init

Flag form:

quartz tofu force-unlock --stage <stage> --lock-id LOCK_ID --init

Prefer this command to manual backend lock-table edits.

Procedure 7 - Inspect Or Remove State Entries

quartz tofu state list --stage <stage> --init
quartz tofu state show --stage <stage> ADDRESS --init
quartz tofu state rm --stage <stage> ADDRESS --init

state show redacts sensitive values. state rm removes objects from state without destroying real infrastructure; use it only when the resource should no longer be managed.

Procedure 8 - Generate Or Refresh Kubeconfig

quartz login
quartz login --out /tmp/kubeconfig
export KUBECONFIG=./out/kubeconfig

Aliases:

quartz kubeconfig
quartz refresh-kubeconfig

Procedure 9 - Refresh External Secrets

quartz refresh-secrets

Alias:

quartz rs

Use after rotating a secret in the upstream secret manager.

Procedure 10 - Restart Workloads

Restart all deployments, daemonsets, and statefulsets:

quartz restart

Restart a specific workload:

quartz restart --kind deployment --namespace <namespace> --name <name>

--kind can be repeated.

Procedure 11 - Export Configured Kubernetes Resources

quartz export

Resources are selected by export.objects and written under export.path.

Procedure 12 - Validate Or Format Stages

quartz tofu validate --stage <stage>
quartz tofu format --stage <stage>
quartz tofu format-all

format is also available as fmt.

Procedure 13 - Clean / Teardown

quartz clean
quartz clean --yes

clean destroys stages in reverse dependency order. If a stage fails, cleanup continues and reports the failures. The remote backend is kept until all stage destroys succeed so operators can re-run cleanup with recoverable state.

Treat the final cleanup report as cumulative: earlier stage failures remain visible even when later stages continue to make progress. A remaining backend is not a leaked resource by itself; it means at least one stage still needs a successful destroy so state can stay available for recovery. Re-run quartz clean --yes after resolving the reported blocker.

While the cluster is still reachable, quartz clean now streams the Quartz pre-delete hook's cumulative breadcrumbs as they change, including degraded but still-progressing signals and the latest residual-resource snapshot reported by the hook.

Procedure 14 - Build And Deploy A Local CLI

For CLI developers:

cd quartzctl
mise run test
mise run deploy
quartz --version

mise run deploy builds the local binary and atomically replaces common local install locations.