README.md

go-pipeline

go-pipeline is a Go library used for building and modifying Buildkite pipelines in golang. It's used internally by the Buildkite Agent to inspect and sign pipelines prior to uploading them, but is also useful for building tools that generate pipelines.

Installation

To install, run

go get -u github.com/buildkite/go-pipeline

This will add go-pipeline to your go.mod file, and make it available for use in your project.

Usage

Loading a pipeline from yaml

const aPipeline = `
env:
  MOUNTAIN: cotopaxi
  COUNTRY: ecuador

steps:
  - command: echo "hello world"
  - wait
  - command: echo "goodbye world"
`

p, err := pipeline.Parse(strings.NewReader(aPipeline))
if err != nil {
  panic(err)
}

pretty.Println(p)
// &pipeline.Pipeline{
//   Env: &ordered.Map[string,string]{
//     items: {
//       {Key:"MOUNTAIN", Value:"cotopaxi", deleted:false},
//       {Key:"COUNTRY", Value:"ecuador", deleted:false},
//     },
//     index: {"MOUNTAIN":0, "COUNTRY":1},
//   },
//   Steps: {
//     &pipeline.CommandStep{
//       Command:         "echo \"hello world\"",
//       Env:             {},
//       RemainingFields: {},
//     },
//     &pipeline.WaitStep{
//       Scalar:  "wait",
//       Contents: {},
//     },
//     &pipeline.CommandStep{
//       Command:         "echo \"goodbye world\"",
//       Env:             {},
//       RemainingFields: {},
//     },
//   },
//   RemainingFields: {},
// }

Marshalling to YAML or JSON

aPipeline := `...`
p, err := pipeline.Parse(strings.NewReader(aPipeline))
if err != nil {
  // ...
}

//... modify the pipeline

// Marshal to YAML
b, err := yaml.Marshal(p)
if err != nil {
  // ...
}

// Marshal to JSON
b, err := json.Marshal(p)
if err != nil {
  // ...
}

Caveats

The pipeline object model (Pipeline, Steps, Plugin, etc) have these caveats:

• It is incomplete: there may be fields accepted by the API that are not listed. Do not treat Pipeline, CommandStep, etc, as comprehensive reference guides for how to write a pipeline.
• It normalises: unmarshaling accepts a variety of step forms, but marshaling back out produces more normalised output. An unmarshal/marshal round-trip may produce different output.
• It is non-canonical: using the object model does not guarantee that a pipeline will be accepted by the pipeline upload API.

Notably, most of the structs defined by this module only contain the elements of a pipeline (and steps) necessary for the agent to understand, and are (at the time of writing) not comprehensive. Where relevant - that is, where there are more fields that are not included in the struct - the RemainingFields field is used to capture the remaining fields as a map[string]any. This allows pipelines to be loaded and modified without losing information, even if the pipeline contains fields that are not yet understood by the agent.

For example, the command step:

command: echo "hello world"
env:
  FOO: bar
  BAZ: qux
artifact_paths:
  - "logs/**/*"
  - "coverage/**/*"
parallelism: 5

would be represented in go as:

&pipeline.CommandStep{
  Command: `echo "hello world"`,
  Env: ordered.MapFromItems(
    ordered.TupleSS("FOO", "bar"),
    ordered.TupleSS("BAZ", "qux"),
  ),
  RemainingFields: map[string]any{
    "artifact_paths": []string{"logs/**/*", "coverage/**/*"},
    "parallelism": 5,
  },
}

This go struct would be marshaled back out to YAML equivalent to the original input.

Checkout

The checkout block configures git checkout behaviour for a pipeline or a command step. The simplest case opts a step out of checkout entirely:

steps:
  - command: echo "no git checkout for me"
    checkout:
      skip: true

Checkout.Skip is a *bool so the model preserves the difference between skip: true, skip: false, and an absent skip. This matters because skip: false at the step level explicitly overrides any pipeline-level or agent-level default that would otherwise skip checkout, while an absent skip inherits whatever default applies. Round-trips preserve the distinction; skip: false does not collapse to an empty mapping.

A pipeline-level checkout is inherited by command steps that do not set their own. The step value wins per leaf when both are set:

checkout:
  skip: true

steps:
  - command: echo "inherits skip: true from the pipeline"

  - command: echo "explicit override - checkout runs"
    checkout:
      skip: false

After calling step.MergeCheckoutFromPipeline(pipeline.Checkout) on the second step, the merged result has skip: false (step wins). The first step inherits skip: true from the pipeline.

checkout: false as a shorthand is rejected at unmarshal time; checkout is a multi-field namespace, so opt-out is spelled checkout: { skip: true }.

Checkout.Depth is a *int for the same reason as Skip, this is distinguishable from any explicit value, so an inherited pipeline level depth can be cleanly overridden at the step level. The backend validates depth >= 1, this library does not.

checkout:
  depth: 10

steps:
  - command: echo "Shallow depth defaulting to 10 from build level checkout"

  - command: echo "Deeper shallow at the step level"
    checkout:
      depth: 50

What's up with the ordered module?

While implementing the pipeline module, we ran into a problem: in some cases, in the buildkite pipeline.yaml, the order of map fields is significant. Because of this, whenever the pipeline gets unmarshaled from YAML or JSON, it needs to be stored in a way that preserves the order of the fields. The ordered module is a simple implementation of an ordered map. In most cases, when the pipeline is dealing with user-input maps, it will store them internally as ordered.Maps. When the pipeline is marshaled back out to YAML or JSON, the ordered.Maps will be marshaled in the correct order.

Contributing

Contributions, bugfixes, issues and PRs are always welcome! Please see CONTRIBUTING.md for more details.