Building an Azure DevOps CI/CD Pipeline

This guide describes how to set up and deploy an Azure DevOps CI/CD pipeline used for continuous integration, automated testing, and package release of SIMATIC AX projects.

Overview

You will create a workflow that provides a complete CI/CD pipeline for SIMATIC AX libraries, including the following steps:

1. Trigger the pipeline from a Git commit to main or a new tag push
2. Initialize a hosted environment and install all required software
3. Build the library
4. Run unit tests with Apax and upload results to Azure DevOps
5. Package and sign the library
6. Store the packaged library in Azure DevOps
7. On a tagged version, publish build artifacts to a private package registry

Prerequisites

• Your project is kept in a GitHub repository. Azure DevOps and BitBucket hosting is also supported but will not be detailed in this guide.
• You have write access to the repository, and you can generate a token with workflows permissions
• You have an Azure DevOps Artifacts registry (feed) set up
• Your Azure DevOps project has access to your GitHub project
• Your project follows the standard SIMATIC AX structure with an apax.yml configuration, and all required dependencies are properly declared in the apax.yml file.
• Your test files are located in the test/ directory.

Environment variables

The full list of environment variables used in this pipeline is:

Variable name	Purpose	Required permissions/scope	Source	Secret
ARTIFACTS_TOKEN	Azure DevOps token for publishing to private registry	Packaging (Read & write)	Personal access token of an Azure DevOps user	Yes
REGISTRY_URL	URL for Azure DevOps package registry	-	Copy from your Azure Artifacts feed	No
SCOPE	Scope of the library package	-	Determined by registry setup	No
GHCR_READ_PACKAGES	Authentication token to log in to the GitHub container registry	read_packages	Personal access token of a GitHub user	Yes
APAX_TOKEN	SIMATIC AX user token for the Apax registry	Read Apax packages	Generate a token via the SIMATIC AX homepage	Yes
SIGN_KEY	Private key for signing Apax packages to enable integrity verification during consumption	-	Generate a key pair using the Apax apax keygen command. Use the private key.	Yes

Note that to populate these values, you must first complete the initial setup described below. For example, you cannot populate any variables until the pipeline has been created, and you cannot populate the registry URL if the registry has not been created.

Set up artifact storage registry

To set up an Azure-hosted artifact storage registry, navigate to the Artifacts page of Azure DevOps and click the Create Feed button.

Next, select the settings you want for your feed. The scope determines whether the packages you publish with Apax are associated with a specific project or your organization as a whole. If you choose organization scope, then the package scope will be the name of the organization. Similarly, if you choose project scope, then the scope of the package will be the name of the project.

To ensure compatibility with npm naming conventions, the chosen scope must be a valid npm scope name. This means it must contain no uppercase letters, no spaces, and no other non-URL-safe characters. In this example, we are creating a project-scoped feed named commands in the project axdevops. If the project were named AxDevops or ax devops, we would be unable to publish a package to this scope because the scope name would not be a valid npm name due to capital letters or spaces, respectively.

To publish your library package to the registry, ensure that the project name in the apax.yml file matches the scope of your registry. For example, if you have a project-scoped registry in the project axdevops and your library is named commands, the name field in the apax.yml must be set to @axdevops/commands.

Setting up the pipeline

The first step in setting up a new pipeline is to click the New pipeline button on the Azure DevOps Pipelines page.

Next, select where your code is hosted. For this guide we will be using GitHub, but Azure Repos Git and Bitbucket Cloud are also supported.

After this, you will be asked to authorize Azure to access your GitHub account. If your repo is part of an organization, the organization will have to approve the integration.

Select which repository you want to link to the pipeline.

Next, select Starter pipeline

This will auto-populate the starter pipeline template.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

By default, this starter pipeline populates the trigger, which specifies the events that cause the pipeline to run. In this case, the default value is main. This means that every time a commit is pushed to the main branch (including merge commits from pull requests), the pipeline will run. We want to also trigger the pipeline when a new tag is pushed, so we will update the trigger field.

trigger:
  branches:
    include:
      - main
  tags:
    include:
      - '*'

The starter pipeline also defines pool, which specifies which types of machines the pipeline will be run on. Since we are using a hosted runner (a computer owned by Microsoft) we can specify what type of environment to use. This pipeline is designed to work with the default value, ubuntu-latest.

Installing Apax

In this pipeline, we install Apax directly on the host machine. To do this, we run the script install-apax-local.sh.

steps:
- script: bash ci-scripts/install-apax-local.sh
  displayName: 'Install Apax'
  env:
    APAX_TOKEN : $(APAX_TOKEN) # recommended way to map a secret pipeline variable to an environment variable

This script is based on a Siemens-prepared Dockerfile, and lives in our repo under the ci-scripts folder. The script installs prerequisite packages such as Git and Node.js. It then logs in to the Apax registry and downloads and installs Apax. To access our secret variable APAX_TOKEN, we must map it in the env section of the step definition.

Building the library and running tests

Once Apax is installed, we must build the project and run the tests.

- script: |
    # Log in to the SIMATIC AX registry
    apax login --password $APAX_TOKEN

    # Log in to the SIMATIC AX GitHub Community
    apax login --registry https://npm.pkg.github.com/ --password $GHCR_READ_PACKAGES

    # Install dependencies
    apax install --immutable

    # Build the project
    apax build

    # Run tests
    apax test

  displayName: 'Build library and run tests'
  env:
    APAX_TOKEN : $(APAX_TOKEN)
    GHCR_READ_PACKAGES : $(GHCR_READ_PACKAGES)

This script consists exclusively of Apax commands. We call apax login twice: once to log in to the SIMATIC AX registry and once to log in to the SIMATIC AX GitHub Community registry. This enables us to install packages from these registries as dependencies of our library. If your library also has dependencies from a private registry, you need to log in to that registry here as well.

After logging in, we call apax install --immutable to install the dependencies. The --immutable flag specifies installing exactly the versions specified in the apax-lock.json, which enables more reproducible builds.

Next, we call apax build to build the project. Finally, we call apax test to run all of the tests associated with the library.

Storing test results

Once we have run the tests, we want to upload the test results XML file to the pipeline. This can be done with the PublishBuildArtifacts@1 task. A task is a pre-configured step for an Azure DevOps pipeline, which can be added via the Edit Pipeline GUI.

In this case we will configure the publish to use:

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: 'bin/axunit-artifacts/test-results/TestResult.xml'
    ArtifactName: 'Test Results'
    publishLocation: 'Container'

Here, PathtoPublish is how we specify which file or folder we want to upload. Since Apax stores the file in the same location every time apax test is run, we can hard-code this path for the pipeline. The ArtifactName is what we call the uploaded file. Finally, publishLocation specifies where in the pipeline the file is uploaded. After the pipeline runs, we can see the artifacts that are uploaded in this step by navigating to the run summary and clicking on the 2 published button (we will also upload the packaged library in a later step).

Version the library (optional)

Before we package the library, we want to check whether there is a tag associated with this commit and, if so, update the project's version.

- script: |
    # Version the package if the commit has a tag
    TAG_VERSION=$(git describe --exact-match --tags HEAD) # Read latest tag
    if [ $? -eq 0 ]; then # If we have a new tag (--exact-match succeeded)
        echo "Detected git tag $TAG_VERSION"
        apax version $TAG_VERSION #Update version if we have a new git tag
    fi
  displayName: 'Update version if new tag'

Here, we read whether there are any tags associated with the current commit and store the tag in the variable TAG_VERSION. We then call apax version to update the version of the library.

If no tag is detected, the library package will be assigned the version that is defined in the library's apax.yml. For this reason, this step is optional if your release workflow involves updating the apax.yml version when creating a new tagged release.

Package the library

Once the previous steps have been completed, we can package the library by calling apax pack and providing a key to sign the package.

- script: apax pack --key $SIGN_KEY
  displayName: 'Package library to tarball'
  env:
    SIGN_KEY : $(SIGN_KEY)

As described in the prerequisites, this key should be the private portion of a key created with the command apax keygen.

Store the packaged library in the pipeline run

After we have packaged the library, we want to store it in the pipeline as a build artifact. Unlike the test results XML, which always has the same filename, the library package tarball includes the version in its name. Because the build artifacts upload doesn't support wildcards, we must first run a short script to find and store the filename in a pipeline variable.

- script: | # Find tarball filename (changes with version increments)
    FILE=$(ls *.apax.tgz)
    echo "##vso[task.setvariable variable=packageFile]$FILE"
  displayName: "Find tarball filename"

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(packageFile)'
    ArtifactName: 'Library tarball'
    publishLocation: 'Container'

Publish library to Azure DevOps private registry

Once our package has been built, we may also want to publish it to a private registry. This example script attempts to publish the package to a private registry every time a Git tag is detected.

- script: |
    TAG_VERSION=$(git describe --exact-match --tags HEAD) # Read latest tag
    if [ $? -eq 0 ]; then
      # Log in to Azure Artifacts registry
      apax config set token $ARTIFACTS_TOKEN --registry $REGISTRY_URL
      apax config set scope "@$SCOPE" --registry $REGISTRY_URL
      # Publish package
      apax publish --package *.apax.tgz --registry $REGISTRY_URL
    else
      echo "No new tag detected, not publishing to registry"
    fi
  displayName: 'Publish to registry if tagged version'
  env:
    ARTIFACTS_TOKEN: $(ARTIFACTS_TOKEN)

Inside the if statement, we first sign in to the private registry using apax config set token. We specify the token and the registry associated with the token. The next line uses apax config set scope to associate the scope of our package with the correct registry. This tells Apax that all packages with the scope defined in the SCOPE variable should be pulled from and published to our provided registry URL. Finally, we call apax publish and specify our tarball as the package and the registry URL we wish to push to.

The publish step may fail for a number of reasons. Some common error codes and causes are:

• A 404 error indicates the specified registry does not exist; there is likely a problem with your REGISTRY_URL.
• A 401 error indicates your credentials are not valid. Check that the token you have configured has at least packaging read and write permissions for the registry you are uploading to.
• A 400 error indicates a generic problem with the upload. Some common reasons include:
  • Package too large. If your tarball package is over 512 MB, Azure DevOps will reject the upload. This can also be the problem if you are close to the limit (on the order of hundreds of MB). If your package is too large, double-check the files that are being included in the files section of the project's apax.yml. If the entire bin folder is included, the subdirectory bin/axunit-artifacts can be very large and may not be necessary to include. Consider specifying only certain subdirectories, such as bin/1500 and bin/llvm.
  • Package scope mismatch. The scope of the package must match the scope of the registry. The example registry specified in the prerequisites section is project scoped, meaning that if we want to publish to this registry, the scope of this package must match. The package name is specified in the name field of the project's apax.yml file, with the syntax @<scope>/<package>. As an example, this project's name is @axdevops/commands. We are publishing it to a registry scoped as axdevops, and commands is the name of the resulting package. If the scope does not match, then Azure DevOps will reject the upload with a 400 status code.
  • Bad registry URL. Ensure that the URL matches the URL displayed on the Connect to feed page of Azure DevOps. It must start with https:// and end with /npm/registry/. In this example pipeline, the value of the variable $REGISTRY_URL is https://pkgs.dev.azure.com/loupeteam/axdevops/_packaging/commands/npm/registry/.

Example pipeline

This is the complete example pipeline described in the above section:

trigger:
  branches:
    include:
      - main
  tags:
    include:
      - '*'

pool:
  vmImage: ubuntu-latest

steps:
- script: bash ci-scripts/install-apax-local.sh
  displayName: 'Install Apax'
  env:
    APAX_TOKEN : $(APAX_TOKEN) # recommended way to map a secret pipeline variable to an environment variable
- script: |
    # Log in to the SIMATIC AX registry
    apax login --password $APAX_TOKEN

    # Log in to the SIMATIC AX GitHub Community
    apax login --registry https://npm.pkg.github.com/ --password $GHCR_READ_PACKAGES

    # Install dependencies
    apax install --immutable

    # Build the project
    apax build

    # Run tests
    apax test

  displayName: 'Build library and run tests'
  env:
    APAX_TOKEN : $(APAX_TOKEN)
    GHCR_READ_PACKAGES : $(GHCR_READ_PACKAGES)

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: 'bin/axunit-artifacts/test-results/TestResult.xml'
    ArtifactName: 'Test Results'
    publishLocation: 'Container'

- script: |
    # Version the package if the commit has a tag
    TAG_VERSION=$(git describe --exact-match --tags HEAD) # Read latest tag
    if [ $? -eq 0 ]; then # If we have a new tag (--exact-match succeeded)
        echo "Detected git tag $TAG_VERSION"
        apax version $TAG_VERSION #Update version if we have a new git tag
    fi
  displayName: 'Update version if new tag'

- script: apax pack --key $SIGN_KEY
  displayName: 'Package library to tarball'
  env:
    SIGN_KEY : $(SIGN_KEY)

- script: | # Find tarball filename (changes with version increments)
    FILE=$(ls *.apax.tgz)
    echo "##vso[task.setvariable variable=packageFile]$FILE"
  displayName: "Find tarball filename"

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(packageFile)'
    ArtifactName: 'Library tarball'
    publishLocation: 'Container'

- script: |
    TAG_VERSION=$(git describe --exact-match --tags HEAD) # Read latest tag
    if [ $? -eq 0 ]; then
        # Log in to Azure Artifacts registry
        apax config set token $ARTIFACTS_TOKEN --registry $REGISTRY_URL
        apax config set scope "@$SCOPE" --registry $REGISTRY_URL
        # Publish package
        apax publish --package *.apax.tgz --registry $REGISTRY_URL
    else
        echo "No new tag detected, not publishing to registry"
    fi
  displayName: 'Publish to registry if tagged version'
  env:
    ARTIFACTS_TOKEN: $(ARTIFACTS_TOKEN)

Related links

• Azure Pipelines documentation
• Azure Pipelines variables
• Azure Artifacts documentation
• Use npm scopes in Azure Artifacts
  • Note: Credentials are handled by Apax via the apax config commands. There is no need to manually create a .npmrc file.
• Troubleshoot parallel jobs in Azure DevOps free tier