Fail fast for real on unusable XDG_CACHE_HOME on Windows

[rdesgroppes]

Motivation

Accidentally found that a couple of jobs were printing fatal error messages without actually failing fast (example) (i.e. continuing past the not-that-fatal error).

The problem is that cmd.exe happens to zero out the code of an exit /b <code> that does not end every enclosing parenthesized block, so tools/bazel.bat returned 0 and let subsequent steps to proceed.

What does this PR do?

Reach both fatal diagnostics through goto labels placed after the main flow, where exit /b 2 is now honored.

Route the two jobs through docker-run-with-bazel-cache.ps1, as the fixed guards would otherwise fail them, on purpose.

Describe how you validated your changes

On Windows Server 2022 with bazelisk 1.28.1, the wrapper returns 2 for an unset and for a relative XDG_CACHE_HOME, and still reaches bazel for a valid one.

dda inv gitlab.print-ci shows both jobs inheriting XDG_CACHE_HOME=c:/bzl, as tests_windows-x64 and others already do.

Additional Notes

The two remaining guards do end their blocks, so they already exit 2.

Yet another reason for rewriting both tools/bazel* hooks in Python.

[dd-octo-sts]

Gitlab CI Configuration Changes

Modified Jobs

.integration_tests_windows_base

  .integration_tests_windows_base:
    before_script:
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
      -parameterField url -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { exit "$lastExitCode" }
    - $vcpkgBlobSaSUrl=$(cat "$tmpfile")
    - Remove-Item "$tmpfile"
    id_tokens:
+     BUILDBARN_ID_TOKEN:
+       aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - . ./tools/ci/sanitize-goproxy.ps1
+   - '.\tools\ci\docker-run-with-bazel-cache.ps1 -m 16384M -v "$(Get-Location):c:\mnt"
-   - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
?   - ------------------------------------------------------------
+     -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
      -e CI_JOB_URL -e CI_JOB_NAME -e CI_PIPELINE_ID -e CI_PROJECT_NAME -e CI_IDENTITIES_GITLAB_ID_TOKEN
      -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e GOPROXY -e GONOSUMDB -e
      VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL ${WINBUILDIMAGE}
      powershell.exe -c "c:\mnt\tasks\winbuildscripts\Invoke-IntegrationTests.ps1 -BuildOutOfSource
      1 -CheckGoVersion 1 -InstallDeps 1"
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: integration_test
    tags:
    - windows-v2:2022
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}\.aws\credentials-by-job-id\${CI_JOB_ID}
+     BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      GIT_STRATEGY: clone
      WINBUILDIMAGE: registry.ddbuild.io/ci/datadog-agent-buildimages/windows_ltsc2022_${ARCH}${CI_IMAGE_WIN_LTSC2022_X64_SUFFIX}:${CI_IMAGE_WIN_LTSC2022_X64}
+     XDG_CACHE_HOME: c:/bzl

.lint_windows_base

  .lint_windows_base:
    before_script:
    - C:\ci-identities-gitlab-job-client.exe assume-role
    id_tokens:
+     BUILDBARN_ID_TOKEN:
+       aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - $ErrorActionPreference = "Stop"
    - . ./tools/ci/sanitize-goproxy.ps1
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$AGENT_API_KEY_ORG2"
      -parameterField token -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { exit "$lastExitCode" }
    - $Env:DATADOG_API_KEY=$(cat "$tmpfile")
    - C:\datadog-ci.exe trace -- docker pull ${WINBUILDIMAGE}
-   - 'docker run --rm -m 24576M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
+   - '.\tools\ci\docker-run-with-bazel-cache.ps1 -m 24576M -v "$(Get-Location):c:\mnt"
-     -e CI -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
?    ------
+     -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
      -e AWS_NETWORKING=true -e CI_PIPELINE_ID -e CI_PROJECT_NAME -e CI_IDENTITIES_GITLAB_ID_TOKEN
      -e GOMODCACHE="c:\modcache" -e GOPROXY -e GONOSUMDB -e RUST_LOG="uv=trace" ${WINBUILDIMAGE}
      powershell.exe -c "c:\mnt\tasks\winbuildscripts\Invoke-Linters.ps1 -BuildOutOfSource
      1 -CheckGoVersion 1 -InstallDeps 1"
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: lint
    tags:
    - windows-v2:2022
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}\.aws\credentials-by-job-id\${CI_JOB_ID}
+     BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      GIT_STRATEGY: clone
      WINBUILDIMAGE: registry.ddbuild.io/ci/datadog-agent-buildimages/windows_ltsc2022_${ARCH}${CI_IMAGE_WIN_LTSC2022_X64_SUFFIX}:${CI_IMAGE_WIN_LTSC2022_X64}
+     XDG_CACHE_HOME: c:/bzl

integration_tests_windows-x64

  integration_tests_windows-x64:
    before_script:
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
      -parameterField url -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { exit "$lastExitCode" }
    - $vcpkgBlobSaSUrl=$(cat "$tmpfile")
    - Remove-Item "$tmpfile"
    id_tokens:
+     BUILDBARN_ID_TOKEN:
+       aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - . ./tools/ci/sanitize-goproxy.ps1
+   - '.\tools\ci\docker-run-with-bazel-cache.ps1 -m 16384M -v "$(Get-Location):c:\mnt"
-   - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
?   - ------------------------------------------------------------
+     -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
      -e CI_JOB_URL -e CI_JOB_NAME -e CI_PIPELINE_ID -e CI_PROJECT_NAME -e CI_IDENTITIES_GITLAB_ID_TOKEN
      -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e GOPROXY -e GONOSUMDB -e
      VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL ${WINBUILDIMAGE}
      powershell.exe -c "c:\mnt\tasks\winbuildscripts\Invoke-IntegrationTests.ps1 -BuildOutOfSource
      1 -CheckGoVersion 1 -InstallDeps 1"
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: integration_test
    tags:
    - windows-v2:2022
    variables:
      ARCH: x64
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}\.aws\credentials-by-job-id\${CI_JOB_ID}
+     BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      GIT_STRATEGY: clone
      WINBUILDIMAGE: registry.ddbuild.io/ci/datadog-agent-buildimages/windows_ltsc2022_${ARCH}${CI_IMAGE_WIN_LTSC2022_X64_SUFFIX}:${CI_IMAGE_WIN_LTSC2022_X64}
+     XDG_CACHE_HOME: c:/bzl

lint_windows-x64

  lint_windows-x64:
    before_script:
    - C:\ci-identities-gitlab-job-client.exe assume-role
    id_tokens:
+     BUILDBARN_ID_TOKEN:
+       aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_TAG != null
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: ($DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null) && $BUCKET_BRANCH =~
        /^(nightly|oldnightly|beta|stable)$/
    script:
    - $ErrorActionPreference = "Stop"
    - . ./tools/ci/sanitize-goproxy.ps1
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$AGENT_API_KEY_ORG2"
      -parameterField token -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { exit "$lastExitCode" }
    - $Env:DATADOG_API_KEY=$(cat "$tmpfile")
    - C:\datadog-ci.exe trace -- docker pull ${WINBUILDIMAGE}
-   - 'docker run --rm -m 24576M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
+   - '.\tools\ci\docker-run-with-bazel-cache.ps1 -m 24576M -v "$(Get-Location):c:\mnt"
-     -e CI -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
?    ------
+     -e GITLAB_CI -e DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS="${DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS}"
      -e AWS_NETWORKING=true -e CI_PIPELINE_ID -e CI_PROJECT_NAME -e CI_IDENTITIES_GITLAB_ID_TOKEN
      -e GOMODCACHE="c:\modcache" -e GOPROXY -e GONOSUMDB -e RUST_LOG="uv=trace" ${WINBUILDIMAGE}
      powershell.exe -c "c:\mnt\tasks\winbuildscripts\Invoke-Linters.ps1 -BuildOutOfSource
      1 -CheckGoVersion 1 -InstallDeps 1"
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: lint
    tags:
    - windows-v2:2022
    timeout: 1h30m
    variables:
      ARCH: x64
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}\.aws\credentials-by-job-id\${CI_JOB_ID}
+     BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      GIT_STRATEGY: clone
      WINBUILDIMAGE: registry.ddbuild.io/ci/datadog-agent-buildimages/windows_ltsc2022_${ARCH}${CI_IMAGE_WIN_LTSC2022_X64_SUFFIX}:${CI_IMAGE_WIN_LTSC2022_X64}
+     XDG_CACHE_HOME: c:/bzl

Changes Summary

Removed	Modified	Added	Renamed
0	4	0	0

ℹ️ Diff available in the job log.

[JSGette]

Since you are modifying this file, could you maybe drop this hint from batch script? Sharing caches between the host and a Windows container simply doesn't work, Bazel doesn't like junctions, apparently

[rdesgroppes]

@JSGette OK, I'll do, but in a distinct PR to be reverted independently if need be.

[dd-octo-sts]

Files inventory check summary

File checks results against ancestor bcbf4865:

Results for datadog-agent_7.84.0~devel.git.227.44c0ce2.pipeline.131286102-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.84.0~devel.git.227.44c0ce2.pipeline.131286102-1_amd64.deb:

No change detected