Try to prevent macos jobs from leaving unwritable files behind by alopezz · Pull Request #54104 · DataDog/datadog-agent

alopezz · 2026-07-24T15:23:49Z

What does this PR do?

It adds a step in the after_script of select macos jobs which are known to potentially leave files with read-only permission so that a best-effort attempt is done at making them writable, to make cleanup possible.

Motivation

#incident-57903

Jobs where we install rtloader and its deps onto the CI_PROJECT_DIR rely on a pkg_install rule, which first copies the files out of bazel's output root to the destination and only after that does it change the permissions (source). If the job gets interrupted between those two steps, as it did for this incident, those files could remain with the existing read-only permissions. Given that macos runners are persistent, the checkout step needs to first cleanup any existing checkout first, and this step fails if there are non-writable files, such that we were seeing errors such as:

warning: failed to remove dev/embedded/lib/python3.13/lib-dynload/termios.cpython-313-darwin.so: Permission denied

Describe how you validated your changes

Additional Notes

dd-octo-sts · 2026-07-24T15:32:38Z

Gitlab CI Configuration Changes

Modified Jobs

.tests_macos_gitlab

  .tests_macos_gitlab:
    after_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
      $?; export CODECOV_TOKEN
    - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - datadog-ci coverage upload --format=go-coverprofile coverage.out || true
+   - "# GitLab Runner 18 fails the next job during clear_worktree when the\n# previous\
+     \ job leaves read-only files in the persistent macOS workspace.\n# Best-effort\
+     \ only: do not mask the job's real result if cleanup fails.\nif [ -n \"$CI_PROJECT_DIR\"\
+     \ ] && [ -d \"$CI_PROJECT_DIR\" ]; then\n  chmod -R u+w \"$CI_PROJECT_DIR\" 2>/dev/null\
+     \ || true\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - ${TEST_OUTPUT_FILE}.json
      - ${TEST_OUTPUT_FILE}_unified.json
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit: '**/junit-out-*.xml'
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - . tools/ci/sanitize-goproxy.sh
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      sudo rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - export GOCACHE="$TMPDIR/go-build"
    - dda inv -- -e install-tools --verbose
    id_tokens:
      BUILDBARN_ID_TOKEN:
        aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - setup_agent_version
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    - allow_failure: true
      variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
      when: manual
    script:
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - dda inv -- -e agent.build
    - dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json ${TEST_OUTPUT_FILE}.json
      --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
    stage: source_test
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials-by-job-id/${CI_JOB_ID}
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
      TEST_OUTPUT_FILE: test_output
      XDG_CACHE_HOME: $RUNNER_TEMP_PROJECT_DIR

generate_config_schema-macos

  generate_config_schema-macos:
+   after_script:
+   - "# GitLab Runner 18 fails the next job during clear_worktree when the\n# previous\
+     \ job leaves read-only files in the persistent macOS workspace.\n# Best-effort\
+     \ only: do not mask the job's real result if cleanup fails.\nif [ -n \"$CI_PROJECT_DIR\"\
+     \ ] && [ -d \"$CI_PROJECT_DIR\" ]; then\n  chmod -R u+w \"$CI_PROJECT_DIR\" 2>/dev/null\
+     \ || true\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - $CI_PROJECT_DIR/pkg/config/schema/yaml/*.yaml
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - . tools/ci/sanitize-goproxy.sh
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      sudo rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - export GOCACHE="$TMPDIR/go-build"
    - dda inv -- -e install-tools --verbose
    id_tokens:
      BUILDBARN_ID_TOKEN:
        aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - setup_agent_version
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/config/**/*
        - tasks/schema/**/*
    script:
    - dda inv -- -e agent.build
    - dda inv -- schema.generate --agent-bin=./bin/agent/agent
    - bash $CI_PROJECT_DIR/tasks/schema/check_config_templates.sh $CI_PROJECT_DIR
    stage: binary_build
    tags:
    - macos:sonoma-amd64
    - specific:true
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials-by-job-id/${CI_JOB_ID}
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
      TEST_OUTPUT_FILE: test_output
      XDG_CACHE_HOME: $RUNNER_TEMP_PROJECT_DIR

tests_macos_gitlab_amd64

  tests_macos_gitlab_amd64:
    after_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
      $?; export CODECOV_TOKEN
    - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - datadog-ci coverage upload --format=go-coverprofile coverage.out || true
+   - "# GitLab Runner 18 fails the next job during clear_worktree when the\n# previous\
+     \ job leaves read-only files in the persistent macOS workspace.\n# Best-effort\
+     \ only: do not mask the job's real result if cleanup fails.\nif [ -n \"$CI_PROJECT_DIR\"\
+     \ ] && [ -d \"$CI_PROJECT_DIR\" ]; then\n  chmod -R u+w \"$CI_PROJECT_DIR\" 2>/dev/null\
+     \ || true\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - ${TEST_OUTPUT_FILE}.json
      - ${TEST_OUTPUT_FILE}_unified.json
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - . tools/ci/sanitize-goproxy.sh
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      sudo rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - export GOCACHE="$TMPDIR/go-build"
    - dda inv -- -e install-tools --verbose
    id_tokens:
      BUILDBARN_ID_TOKEN:
        aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - setup_agent_version
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    script:
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - dda inv -- -e agent.build
    - dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json ${TEST_OUTPUT_FILE}.json
      --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
    stage: source_test
    tags:
    - macos:sonoma-amd64
    - specific:true
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials-by-job-id/${CI_JOB_ID}
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
      TEST_OUTPUT_FILE: test_output
      XDG_CACHE_HOME: $RUNNER_TEMP_PROJECT_DIR

tests_macos_gitlab_arm64

  tests_macos_gitlab_arm64:
    after_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
      $?; export CODECOV_TOKEN
    - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - datadog-ci coverage upload --format=go-coverprofile coverage.out || true
+   - "# GitLab Runner 18 fails the next job during clear_worktree when the\n# previous\
+     \ job leaves read-only files in the persistent macOS workspace.\n# Best-effort\
+     \ only: do not mask the job's real result if cleanup fails.\nif [ -n \"$CI_PROJECT_DIR\"\
+     \ ] && [ -d \"$CI_PROJECT_DIR\" ]; then\n  chmod -R u+w \"$CI_PROJECT_DIR\" 2>/dev/null\
+     \ || true\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - ${TEST_OUTPUT_FILE}.json
      - ${TEST_OUTPUT_FILE}_unified.json
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - . tools/ci/sanitize-goproxy.sh
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  robust_curl=\"curl -fsSL --retry 4\"  # recommended flags + resist\
      \ transient errors like `Connection reset by peer`\n  # Get the commit from the\
      \ build image variable in the format `vPIPELINE_ID-COMMIT`\n  export BUILDIMAGES_COMMIT=\"\
      ${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"$($robust_curl https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  $robust_curl -o dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      sudo rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - export GOCACHE="$TMPDIR/go-build"
    - dda inv -- -e install-tools --verbose
    id_tokens:
      BUILDBARN_ID_TOKEN:
        aud: buildbarn.us1.ddbuild.io
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs:
    - setup_agent_version
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    - allow_failure: true
      variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
      when: manual
    script:
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - dda inv -- -e agent.build
    - dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json ${TEST_OUTPUT_FILE}.json
      --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
    stage: source_test
    tags:
    - macos:sonoma-arm64
    - specific:true
    variables:
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials-by-job-id/${CI_JOB_ID}
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
      TEST_OUTPUT_FILE: test_output
      XDG_CACHE_HOME: $RUNNER_TEMP_PROJECT_DIR

Added Jobs

.macos_runner_make_workspace_writable

.macos_runner_make_workspace_writable:
- "# GitLab Runner 18 fails the next job during clear_worktree when the\n# previous\
  \ job leaves read-only files in the persistent macOS workspace.\n# Best-effort only:\
  \ do not mask the job's real result if cleanup fails.\nif [ -n \"$CI_PROJECT_DIR\"\
  \ ] && [ -d \"$CI_PROJECT_DIR\" ]; then\n  chmod -R u+w \"$CI_PROJECT_DIR\" 2>/dev/null\
  \ || true\nfi\n"

Changes Summary

Removed	Modified	Added	Renamed
0	4	1	0

ℹ️ Diff available in the job log.

datadog-prod-us1-3 · 2026-07-24T15:49:00Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 52.06% (+0.13%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5365988 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53659888f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-24T15:51:16Z

    - !reference [.upload_junit_source]
    - !reference [.upload_coverage]
    - !reference [.upload_coverage_datadog]
+    - !reference [.macos_runner_make_workspace_writable]


Run workspace chmod before fallible upload steps

In the macOS test job, this cleanup is placed after .vault_login, .install_python_dependencies, and the upload blocks; those blocks contain hard failures such as fetch_secret.sh ... || exit $?, and GitLab runs the section with set -e, so a Vault/secret/upload setup failure prevents the chmod from running. That leaves exactly the interrupted/read-only workspace behind for the next persistent runner job. Please run this best-effort chmod before the fallible uploads, or wrap the preceding refs, so cleanup is attempted regardless of reporting failures.

Useful? React with 👍 / 👎.

dd-octo-sts · 2026-07-24T15:57:21Z

Files inventory check summary

File checks results against ancestor e2ad846e:

Results for datadog-agent_7.83.0~devel.git.371.5365988.pipeline.126824338-1_amd64.deb:

No change detected

Try to prevent macos jobs from leaving unwritable files behind

5365988

dd-octo-sts Bot added internal Identify a non-fork PR team/agent-devx team/fleet-automation team/windows-products labels Jul 24, 2026

github-actions Bot added the short review PR is simple enough to be reviewed quickly label Jul 24, 2026

alopezz added changelog/no-changelog No changelog entry needed qa/no-code-change No code change in Agent code requiring validation labels Jul 24, 2026

alopezz marked this pull request as ready for review July 24, 2026 15:46

alopezz requested review from a team as code owners July 24, 2026 15:46

alopezz requested review from Hitsuji-M and lrbhatia July 24, 2026 15:46

github-actions Bot deleted a comment from chatgpt-codex-connector Bot Jul 24, 2026

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Try to prevent macos jobs from leaving unwritable files behind#54104

Try to prevent macos jobs from leaving unwritable files behind#54104
alopezz wants to merge 1 commit into
mainfrom
alopez/macos-runners-permissions

alopezz commented Jul 24, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Jul 24, 2026

Uh oh!

datadog-prod-us1-3 Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Uh oh!

dd-octo-sts Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alopezz commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

dd-octo-sts Bot commented Jul 24, 2026

Gitlab CI Configuration Changes

Modified Jobs

Added Jobs

Changes Summary

Uh oh!

datadog-prod-us1-3 Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Jul 24, 2026

Files inventory check summary

Results for datadog-agent_7.83.0~devel.git.371.5365988.pipeline.126824338-1_amd64.deb:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

alopezz commented Jul 24, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Jul 24, 2026 •

edited

Loading