Skip to content

fix: update apt package lists before installing cbt CLI - #122

Merged
mutianf merged 1 commit into
GoogleCloudDataproc:mainfrom
mutianf:fix-cbt-apt-install
Aug 31, 2026
Merged

fix: update apt package lists before installing cbt CLI#122
mutianf merged 1 commit into
GoogleCloudDataproc:mainfrom
mutianf:fix-cbt-apt-install

Conversation

@mutianf

@mutianf mutianf commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Description

Update cbt installation in .kokoro/build.sh to update apt package lists before installing, and support installing via gcloud components or modern google-cloud-cli-cbt package.

Root Cause

Previously, apt install -y google-cloud-sdk-cbt was called directly without running apt-get update. In older base images (such as gcr.io/cloud-devrel-kokoro-resources/java11), the cached package index referenced older package versions (e.g. google-cloud-cli-cbt 514.0.0-0) that had since been replaced and removed from packages.cloud.google.com/apt, causing 404 Not Found errors:

Err:1 http://packages.cloud.google.com/apt cloud-sdk/main amd64 google-cloud-cli-cbt amd64 514.0.0-0 404 Not Found
E: Failed to fetch http://packages.cloud.google.com/apt/pool/cloud-sdk/google-cloud-cli-cbt_514.0.0-0_amd64_03a51de04d51cdd23f7f8cb7ca02644d.deb 404 Not Found

Because set +e was active, the build continued without cbt installed, which caused subsequent tests requiring cbt (such as fuzz) to fail with cbt: command not found when attempting to create/delete test tables.

Solution

  1. Check if cbt is already in PATH or in the same directory as gcloud.
  2. If not found, attempt gcloud components install cbt --quiet.
  3. If that fails, run apt-get update -y || true followed by apt-get install -y google-cloud-cli-cbt || apt-get install -y google-cloud-sdk-cbt (supporting sudo if present).
  4. Verify and log cbt availability.

Update cbt installation in .kokoro/build.sh to update apt package lists
before installing, and support installing via gcloud components or modern
google-cloud-cli-cbt package.

Previously, apt install -y google-cloud-sdk-cbt was called without running
apt-get update, causing package downloads to 404 when the local package
index in older base images pointed to older package versions that were no
longer available in the remote pool. This resulted in cbt not being installed
and subsequent tests failing with table not found errors.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates .kokoro/build.sh to robustly locate or install the cbt CLI tool, falling back to gcloud components or apt-get with or without sudo if necessary. The reviewer suggested refactoring the script to eliminate code duplication by extracting the PATH resolution logic into a helper function and using a bash array to conditionally prepend sudo for apt-get commands.

Comment thread .kokoro/build.sh
Comment on lines +49 to +82
# Ensure cbt is in PATH if it's installed alongside gcloud but not in PATH
if ! command -v cbt &> /dev/null && command -v gcloud &> /dev/null; then
GCLOUD_PATH=$(readlink -f "$(command -v gcloud)")
GCLOUD_BIN_DIR=$(dirname "${GCLOUD_PATH}")
if [[ -x "${GCLOUD_BIN_DIR}/cbt" ]]; then
export PATH="${PATH}:${GCLOUD_BIN_DIR}"
fi
fi

# Install cbt if it is not present
if ! command -v cbt &> /dev/null; then
echo "Installing the cbt CLI command."
if command -v gcloud &> /dev/null && gcloud components install cbt --quiet 2>/dev/null; then
echo "cbt installed via gcloud components."
else
echo "Installing cbt via apt..."
if command -v sudo &> /dev/null; then
sudo apt-get update -y || true
sudo apt-get install -y google-cloud-cli-cbt || sudo apt-get install -y google-cloud-sdk-cbt
else
apt-get update -y || true
apt-get install -y google-cloud-cli-cbt || apt-get install -y google-cloud-sdk-cbt
fi
fi

# Ensure the newly installed cbt is in PATH
if ! command -v cbt &> /dev/null && command -v gcloud &> /dev/null; then
GCLOUD_PATH=$(readlink -f "$(command -v gcloud)")
GCLOUD_BIN_DIR=$(dirname "${GCLOUD_PATH}")
if [[ -x "${GCLOUD_BIN_DIR}/cbt" ]]; then
export PATH="${PATH}:${GCLOUD_BIN_DIR}"
fi
fi
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is significant code duplication in this block:

  1. The logic to find cbt next to gcloud and add it to PATH is duplicated exactly twice (lines 50-56 and lines 75-81).
  2. The apt-get update and install commands are duplicated with and without sudo (lines 65-71).

We can refactor this by:

  • Extracting the PATH resolution logic into a helper function add_cbt_to_path.
  • Using a bash array sudo_cmd to conditionally prepend sudo if available, avoiding the duplicated if/else block for apt-get.
  • Removing the redundant -y flag from apt-get update (since update does not prompt for confirmation).
# Ensure cbt is in PATH if it's installed alongside gcloud but not in PATH
add_cbt_to_path() {
  if ! command -v cbt &> /dev/null && command -v gcloud &> /dev/null; then
    local gcloud_path
    gcloud_path=$(readlink -f "$(command -v gcloud)")
    local gcloud_bin_dir
    gcloud_bin_dir=$(dirname "${gcloud_path}")
    if [[ -x "${gcloud_bin_dir}/cbt" ]]; then
      export PATH="${PATH}:${gcloud_bin_dir}"
    fi
  fi
}

add_cbt_to_path

# Install cbt if it is not present
if ! command -v cbt &> /dev/null; then
  echo "Installing the cbt CLI command."
  if command -v gcloud &> /dev/null && gcloud components install cbt --quiet 2>/dev/null; then
    echo "cbt installed via gcloud components."
  else
    echo "Installing cbt via apt..."
    sudo_cmd=()
    if command -v sudo &> /dev/null; then
      sudo_cmd=(sudo)
    fi
    "${sudo_cmd[@]}" apt-get update || true
    "${sudo_cmd[@]}" apt-get install -y google-cloud-cli-cbt || "${sudo_cmd[@]}" apt-get install -y google-cloud-sdk-cbt
  fi

  # Ensure the newly installed cbt is in PATH
  add_cbt_to_path
fi

@mutianf
mutianf merged commit 15d4150 into GoogleCloudDataproc:main Aug 31, 2026
9 checks passed
@mutianf
mutianf deleted the fix-cbt-apt-install branch August 31, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants