Add cache fallback to "latest" for cloud-build-docker module (#12)

## Summary:
No one likes slow PR checks. With the current cloud-build-docker implementation, caching is based on the branch name. This means the first docker build for a given branch will take longer because it won't have a cache to pull from.

This change enhances the cloud-build-docker module to automatically fall back to using "latest" as the cache tag when the specified cache tag doesn't exist in the container registry. This improves build performance by ensuring builds can still benefit from layer caching even when branch-specific caches are not available.

Key changes:
- Added `check_cache_tag_exists()` function to verify if a cache tag exists
- Added `get_effective_cache_tag()` function that determines the effective cache tag with fallback logic
- Modified `build_image()` to use the effective cache tag instead of always using the provided tag
- Enhanced Cloud Build configuration with better logging and error handling
- Updated documentation to reflect the new cache fallback behavior

The implementation provides clear logging to indicate when fallback occurs, making it easy to understand the caching behavior during builds.

## Test plan:
This was tested [here](Khan/internal-services#355)

Author: jwbron

Reviewers: csilvers

Required Reviewers:

Approved By: csilvers

Checks: ✅ 1 check was successful

Pull Request URL: #12

Author: jwbron

terraform/modules/cloud-build-docker/README.md

@@ -6,6 +6,7 @@ A reusable Terraform module for building Docker images using Google Cloud Build
 
 - **Cloud Build Integration**: Uses Google Cloud Build for reliable, scalable Docker image building
 - **Branch-based Caching**: Optimizes build times by caching layers based on branch names
+- **Cache Fallback**: Automatically falls back to "latest" tag if the specified cache tag doesn't exist, ensuring we always have some level of caching
 - **Digest Tracking**: Returns full image digests for precise versioning in Terraform
 - **Flexible Dockerfile Support**: Supports custom Dockerfile names and locations
 - **Build Arguments**: Supports custom build arguments and base image digests
@@ -132,13 +133,15 @@ module "secure_app" {
 
 ### Build Process
 1. **Context Preparation**: The module prepares the build context and handles Dockerfile symlinks if needed
-2. **Cloud Build Submission**: Submits the build to Google Cloud Build with appropriate substitutions
-3. **Caching**: Uses branch-based caching to optimize build times
-4. **Digest Retrieval**: Queries the built image to get its full digest
-5. **Output**: Returns the digest for use in other Terraform resources
+2. **Cache Tag Resolution**: Determines the effective cache tag by checking if the specified tag exists, falling back to "latest" if not
+3. **Cloud Build Submission**: Submits the build to Google Cloud Build with appropriate substitutions
+4. **Caching**: Uses the effective cache tag for layer caching to optimize build times
+5. **Digest Retrieval**: Queries the built image to get its full digest
+6. **Output**: Returns the digest for use in other Terraform resources
 
 ### Caching Strategy
 - **Branch-based**: Uses the `image_tag_suffix` as a cache tag
+- **Fallback to Latest**: If the specified cache tag doesn't exist, falls back to using "latest" as the cache tag, ensuring we always have some level of caching
 - **Layer Reuse**: Subsequent builds reuse cached layers when possible
 - **Cache Invalidation**: Cache is automatically invalidated when Dockerfile or context changes
 
@@ -265,3 +268,4 @@ resource "google_cloud_run_v2_service" "app" {
 - Clear cache by using a unique `image_tag_suffix`
 - Check if base images are accessible
 - Verify network connectivity during builds
+- **Cache Fallback**: If you see "falling back to 'latest'" in logs, the specified cache tag doesn't exist and the module is using "latest" instead

terraform/modules/cloud-build-docker/build_image.py

@@ -54,6 +54,48 @@ def get_image_digest(image_uri, tag, project_id):
         return None
 
 
+def check_cache_tag_exists(image_uri, cache_tag, project_id):
+    """Check if a cache tag exists for the given image."""
+    try:
+        result = run_command(
+            [
+                "gcloud",
+                "container",
+                "images",
+                "list-tags",
+                image_uri,
+                "--filter",
+                f"tags:{cache_tag}",
+                "--limit",
+                "1",
+                "--format=get(digest)",
+                "--project",
+                project_id,
+            ]
+        )
+        return bool(result.stdout.strip())
+    except subprocess.CalledProcessError:
+        return False
+
+
+def get_effective_cache_tag(image_uri, image_tag_suffix, project_id):
+    """Determine the effective cache tag.
+    
+    Ideally, the cache tag we use is the requested tag, `image_tag_suffix`.
+    But if that doesn't exist, we fall back to 'latest'. In that case, we may 
+    still be able to reuse some or all layers, depending on where changes were 
+    made in the branch running this build.
+    """
+    # First check if the provided cache tag exists
+    if check_cache_tag_exists(image_uri, image_tag_suffix, project_id):
+        print(f"Using cache tag: {image_tag_suffix}", file=sys.stderr)
+        return image_tag_suffix
+    
+    # Fall back to 'latest' if the provided tag doesn't exist
+    print(f"Cache tag '{image_tag_suffix}' not found, falling back to 'latest'", file=sys.stderr)
+    return "latest"
+
+
 def build_image(
     image_name,
     context_path,
@@ -70,6 +112,9 @@ def build_image(
 
     print(f"Building image: {image_name} with tag: {image_tag_suffix}", file=sys.stderr)
 
+    # Determine the effective cache tag
+    effective_cache_tag = get_effective_cache_tag(image_uri, image_tag_suffix, project_id)
+
     # Handle Dockerfile symlink if needed
     # The symlinking logic allows using Dockerfiles with:
     # - Custom names (e.g., listener.Dockerfile, base.Dockerfile)
@@ -117,7 +162,7 @@ def build_image(
             "_IMAGE_NAME": image_uri,
             "_IMAGE_TAG": image_tag,
             "_BASE_DIGEST": base_digest,
-            "_CACHE_TAG": image_tag_suffix,  # Use branch name for caching
+            "_CACHE_TAG": effective_cache_tag,  # Use effective cache tag (with fallback to latest)
         }
         subs_str = ",".join(f"{k}={v}" for k, v in substitutions.items())
 

terraform/modules/cloud-build-docker/cloudbuild.yml

@@ -1,10 +1,14 @@
 # Cloud Build configuration with branch-based caching.
-# Uses branch name as cache tag for optimal layer reuse.
+# Enables caching by pulling the previous image with a given "cache tag", if it exists, and reusing its layers.
 steps:
   - name: 'gcr.io/cloud-builders/docker'
     id: Pull previous image
     entrypoint: bash
-    args: ['-c', 'docker pull "$_IMAGE_NAME:$_CACHE_TAG" || true']
+    args: 
+      - -c
+      - |
+        echo "Attempting to pull cache image: $_IMAGE_NAME:$_CACHE_TAG"
+        docker pull "$_IMAGE_NAME:$_CACHE_TAG" || echo "Cache image not found, will build without cache"
 
   - name: 'gcr.io/cloud-builders/docker'
     id: Build image
@@ -16,7 +20,7 @@ steps:
           echo "Using cache from $_IMAGE_NAME:$_CACHE_TAG"
           docker build --tag="$_IMAGE_TAG" --cache-from="$_IMAGE_NAME:$_CACHE_TAG" --build-arg BASE_IMAGE="$_BASE_DIGEST" .
         else
-          echo "No cache found, building without --cache-from"
+          echo "No cache found for $_IMAGE_NAME:$_CACHE_TAG, building without --cache-from"
           docker build --tag="$_IMAGE_TAG" --build-arg BASE_IMAGE="$_BASE_DIGEST" .
         fi