Wrong reusing cached layer if context.WorkingDir changed between builds

[aitchjoe]

Expected Behavior

We use pack CLI and GitLab CI to build same image, but the default working dir of pack CLI is /workspace and in GitLab CI is /builds/group/project. If we use two methods together, the image should not run error.

Current Behavior

We create a Vue demo with env BP_WEB_SERVER=nginx, but when we run this image we got error:

# docker run -d --name=demo --pull=always -q gitlab-registry.example.com/k8s/buildpacks-vue-demo
# docker logs demo
2023/10/31 03:58:13 [emerg] 1#0: directive "client_body_temp_path" is not terminated by ";" in /workspace/nginx.conf:16

From run.go, we think that tempDir replacement is missed before nginx start. But why this happened? We check the image content:

docker run --rm --pull=always -q --entrypoint="/bin/bash" \
  gitlab-registry.example.com/k8s/buildpacks-vue-demo \
  -c "tail -n 6 /layers/config/metadata.toml && echo -n APP_ROOT: && cat /layers/paketo-buildpacks_nginx/nginx/env.launch/APP_ROOT.default"
[[processes]]
  type = "web"
  command = ["nginx"]
  args = ["-p", "/workspace", "-c", "/workspace/nginx.conf", "-g", "pid /tmp/nginx.pid;"]
  direct = true
  buildpack-id = "paketo-buildpacks/nginx"
APP_ROOT:/builds/k8s/buildpacks-vue-demo

the nginx args is /workspace but app root is /builds/k8s/buildpacks-vue-demo, they shoule be the same.

Steps to Reproduce

Paketo Buildpack for Nginx Server 0.15.6 and more versions see the followed scripts.

.gitlab-ci.yml

build-image:
  image: docker.io/paketobuildpacks/builder-jammy-base:0.4.244
  ...
  script:
    - |
      mkdir -p ./platform/env
      echo -n nginx > ./platform/env/BP_WEB_SERVER
      echo -n ./dist > ./platform/env/BP_WEB_SERVER_ROOT
    - CNB_PLATFORM_API=0.12
      /cnb/lifecycle/creator
        -app=.
        -run-image=docker.io/paketobuildpacks/run-jammy-base:0.1.84
        -platform=./platform
        $CI_REGISTRY_IMAGE:latest

pack command:

pack build \
  --builder docker.io/paketobuildpacks/builder-jammy-base:0.4.244 \
  --run-image=docker.io/paketobuildpacks/run-jammy-base:0.1.84 \
  --lifecycle-image=docker.io/buildpacksio/lifecycle:0.17.2 \
  --network=host \
  gitlab-registry.example.com/k8s/buildpacks-vue-demo

(env set in project.toml)

check script:

docker run --rm --pull=always -q --entrypoint="/bin/bash" \
  gitlab-registry.example.com/k8s/buildpacks-vue-demo \
  -c "tail -n 6 /layers/config/metadata.toml && echo -n APP_ROOT: && cat /layers/paketo-buildpacks_nginx/nginx/env.launch/APP_ROOT.default"

1. Fresh build in GitLab CI, no previouse image

Job log:

Paketo Buildpack for Nginx Server 0.15.6
  Resolving Nginx Server version
    Candidate version sources (in priority order):
      buildpack.toml -> "1.25.*"
    Selected Nginx Server version (using buildpack.toml): 1.25.2
  Generating /builds/k8s/buildpacks-vue-demo/nginx.conf
    Setting server root directory to '{{ env "APP_ROOT" }}/dist'
    Setting server location path to '/'
  Executing build process
    Installing Nginx Server 1.25.2
      Completed in 492ms
  Configuring build environment
    PATH -> "$PATH:/layers/paketo-buildpacks_nginx/nginx/sbin"
  Configuring launch environment
    APP_ROOT   -> "/builds/k8s/buildpacks-vue-demo"
    EXECD_CONF -> "/builds/k8s/buildpacks-vue-demo/nginx.conf"
    PATH       -> "$PATH:/layers/paketo-buildpacks_nginx/nginx/sbin"
    PORT       -> "8080"
  Assigning launch processes:
    web (default): nginx -p /builds/k8s/buildpacks-vue-demo -c /builds/k8s/buildpacks-vue-demo/nginx.conf -g pid /tmp/nginx.pid;
  Generating SBOM for /layers/paketo-buildpacks_nginx/nginx
      Completed in 0s

Check result (same args and app root):

[[processes]]
  type = "web"
  command = ["nginx"]
  args = ["-p", "/builds/k8s/buildpacks-vue-demo", "-c", "/builds/k8s/buildpacks-vue-demo/nginx.conf", "-g", "pid /tmp/nginx.pid;"]
  direct = true
  buildpack-id = "paketo-buildpacks/nginx"
APP_ROOT:/builds/k8s/buildpacks-vue-demo

2. Pack build with local previouse image by GitLab CI

Command output:

[builder] Paketo Buildpack for Nginx Server 0.15.6
[builder]   Resolving Nginx Server version
[builder]     Candidate version sources (in priority order):
[builder]       buildpack.toml -> "1.25.*"
[builder] 
[builder]     Selected Nginx Server version (using buildpack.toml): 1.25.2
[builder] 
[builder]   Generating /workspace/nginx.conf
[builder]     Setting server root directory to '{{ env "APP_ROOT" }}/dist'
[builder]     Setting server location path to '/'
[builder] 
[builder]   Reusing cached layer /layers/paketo-buildpacks_nginx/nginx
[builder] 
[builder] Timer: Builder ran for 1.891638438s and ended at 2023-10-31T03:52:19Z

Compare to step 1, there are no "Configuring launch environment" and other jobs, just "Reusing cached layer". docker push gitlab-registry.example.com/k8s/buildpacks-vue-demo then check, see the result in above "Current Behavior".

3. Pack build after delete local and remote images

Command output:

[builder] Paketo Buildpack for Nginx Server 0.15.6
[builder]   Resolving Nginx Server version
[builder]     Candidate version sources (in priority order):
[builder]       buildpack.toml -> "1.25.*"
[builder] 
[builder]     Selected Nginx Server version (using buildpack.toml): 1.25.2
[builder] 
[builder]   Generating /workspace/nginx.conf
[builder]     Setting server root directory to '{{ env "APP_ROOT" }}/dist'
[builder]     Setting server location path to '/'
[builder] 
[builder]   Executing build process
[builder]     Installing Nginx Server 1.25.2
[builder]       Completed in 107ms
[builder] 
[builder]   Configuring build environment
[builder]     PATH -> "$PATH:/layers/paketo-buildpacks_nginx/nginx/sbin"
[builder] 
[builder]   Configuring launch environment
[builder]     APP_ROOT   -> "/workspace"
[builder]     EXECD_CONF -> "/workspace/nginx.conf"
[builder]     PATH       -> "$PATH:/layers/paketo-buildpacks_nginx/nginx/sbin"
[builder]     PORT       -> "8080"
[builder] 
[builder]   Assigning launch processes:
[builder]     web (default): nginx -p /workspace -c /workspace/nginx.conf -g pid /tmp/nginx.pid;
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_nginx/nginx
[builder]       Completed in 0s

No "Reusing cached layer" but "Configuring launch environment", same as step 1, docker push then check (same args and app root):

[[processes]]
  type = "web"
  command = ["nginx"]
  args = ["-p", "/workspace", "-c", "/workspace/nginx.conf", "-g", "pid /tmp/nginx.pid;"]
  direct = true
  buildpack-id = "paketo-buildpacks/nginx"
APP_ROOT:/workspace

4. GitLab CI with previouse image by pack CLI

Job log:

Paketo Buildpack for Nginx Server 0.15.6
  Resolving Nginx Server version
    Candidate version sources (in priority order):
      buildpack.toml -> "1.25.*"
    Selected Nginx Server version (using buildpack.toml): 1.25.2
  Generating /builds/k8s/buildpacks-vue-demo/nginx.conf
    Setting server root directory to '{{ env "APP_ROOT" }}/dist'
    Setting server location path to '/'
  Reusing cached layer /layers/paketo-buildpacks_nginx/nginx
Timer: Builder ran for 2m23.572470167s and ended at 2023-10-31T04:08:48Z

Reusing cache again, check result (different args and app root):

[[processes]]
  type = "web"
  command = ["nginx"]
  args = ["-p", "/builds/k8s/buildpacks-vue-demo", "-c", "/builds/k8s/buildpacks-vue-demo/nginx.conf", "-g", "pid /tmp/nginx.pid;"]
  direct = true
  buildpack-id = "paketo-buildpacks/nginx"
APP_ROOT:/workspace

Possible Solution

• Fix the determining condition before "Reusing cached layer".
• Keep same working dir. , we can custom build directories by set GIT_CLONE_PATH=/workspace. (We cant: the GIT_CLONE_PATH="/workspace" has to be within "/builds")
  • From pack CLI side: pack build --workspace=/builds/group/project.
  • From GitLab CI side: move $CI_PROJECT_DIR/* to /workspace first, but it is ugly.
  • Even no this problem, we should keep same directory layout in app image.

and in run.go:

	if _, err := os.Stat(mainConf); err != nil {
		return nil
	}

this slient fail is why client_body_temp_path error but not nginx.conf 404?