Application droplet can be started with a different stack than the buildpack used to produce the droplet supports

[beyhan]

Issue

Using the CF APIs it is possible to launch an application droplet with a CF stack which is different than the stack supported by the buildpack used to produce the droplet.

Steps to Reproduce

Prerequisite is a CF offering two different stacks. E.g. cflinuxfs3 and cflinuxfs4. For the steps below we want to switch an application from the cflinuxfs3 to cflinuxfs4 stack.

1. Get the app latest package guid with: cf curl "/v3/packages?app_guids=<app-guid>&order_by=-created_at&states=READY"
2. Create a new droplet with the cflinuxfs4 stack and the corresponding buildpack supporting the cflinuxfs4 stack:

  curl curl "/v3/builds" \
      -X POST \
      -H "Content-type: application/json" \
      -d '{
        "package": {
          "guid": "[package-guid]"
         },
         lifecycle: {
           type: "buildpack",
           data: { buildpacks: [<cflinuxfs4-buildpack>], stack: cflinuxfs4}
        }
      }'

3. Wait until the build is done and get the new droplet guid with: cf curl "/v3/builds/<build-guid>"
4. Stop the app with: cf curl "/v3/apps/<guid>/actions/stop" -X POST"
5. Set the app's current droplet:

cf curl "/v3/apps/[guid]/relationships/current_droplet" \
  -X PATCH \
  -H "Content-type: application/json" \
  -d '{ "data": { "guid": "[<droplet_guid>]" } }'

6. Restart the app: cf curl "/v3/apps/[guid]/actions/restart" -X POST"

Expected result

CF restart request fails because the current stack of the application configured in its lifecycle is cflinuxfs3 but the current droplet is built for the cflinuxfs4 stack.

Current result

The application is restarted with its configured stack which is cflinuxfs3 and e.g. node apps are crashing with strange errors like:

 [APP/PROC/WEB/0] STDERR node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by node)
 [APP/PROC/WEB/0] STDERR node: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by node)

This is happening because the droplet has dependencies to the CF stack and they can't be resolved.

Possible Fix

Possible fix could be to reject such a start/restart request when the droplet stack is different than the app lifecycle stack. Such an error is happening when you try to push an app with a buildpack which doesn't support the current lifecycle stack of the app:

 [STG/0] OUT **ERROR** Stack not supported by buildpack: required stack cflinuxfs3 was not found
 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 11

[Gerg]

Yea, it seems counterproductive to let people run droplets that were staged on an incompatible stack.

---

Funnily enough, @dsboulder and I were chatting earlier about having divergent "build" and "run" stacks, like they do with Paketo. That could help reduce the number of OS packages included in the run container.

If we do something like that in the future (maybe along with CNB support), then running with a different stack than the droplet's stack would be desirable.

[beyhan]

Having "build" and "run" stacks will be another step into improving the hardening of the app container. The information regarding the OS on which the stacks are based on will be still available and for me it make sense to support running apps on "run" stacks which were built with "build" stacks based on the same OS version. Finding the root cause of this issue could be challenging for none experience users.

[Gerg]

As a follow-up to my comment above, we ended up making it so a single stack can use different run and build rootFSs: https://v3-apidocs.cloudfoundry.org/version/3.164.0/index.html#the-stack-object. This is an alternative to using differnt stacks for run & build, which makes it easier to enforce compatibility.