with-docker and shared dependancies - how to cache the non-shared dependancies?

[yuval-illumex]

Which project is this feature idea for?

Turborepo

Describe the feature you'd like to request

Describe the feature you'd like to request
Upon reviewing the with-docker example, I have got a question regarding best practice for the final production image.

Here is the structure of my projects:

• applications
  • webserver (nest.js)
  • client
• libs
  • shared
    • (package.json)

Here is the dockerfile, based on the docker example in the documentation:

FROM node:18-alpine AS base
RUN apk add --no-cache libc6-compat
RUN apk update
WORKDIR /app
RUN yarn global add turbo
RUN corepack enable

FROM base AS builder-server
COPY . .
RUN turbo prune --scope=@myOrg/server --docker

FROM base AS build-server
COPY tsconfig.json tsconfig.json
# COPY /libs/shared /app/libs/shared
COPY --from=builder-server /app/out/json/ .
COPY --from=builder-server /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
RUN pnpm install --frozen-lockfile --filter=@myOrg/server...
COPY --from=builder-server /app/out/full/ .
RUN pnpm build:server

FROM node:18-alpine AS runtime
COPY --from=build-server /app/apps/server/dist /app/apps/server/dist
COPY --from=build-server /app/apps/server/node_modules /app/apps/server/node_modules
COPY --from=build-server /app/libs/shared /app/libs/shared
COPY --from=build-server /app/node_modules /app/node_modules
WORKDIR /app
ENTRYPOINT ["node", "apps/server/dist/main"]

My question is about caching the production node_modules.
In the build-server layer, I can't install with the --production flag - because I need the devDependices for the build process.
However, If I will add a layer, that only install the node_modules for running in production:

FROM base AS build-server-deps
COPY --from=builder-server /app/out/json/ .
COPY --from=builder-server /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
RUN pnpm install --frozen-lockfile --filter=@myOrg/server... --production

I have a problem with the "shared" lib. In build-server stage, the "shared" lib is installed into the node_modules folder, but in the build-server-deps it won't exists. In addition, the "shared" lib is installed as symlink, so I don't see any other option.

Describe the solution you'd like

To have the ability to install only "source npm" libraries and cache them, and then to build the internal packages. Of course the internal package can use another "source npm" package, and it should be cached as expected.

Maybe there is an option to "merge" node modules folder, and then to copy @myorg from the node_modules of the build-server stage, and all the rest from build-server-deps stage.

Describe alternatives you've considered

At the moment I copy all the build dependencies, and not only the production one. Didn't found a better way
I thought about adding RUN pnpm prune --prod after building the server, but in the docker build logs I see the command is waiting for input from the user

[DesignByOnyx]

I do two separate npm installs: one so that I can build and a second one for --production.

The thing that is regrettably missing from the turbo docs is that once you build, you only want to keep the dist folders and other build artifacts required for production. Here's what I'm doing:

FROM base AS build-server
COPY tsconfig.json tsconfig.json
COPY --from=builder-server /app/out/json/ .
COPY --from=builder-server /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
# do a full install - required for build
RUN pnpm install --frozen-lockfile --filter=@myOrg/server...
COPY --from=builder-server /app/out/full/ .
# this calls `turbo build` eventually
RUN pnpm build:server

# Now, we only want to keep the `dist` folders, package.json files, and whatever else necessary for production
# This is sort of like a "production prune", and this is the part that is woefully omitted from the turbo docs or guides
# I am using linux commands to copy all production assets to a "release" folder
RUN mkdir release
COPY --from=builder-server /app/out/json/ .
COPY --from=builder-server /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
# Use rsync to copy files to the release folder. You can easily configure additional files to include or exclude as necessary.
RUN rsync -avz --include='packages/*/dist/' --exclude="packages/*/*" ./packages ./release/
RUN rsync -avz --include='services/*/dist/' --exclude="services/*/*"  ./services ./release/

# Change dirctories to the release folder
WORKDIR /app/release
# Run production pnpm install - this should be fast because the previous install hydrated the cache
# You should not need to use `--filter` here - but do what you will
RUN pnpm install --frozen-lockfile --production

# Now copy your release assets into your final prod image
FROM base AS prod
WORKDIR /app
COPY --from=prod-staging /app/release/ ./
# I recommend rebuilding on the final machine in case you are using a different distro
RUN pnpm rebuild
ENTRYPOINT ["node", "apps/server/dist/main"]