OOM during sysimage creation

[ValentinKaisermayer]

I'm trying to package a Julia app as a docker image. Since, as of writing this, package relocatability is not solved; hence, this is the only way to share an app.

In my dockerfile I run:

###############################################################################################
# Stage 1: Build Stage
FROM julia:1.11.2-bullseye AS builder

# Install C compiler required for PackageCompiler and clean up afterward to reduce image size
RUN apt-get update && \
    apt-get install -y --no-install-recommends g++ && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Set environment for Julia compilation
ENV JULIA_CPU_TARGET=generic

# Create a non-root user and set up directories
RUN useradd --create-home --shell /bin/bash jl && mkdir /home/jl/app && chown -R jl:jl /home/jl/
WORKDIR /home/jl/app

# Switch to non-root user
USER jl

ENV JULIA_DEPOT_PATH "/home/jl/.julia"
ENV JULIA_REVISE "off"
ENV EARLYBIND "true"
ENV JULIA_NUM_PRECOMPILE_TAKS "4"

# Copy dependency files first to leverage Docker caching for dependencies
COPY --chown=jl:jl Project.toml .
COPY --chown=jl:jl deps/ deps/
RUN julia --project -e "using Pkg; Pkg.instantiate(); Pkg.precompile();"

# Copy the full application code and precompile with sysimage
COPY --chown=jl:jl compiled/ compiled/
COPY --chown=jl:jl App/ App/
# copy test data so that precompiling works
COPY --chown=jl:jl data/ data/
RUN julia --project -t auto -O3 --startup-file=no compiled/make.jl

###############################################################################################
# Stage 2: Production Stage
FROM debian:bullseye-slim  
# Install curl (for BinaryProvider) compiler required for PackageCompiler and clean up afterward to reduce image size
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Set up a non-root user for running the app
RUN useradd --create-home --shell /bin/bash jl && mkdir /home/jl/app && chown -R jl:jl /home/jl/
WORKDIR /home/jl/app
USER jl

# Set environment variables
ENV JULIA_REVISE "off"
ENV EARLYBIND "true"

# Copy only essential files from the build stage to the production stage
COPY --from=builder /home/jl/app/compiled/App /home/jl/app

# Since Julia's relocatability depends on each package, simply make a symlink to the artifacts directory in the compiled app
RUN mkdir -p /home/jl/.julia/artifacts && ln -s /home/jl/app/share/julia/artifacts /home/jl/.julia/artifacts

# Command to run the application
ENTRYPOINT ["/home/jl/app/bin/App"]
CMD ["--julia-args", "-t", "auto", "-O3", "--startup-file=no"]

and my make.jl file is:

using PackageCompiler

PackageCompiler.create_app(
    "App",
    "compiled/App";
    cpu_target="generic",
    sysimage_build_args=`-O3`,
    precompile_execution_file="compiled/precompile.jl",
    include_transitive_dependencies=false,
    include_lazy_artifacts=true,
    incremental=true,
    # filter_stdlibs=true,
)

I also have a precompile.jl script with a representative workload.
When I run docker build I get as far as to compiling the incremental sysimage. Then I get OOMs. I just have 8 GB of RAM and 2 GB of swap availiable. The same runs fine on a machine with 16 GB or 24 GB RAM.

My question: How can I reduce the resource need of PackageCompiler.jl?

BTW, the secrete sauce is this line:

RUN mkdir -p /home/jl/.julia/artifacts && ln -s /home/jl/app/share/julia/artifacts /home/jl/.julia/artifacts

which solves the relocatable issue, since I copy over the app from the build stage.

[ValentinKaisermayer]

I'm using WSL under Windows 11. Which by default has 50% of the host's memory available.

[ValentinKaisermayer]

I increased the swap space, now it works. But is very slow.

[PatrickHaecker]

I haven't checked this, but try to remove the -t auto. Lowering JULIA_NUM_PRECOMPILE_TAKS "4" might help, too. If this does not help, please try the other thread-related variables in the documentation and report back here.

As a rule of thumb, you should have around 1 GB of free RAM per Julia process.
What is your free RAM for Julia? Is it the 8 GB or 50% of 8 GB or less due to other WSL processes?
How many cores are available to Julia?

What is actionable about this issue?

• improve the documentation

We already warn the user when the problem is detected, but that probably does not help too much in cases like docker.
We probably do not want to redefine the semantics of --threads auto.
Reducing memory usage is always a fine goal, but a generic point here won't be actionable.

So I think after we learned what helped in this case, we can improve the documentation and close this issue.