Alpine images leak memory for each OpenSSL connection

[jgaskins]

TL;DR

Every time an OpenSSL connection is made in a Crystal program built with the Alpine containers (tested on linux/arm64 images), there is a memory leak. I thought this was a Crystal bug at first, but changing the version of OpenSSL used in this repo's Alpine containers eliminated the leak.

Reproducing the bug

To reproduce this bug, you can use this Crystal file:

# src/http_get.cr
require "http/client"

loop { HTTP::Client.get "https://example.com" }

And build a container defined by this Dockerfile:

FROM 84codes/crystal:1.17.0-alpine AS builder

WORKDIR /build
COPY src/http_get.cr .
RUN crystal build --release --static --progress --stats http_get.cr -o http_get

FROM scratch

COPY --from=builder /build/http_get /
CMD ["/http_get"]

If I've copy/pasted this all from the right places, when you run that container, it will consume an ever-increasing amount of RAM.

Note that reusing TCP connections and/or using a connection pool mitigates this memory leak. It doesn't eliminate it, but there is a workaround as long as you're able to maintain long-lived connections to a low-cardinality set of remote hosts. For most of us, I assume that's true.

Solution (kinda)

Following this advice, I based another container image off of it using this Dockerfile:

ARG version=latest

FROM 84codes/crystal:$version-alpine AS builder

# The OpenSSL packages that come with Alpine have a memory leak, causing
RUN apk add curl perl linux-headers && \
    curl -Ls "https://github.com/openssl/openssl/releases/download/openssl-3.3.3/openssl-3.3.3.tar.gz" | tar xz && \
    cd openssl-3.3.3 && \
    ./Configure --openssldir=/etc/ssl && \
    make -j

FROM 84codes/crystal:$version-alpine

COPY --from=builder /openssl-3.3.3/*.so.3 /openssl-3.3.3/*.a /usr/lib/
COPY --from=builder /openssl-3.3.3/*.pc /usr/lib/pkgconfig/

I've published this container here. If you change the FROM ... AS builder line in the first Dockerfile above to the following, the memory leak disappears:

FROM jgaskins/crystal:1.17.0 AS builder

My container image isn't a drop-in replacement, though. It's just a proof of concept. Simply overwriting OpenSSL lib files, as you might imagine, breaks things like apk. I assume crystal can pull from a different OpenSSL installation than the one in /usr/lib, which would probably be a more useful solution.

[carlhoerberg]

Ugh, sounds like yet another Alpine bug then. Their shipped openssl version shouldn't leak memory? Personally I try to stay clear of alpine whenever possible. It's small and easy to get started with, but very slow because of musl and as here, bug prone shipped libraries. Will take a look, but downloading and building a custom openssl version does not sound like the right approach.

[carlhoerberg]

I recommend everybody to base their containers of debian:slim, it's just a few MB larger than alpine and way faster.

[carlhoerberg]

I can't reproduce, I had to copy the ssl certs in the dockerfile like so:

FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/cert.pem
COPY --from=builder /build/http_get /
CMD ["/http_get"]

but then i had no memory leak. Did you pull the latest image when you compiled? podman build --pull .

[carlhoerberg]

ah, i was testing in an amd64 environment, not arm, maybe it's platform specific.

[jgaskins]

Their shipped openssl version shouldn't leak memory

I agree, I was very surprised to find discover that the leak was in there. I hadn't even considered the possibility until one of the Crystal core devs pointed it out. And even then I only looked into it because my whole Crystal program was only about 60 lines of code with no state — it just fetches data from one HTTPS service, transformed it, and sent it to another HTTPS service.

I'll try debian:slim, too. I only use Alpine so I can ship single-binary images.

ah, i was testing in an amd64 environment, not arm, maybe it's platform specific.

It's possible, but I can't imagine why the arm64 version wouldn't call free under the same conditions that the amd64 version does. I'll see if I can reproduce it today on amd64.

Also, to clarify, it does take significant time to show the leak so presumably it's only a small number of bytes leaked for each connection. Below is the graph of memory usage where I finally began looking into it, sending 1-2 HTTPS requests per minute, each on a fresh TCP connection. It took a few days at that pace to get to 70MB. If you're able to make an HTTPS connection to something with very low latency (on the order of 5ms) very quickly, you might be able to produce the leak much faster.

[carlhoerberg]

Could try to replace openssl with libressl, but libressl-static is only available in edge, not latest (3.12)

[erusc]

For 3.22 and older, libssl.a and libcrypto.a are included in libressl-dev.