Essentia-Tensorflow on ARM fail compilation #1478
Description
Hi everyone,
I need to compile essentia-tensorflow to run on ARM. More in details I want to create a container with essentia-tensorflow that then can run my open-source app based on essentia AND tensorflow on ARM.
I was ableto create a dockerfile that work on Intel because, basically, I didn't recompile nothing:
FROM ubuntu:22.04
ENV LANG=C.UTF-8 \
PYTHONUNBUFFERED=1 \
DEBIAN_FRONTEND=noninteractive
WORKDIR /app
RUN apt-get update -o Acquire::Retries=5 -o Acquire::Timeout=30 && \
apt-get install -y --no-install-recommends \
python3 python3-pip python3-dev \
libfftw3-3 libyaml-0-2 libtag1v5 libsamplerate0 \
ffmpeg wget git vim \
redis-tools curl \
supervisor \
strace \
procps \
iputils-ping \
libopenblas-dev \
liblapack-dev \
# Added dependencies for psycopg2-binary
libpq-dev \
gcc \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
RUN pip3 install --no-cache-dir numpy==1.26.4
RUN pip3 install --no-cache-dir \
Flask \
Flask-Cors \
redis \
requests \
scikit-learn \
rq \
pyyaml \
six \
# Added psycopg2-binary for PostgreSQL connectivity
psycopg2-binary \
ftfy \
flasgger \
sqlglot \
google-generativeai
RUN pip3 install --no-cache-dir essentia-tensorflow
# Create the model directory
RUN mkdir -p /app/model
# Download models from the GitHub release
RUN wget -q -P /app/model \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/audioset-vggish-3.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/danceability-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/danceability-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_aggressive-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_aggressive-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_happy-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_happy-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_party-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_party-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_relaxed-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_relaxed-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_sad-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_sad-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/msd-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/msd-musicnn-1.pb
# Copy the application code
COPY . /app
# Copy the supervisor configuration
COPY deplyment/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ENV PYTHONPATH=/usr/local/lib/python3/dist-packages:/app
EXPOSE 8000
WORKDIR /workspace
CMD ["bash", "-c", "if [ \"$SERVICE_TYPE\" = \"worker\" ]; then echo 'Starting worker processes via supervisord...' && /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf; else echo 'Starting web service...' && python3 /app/app.py; fi"]
Starting from this I try to add dependency and making different test to run the same app but on ARM
Looking here tensorflow already exist for arm:
So I just installed it with PIP.
Then I re-view all the instruction on how to compiling it from here:
and I also take advantages from the instruction in this issue:
and I created this Dockerfile.arm:
# STAGE 1: Build Essentia from source for the target architecture
FROM ubuntu:22.04 AS builder
# Install build dependencies for Essentia.
# Added python3-setuptools, a core requirement for building Python extensions.
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
wget \
python3-pip \
pkg-config \
libfftw3-dev \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libswresample-dev \
libchromaprint-dev \
libyaml-dev \
libyaml-cpp-dev \
libtag1-dev \
libsamplerate0-dev \
python3-dev \
python3-numpy \
python3-setuptools \
&& rm -rf /var/lib/apt/lists/*
# Install TensorFlow for ARM64 using pip. This provides the necessary C libraries and headers.
RUN pip3 install tensorflow-aarch64==2.16.1
# Download and extract the Essentia source code
WORKDIR /
RUN wget -qO essentia.tar.gz https://github.com/MTG/essentia/archive/refs/tags/v2.1_beta5.tar.gz && \
tar -xzf essentia.tar.gz && \
rm essentia.tar.gz
# Set the working directory to the extracted source folder.
WORKDIR /essentia-2.1_beta5
# Set the PKG_CONFIG_PATH to help the build script find the Python development files.
ENV PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
# Build and install using the Waf build system.
# Waf will automatically detect the TensorFlow library installed by the pip package.
RUN python3 ./waf configure --build-static --with-python && \
python3 ./waf && \
python3 ./waf install
# --- End of builder stage ---
# STAGE 2: Final application image
FROM ubuntu:22.04
ENV LANG=C.UTF-8 \
PYTHONUNBUFFERED=1 \
DEBIAN_FRONTEND=noninteractive
WORKDIR /app
# Install runtime dependencies
RUN apt-get update -o Acquire::Retries=5 -o Acquire::Timeout=30 && \
apt-get install -y --no-install-recommends \
python3 python3-pip python3-dev \
libfftw3-3 libyaml-0-2 libtag1v5 libsamplerate0 \
ffmpeg wget git vim \
redis-tools curl \
supervisor \
strace \
procps \
iputils-ping \
libopenblas-dev \
liblapack-dev \
# Added dependencies for psycopg2-binary
libpq-dev \
gcc \
g++ \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
# Install Python packages
RUN pip3 install --no-cache-dir numpy==1.26.4
RUN pip3 install --no-cache-dir \
Flask \
Flask-Cors \
redis \
requests \
scikit-learn \
rq \
pyyaml \
six \
annoy \
psycopg2-binary \
ftfy \
flasgger \
sqlglot \
google-generativeai
# Copy pre-built Essentia libraries and Python bindings from builder
COPY --from=builder /usr/local/lib/python3.10/dist-packages/essentia* /usr/local/lib/python3.10/dist-packages/
COPY --from=builder /usr/local/lib/libessentia.so /usr/local/lib/
# Create model directory
RUN mkdir -p /app/model
# Download models
RUN wget -q -P /app/model \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/audioset-vggish-3.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/danceability-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/danceability-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_aggressive-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_aggressive-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_happy-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_happy-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_party-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_party-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_relaxed-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_relaxed-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_sad-audioset-vggish-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/mood_sad-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/msd-msd-musicnn-1.pb \
https://github.com/NeptuneHub/AudioMuse-AI/releases/download/v1.0.0-model/msd-musicnn-1.pb
# Copy application code
COPY . /app
# Copy Supervisor config
COPY deplyment/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ENV PYTHONPATH=/usr/local/lib/python3/dist-packages:/app
EXPOSE 8000
WORKDIR /workspace
CMD ["bash", "-c", "if [ \"$SERVICE_TYPE\" = \"worker\" ]; then echo 'Starting worker processes via supervisord...' && /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf; else echo 'Starting web service...' && python3 /app/app.py; fi"]
This seems to compile but block in the connection to python:
#14 [builder 5/7] WORKDIR /essentia-2.1_beta5
#14 DONE 0.0s
#15 [builder 6/7] RUN python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install
#15 1.401 /essentia-2.1_beta5/wscript:301: SyntaxWarning: "is not" with a literal. Did you mean "!="?
#15 1.401 if ctx.env.CXX_NAME is not "clang":
#15 1.854 Setting top to : /essentia-2.1_beta5
#15 1.861 Setting out to : /essentia-2.1_beta5/build
#15 1.877 → configuring the project in /essentia-2.1_beta5
#15 1.877 → Building in release mode
#15 1.877 Checking for 'g++' (C++ compiler) : /usr/bin/g++
#15 3.303 Checking for 'gcc' (C compiler) : /usr/bin/gcc
#15 3.789 → Searching *.pc pkg-config files for dependencies in /usr/lib/aarch64-linux-gnu/pkgconfig/
#15 3.789 Checking for program 'pkg-config' : /usr/bin/pkg-config
#15 3.793 Checking for 'libavcodec' >= 55.34.1 : yes
#15 3.848 Checking for 'libavformat' : yes
#15 3.900 Checking for 'libavutil' : yes
#15 3.949 Checking for 'libavresample' : not found
#15 3.999 Checking for 'samplerate' : yes
#15 4.049 Checking for 'taglib' >= 1.9 : yes
#15 4.098 Checking for 'yaml-0.1' : yes
#15 4.147 Checking for 'fftw3f' : yes
#15 4.196 Checking for 'libchromaprint' : yes
#15 4.266 Checking for 'gcc' (C compiler) : /usr/bin/gcc
#15 4.737 Checking for program 'python' : /usr/bin/python3
#15 ...
#12 [stage-1 5/12] RUN pip3 install --no-cache-dir Flask Flask-Cors redis requests scikit-learn rq pyyaml six annoy psycopg2-binary ftfy flasgger sqlglot google-generativeai
#12 107.0 Collecting grpcio<2.0.0,>=1.33.2
#12 107.1 Downloading grpcio-1.73.1-cp310-cp310-manylinux_2_17_aarch64.whl (5.8 MB)
#12 107.3 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 26.0 MB/s eta 0:00:00
#12 108.2 Collecting grpcio-status<2.0.0,>=1.33.2
#12 108.2 Downloading grpcio_status-1.73.1-py3-none-any.whl (14 kB)
#12 CANCELED
#15 [builder 6/7] RUN python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install
#15 6.249 Checking for python version >= 2.7.0 : 3.10.12
#15 6.663 → Configuring for python3
#15 6.663 python-config : /usr/bin/python3-config
#15 6.666 Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
#15 7.102 Testing pyext configuration : Could not build python extensions
#15 7.270 Testing pyext configuration : Could not build python extensions
#15 7.415 The configuration failed
#15 7.415 (complete log in /essentia-2.1_beta5/build/config.log)
#15 ERROR: process "/bin/sh -c python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install" did not complete successfully: exit code: 1
------
> [builder 6/7] RUN python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install:
/usr/bin/gcc
4.737 Checking for program 'python' : /usr/bin/python3
6.249 Checking for python version >= 2.7.0 : 3.10.12
6.663 → Configuring for python3
6.663 python-config : /usr/bin/python3-config
6.666 Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
7.102 Testing pyext configuration : Could not build python extensions
7.270 Testing pyext configuration : Could not build python extensions
7.415 The configuration failed
7.415 (complete log in /essentia-2.1_beta5/build/config.log)
------
Dockerfile.arm:44
--------------------
43 | # Waf will automatically detect the TensorFlow library installed by the pip package.
44 | >>> RUN python3 ./waf configure --build-static --with-python && \
45 | >>> python3 ./waf && \
46 | >>> python3 ./waf install
47 |
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c python3 ./waf configure --build-static --with-python && python3 ./waf && python3 ./waf install" did not complete successfully: exit code: 1
I have the impression that essentia 2.1_beta5, even if the last version, is old of many years so when I go to installthe dependencies I have to new version.
How can I discover the exact version of the dependencies to sovle this?
In my different test I'm also trying to use an older version of numpy that then ask for an older version of ubuntu, thinking that it could be the issue (for the intel version infact I had to fix to specific version numpy) but still nothing.
Maybe also install tensorflow by PIP bring me to a newer version?
If needed the Dockerfile.arm is build with this github workflow, if you want to reproduce the issue:
https://raw.githubusercontent.com/NeptuneHub/AudioMuse-AI/refs/heads/main/.github/workflows/build-arm.yml