failure at TASK [Ensure internet-monitoring environment is running.]

[ventermaarten-ui]

[ERROR]: Task failed: Module failed: General error: failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:c731b7986d62e5b932f888fa4b4c6539247b88d5c795bc180f5aebf82d0821b3) to overlayfs as "extract-501462120-_dZ- sha256:3d2c676562d0a0ece7d4c32229b8c830c5865424af78a3f55d61f97ca632ae01": failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not supported
Origin: /home/pi/internet-pi/tasks/internet-monitoring.yml:47:3

45 become: false
46
47 - name: Ensure internet-monitoring environment is running.
^ column 3

fatal: [127.0.0.1]: FAILED! => {"actions": [{"id": "prom/prometheus:latest", "status": "Pulling", "what": "image"}, {"id": "prom/blackbox-exporter:latest", "status": "Pulling", "what": "image"}, {"id": "grafana/grafana:latest", "status": "Pulling", "what": "image"}, {"id": "miguelndecarvalho/speedtest-exporter:latest", "status": "Pulling", "what": "image"}, {"id": "prom/node-exporter:latest", "status": "Pulling", "what": "image"}], "changed": false, "cmd": "/usr/bin/docker compose --ansi never --progress json --project-directory /root/internet-monitoring up --detach --no-color --quiet-pull --no-build --", "containers": [], "images": null, "msg": "General error: failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:c731b7986d62e5b932f888fa4b4c6539247b88d5c795bc180f5aebf82d0821b3) to overlayfs as "extract-501462120-_dZ- sha256:3d2c676562d0a0ece7d4c32229b8c830c5865424af78a3f55d61f97ca632ae01": failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not supported", "rc": 1, "stderr": "{"id":"Image prom/prometheus:latest","status":"Working","text":"Pulling"}\n{"id":"Image prom/blackbox-exporter:latest","status":"Working","text":"Pulling"}\n{"id":"Image grafana/grafana:latest","status":"Working","text":"Pulling"}\n{"id":"Image miguelndecarvalho/speedtest-exporter:latest","status":"Working","text":"Pulling"}\n{"id":"Image prom/node-exporter:latest","status":"Working","text":"Pulling"}\n{"error":true,"message":"failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:c731b7986d62e5b932f888fa4b4c6539247b88d5c795bc180f5aebf82d0821b3) to overlayfs as \"extract-501462120-_dZ- sha256:3d2c676562d0a0ece7d4c32229b8c830c5865424af78a3f55d61f97ca632ae01\": failed to convert whiteout file \"etc/ca-certificates/.wh..wh..opq\": operation not supported"}\n", "stderr_lines": ["{"id":"Image prom/prometheus:latest","status":"Working","text":"Pulling"}", "{"id":"Image prom/blackbox-exporter:latest","status":"Working","text":"Pulling"}", "{"id":"Image grafana/grafana:latest","status":"Working","text":"Pulling"}", "{"id":"Image miguelndecarvalho/speedtest-exporter:latest","status":"Working","text":"Pulling"}", "{"id":"Image prom/node-exporter:latest","status":"Working","text":"Pulling"}", "{"error":true,"message":"failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:c731b7986d62e5b932f888fa4b4c6539247b88d5c795bc180f5aebf82d0821b3) to overlayfs as \"extract-501462120-_dZ- sha256:3d2c676562d0a0ece7d4c32229b8c830c5865424af78a3f55d61f97ca632ae01\": failed to convert whiteout file \"etc/ca-certificates/.wh..wh..opq\": operation not supported"}"], "stdout": "", "stdout_lines": []}

[ventermaarten-ui]

Could you please help?

[quadespresso]

How/where did you try to install to? Specifically, what is the filesystem on the target? Is it ext4, or is it something else?

[ventermaarten-ui]

Filesystem Size Used Avail Use% Mounted on
udev 963M 0 963M 0% /dev
tmpfs 196M 408K 196M 1% /run
overlay 26G 112M 25G 1% /
tmpfs 978M 0 978M 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 978M 0 978M 0% /sys/fs/cgroup
tmpfs 196M 0 196M 0% /run/user/1000

[quadespresso]

Over on the target host, run these commands:

docker info | egrep -i 'Storage Driver|Backing Filesystem|Docker Root Dir'

df -T "$(docker info -f '{{.DockerRootDir}}')"

and share back here.

Also, could you include formatting of any shell-based results? Just the triple-backticks for blocks would help, thanks.

[ventermaarten-ui]

Dear [quadespresso]

If I run the [docker info | egrep -i 'Storage Driver|Backing Filesystem|Docker Root Dir'] I get:

-bash: docker: command not found

[ventermaarten-ui]

It seems docker is not installed ... thus I installed it below:

To install Docker on Ubuntu, the recommended method is to use the official Docker APT repository to ensure you get the latest stable version.
Prerequisites
An Ubuntu system (20.04, 22.04, 24.04, or the latest non-LTS version are supported).
A user account with sudo privileges.
An active internet connection.

Step 1: Set up the Docker APT repository
First, update your package lists and install necessary packages to allow apt to use a repository over HTTPS.
bash
sudo apt update
sudo apt install ca-certificates curl gnupg software-properties-common -y

Next, add the official Docker GPG key:
bash
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Finally, add the Docker repository to your APT sources:
bash
echo
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Update your package index again to include the Docker packages from the newly added repo:
bash
sudo apt update

Step 2: Install Docker Engine
Now you can install the latest version of Docker Engine, CLI, and Compose plugin:
bash
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
The Docker service should start automatically after installation. You can verify its status:
bash
sudo systemctl status docker
The output should show that the service is active (running).

Step 3: Verify the installation
Confirm that the installation was successful by running the hello-world test image:
bash
sudo docker run hello-world
This command will download a test image, run it in a container, and print a confirmation message.

Step 4: Manage Docker without sudo (Optional)
By default, you need sudo to run Docker commands. To run Docker commands as a non-root user, add your user to the docker group:
bash
sudo usermod -aG docker ${USER}
For this change to take effect, you must log out and log back in to your session, or restart your system. Once you have relogged in, you can verify it works by running the hello-world command without sudo:
bash
docker run hello-world

[ventermaarten-ui]

I tried to run: ansible-playbook main.yml

And it worked!

Now when I run: docker info | egrep -i 'Storage Driver|Backing Filesystem|Docker Root Dir'

I get:
Storage Driver: vfs
Docker Root Dir: /var/lib/docker

and when I run: df -T "$(docker info -f '{{.DockerRootDir}}')"

I get:
Filesystem Type 1K-blocks Used Available Use% Mounted on
overlay overlay 27295148 10455684 15387920 41% /

Thank you so much!!!

I did not know that I have to install docker seperately?

[quadespresso]

I did not know that I have to install docker seperately?

You got me wondering, as it's been a while for me. Here we go:

Ref: https://github.com/geerlingguy/internet-pi/tree/master/internet-monitoring#pre-requisites

[ventermaarten-ui]

Dear quadespresso

Is there a way to tell the inetrnet-pi to which server to do the speed test?

[quadespresso]

Is there a way to tell the inetrnet-pi to which server to do the speed test?

This is somewhat speculative and untested on my part, but... if we start here: https://github.com/geerlingguy/internet-pi/blob/master/internet-monitoring/README.md

In particular this line:

Stand-up a Docker Prometheus stack containing Prometheus, Grafana with blackbox-exporter, and speedtest-exporter to collect and graph home Internet reliability and throughput.

then follow this link: https://github.com/MiguelNdeCarvalho/speedtest-exporter
to this link: https://docs.miguelndecarvalho.pt/projects/speedtest-exporter/

where it provides this docker compose YAML:

version: "3.0"
services:
  speedtest-exporter:
    image: miguelndecarvalho/speedtest-exporter
    container_name: speedtest-exporter
    environment:
      - SPEEDTEST_PORT=<speedtest-port> #optional
      - SPEEDTEST_SERVER=<server-id> #optional
    ports:
      - 9798:9798
    restart: unless-stopped

Then we can have a look on the target Raspberry Pi under ~/internet-monitoring for the file docker-compose.yaml and look for the block that looks like this:

  speedtest:
    image: miguelndecarvalho/speedtest-exporter:latest
    restart: always
    healthcheck:
      test: "wget --no-verbose --tries=1 --spider http://0.0.0.0:9798/"
      timeout: 10s
    expose:
      - 9798
    ports:
      - 9798:9798
    networks:
      - back-tier

then we should be able to add an environment: block, something like this (be vigilant regarding the indentation):

  speedtest:
    image: miguelndecarvalho/speedtest-exporter:latest
    environment:
      - SPEEDTEST_SERVER=<server-id>
    restart: always
    healthcheck:
      test: "wget --no-verbose --tries=1 --spider http://0.0.0.0:9798/"
      timeout: 10s
    expose:
      - 9798
    ports:
      - 9798:9798
    networks:
      - back-tier

Before you restart the docker-compose stack, you will of course want to know which speedtest server ID to use. To do that, you will want to hop on over to your Raspberry Pi, exec into the correct container, run the right command to determine what alternative servers you can use, give the desired ID a quick test, then update the above YAML accordingly.

To find out which container to exec into:

docker ps | grep speedtest

The container name will almost certainly be internet-monitoring-speedtest-1. On my particular server, the output from that command looks like this:

f0cbf4610788   miguelndecarvalho/speedtest-exporter:latest   "python -u exporter.…"   4 months ago   Up 3 months (healthy)   0.0.0.0:9798->9798/tcp, [::]:9798->9798/tcp   internet-monitoring-speedtest-1

Assuming you also see internet-monitoring-speedtest-1 as the last column in the output, run this next:

docker exec -it internet-monitoring-speedtest-1 /bin/sh

which should land you at a prompt like this:

/app $

At this point, just run this:

speedtest --servers

where the output will begin with:

Closest servers:

    ID  Name                           Location             Country
==============================================================================

Decide which of the servers you're interested in, and make note of that server's ID. Using that ID, give it a quick sanity check:

speedtest --server-id <id>

If you get sane/satisfactory results from it, make note of that ID. Revisit the aforementioned docker-compose.yml file and be sure to plug in that numeric ID as appropriate, here:

    environment:
      - SPEEDTEST_SERVER=<server-id>

Once you've updated the file, you should be good to restart the docker-compose stack.

For general usage questions, I would recommend creating a new discussion over here: https://github.com/geerlingguy/internet-pi/discussions