Bash tools missing from 3.3.0 release

[thomasritz]

If one ECS Fargate container of a task has a health check configured, then all containers must have a health check.
With 3.2.0, I could use true as dummy health check.
With 3.3.0, this health check fails due to true no longer being available in the image.

It would be nice if there was an explicit health check command or so for the daemon.

[rbpltr]

Same here. Just had to spend over an hour picking apart a CloudFormation stack after an ECS service deployment failed and then failed to rollback because we were using the latest version of this container.

[awssandra]

Hi all,

This is likely due to a bug that was released in 3.3.0 - please see: #120

We have release v3.3.1 as of last night. Please let us know if this has alleviated your issues.
https://github.com/aws/aws-xray-daemon/releases

Thanks,
Sandra

[thomasritz]

Hi,

I think #120 is unrelated.
My problem was simply that due to changes in how the images was built some dependencies were removed from 3.2.0 to 3.3.0.

The image of 3.2.0 was built with a lot of tools included such as /usr/bin/true (which I used as dummy health check command).

Newer releases are built from scratch and therefore almost empty.

I'm a big fan of small images, don't get me wrong :-)
Thanks for having a look into this.

I solved this issue for now by locking my taskdef.json to v3.2.0: "image": "amazon/aws-xray-daemon:3.2.0"

[awssandra]

Gotcha, thanks for the explanation.

Apologies for the trouble!
Yes we cut down the image size from 150 MB -> 3.5 MB, by moving from an amazonlinux base image to scratch, which consequently, removed a lot of tools available.
We're are reviewing currently, will get back to you as soon as we can.

Related issue: #126

[CarlosDomingues]

I was using the latest tag and defining the entrypoint explicitly to use local mode (/usr/bin/xray, you can find this configuration in the official AWS docs) and those changes also broke X-Ray daemon in prod for me.

Suggestion: maintain the old images and create a slim tag for folks who want smaller images.

[awssandra]

Hey CarlosDomingues,

Apologies for the issues.

We will be republishing the Dockerhub images using the base amazonlinux, and maintain this as the standard solution for Dockerhub users, as it was established as the standard from day one.

We will be updating guidance on this in our public documentation, as well as supplying build instructions for those who wish to incorporate the extra tools into their ECR images for healthchecks or other operational items.

[awssandra]

#128

[awssandra]

New release 3.3.2 has been completed. Will be updating the docs accordingly.

[ollypom]

Hey @awssandra , shipping the same image tag but as 2 different artefacts to 2 different registries seems a bit of anti-pattern to me. I would expect aws-xray-daemon:3.3.2 to be the same image regardless of the registry I pull it from.

I understand the reason for having 2 Dockerfiles, the amount of embedded tools the end user requires varies, but I would expect the tag to be consistent regardless of its location. I think the problem you are going to have is that folks hitting the Docker Hub rate limitting issue are just going to swap out the registries in their Task definitions and expect it to work. But as we've seen in this thread, their existing health checks will break when moving from Hub to ECR and they won't know why.

If you look at some of the Docker official images the naming convention is x.y.z-baseimage when multiple base images are used:

• Python has 3.9.4-buster and 3.9.4-alpine
• Nginx has 1.20 and 1.20-alpine
• Haproxy has 2.30.10 and 2.30.10-alpine

So my suggestion would be to keep DockerHub and ECR Public in sync, with aws-xray-daemon:3.3.2 being the same image on both registries. But at the same time create a 3.3.2-amazonlinux or 3.3.2-slim (or whatever naming convention suits) stored in both registries with more tools and the ability to use a healthcheck.

For example:

aws-xray-daemon:3.3.2 from Dockerfile
aws-xray-daemon:3.3.2-amazonlinux from Dockerfile.amazonlinux

[fosrias]

aws-xray-daemon:3.3.2 to be the same image regardless of the registry I pull it from

Already ran into this. Was confusing as I expected to be fixed, but was getting the same problems when using ECR, IIUC.