feat: Add reproducible builds release workflows and push images to DockerHub

[MoeMahhouk]

Issue Addressed

This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the Makefile to streamline reproducible build configurations, and modifies the Dockerfile.reproducible to align with the new build process. Additionally, it removes the reproducible profile from Cargo.toml.

Proposed Changes

New GitHub Actions Workflows:

• .github/workflows/docker-reproducible.yml: Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs without pushing an image.

Build Configuration Updates:

• Makefile: Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments for x86_64 and aarch64 architectures.
• Dockerfile.reproducible: Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to the Makefile.
• Switch to using jemalloc-sys from Debian repos instead of building it from source. A Debian version is reproducible which is hard to achieve if you build it from source.

Profile Removal:

• Cargo.toml: Removes the reproducible profile, simplifying build configurations and relying on external tooling for reproducibility.

Additional Info

This is mainly a follow up to this work #6799 where I refine the reproducible build configuration to simplify the CI workflow to generate the reproducible images and pushes them to DockerHub. I also added a cron job workflow (inspired from the Reth repo) that checks every two days or pull requests that touches files that might affect reproducibility to catch potential regressions.
In case, this is too much, let me know and I can create a separate PR for this to be merged later when necessary

close #7486
close #7485

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ MoeMahhouk
❌ Ubuntu

---

Ubuntu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chong-he]

Doing some testing on this, will post the comment when ready

[mergify]

Some required checks have failed. Could you please take a look @MoeMahhouk? 🙏

[chong-he]

Thanks for the PR. I left some comments after doing some testing

[MoeMahhouk]

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it.
I will do some rework and refinements to this PR to fix the issues and address your comments.

[chong-he]

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it. I will do some rework and refinements to this PR to fix the issues and address your comments.

Is there a reason why is this PR still a draft?

[MoeMahhouk]

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it. I will do some rework and refinements to this PR to fix the issues and address your comments.

Is there a reason why is this PR still a draft?

Not really, I am waiting for your final feedback to open it for review/merge.
I will open it now for review and if you find anything needed for iteration, we can switch it back again.

[mergify]

Merge Queue Status

🚫 The pull request has left the queue (rule: default)

This pull request spent 33 minutes 42 seconds in the queue, including 32 minutes 3 seconds waiting for CI.
The checks were run on draft #8483.

Required conditions to merge

• check-success=local-testnet-success
• check-success=test-suite-success

Reason

Pull request #7614 has been dequeued. The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. (details: 1 review requesting changes and 2 approving reviews by reviewers with write access.)

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

Already addressed

[michaelsproul]

@mergify requeue

[mergify]

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

[mergify]

Merge Queue Status

✅ The pull request has been merged

This pull request spent 33 minutes 12 seconds in the queue, including 31 minutes 46 seconds waiting for CI.
The checks were run on draft #8486.

Required conditions to merge

• check-success=local-testnet-success
• check-success=test-suite-success

[MoeMahhouk]

Thank you for reviewing and merging the PR.
I just noticed that it successfully verified a reproducible build on unstable but it failed due to docker hub token permission issues.
Is the passed token has read-only permissions or did it expire and needs renewal? @michaelsproul

[antondlr]

Thank you for reviewing and merging the PR. I just noticed that it successfully verified a reproducible build on unstable but it failed due to docker hub token permission issues. Is the passed token has read-only permissions or did it expire and needs renewal? @michaelsproul

the namespace repository needs to be initialized first with a manual push, @realbigsean should have the correct permissions to do this

[MoeMahhouk]

Thank you for reviewing and merging the PR. I just noticed that it successfully verified a reproducible build on unstable but it failed due to docker hub token permission issues. Is the passed token has read-only permissions or did it expire and needs renewal? @michaelsproul

the namespace repository needs to be initialized first with a manual push, @realbigsean should have the correct permissions to do this

oh yes, that makes sense. Thank you for confirming and the fast response!

[michaelsproul]

I've just created the sigp/lighthouse-reproducible project and am re-running the build. Lets see if it works.

[michaelsproul]

Didn't work. Are we sure the token isn't a fine-grained token that only works for sigp/lighthouse?

[antondlr]

Bummer. I don't have the necessary permissions to check, unfortunately.
I'll rustle some leaves.

[realbigsean]

updated the deploy key in this repo with perms to deploy to the new docker repo

[antondlr]

it works!
thanks Sean 🫶