Reproducible builds

[AmirGamilDev]

Hello.

Excellent initiative.

Is there a way to verify that the build on the play store is produced from the code in this repo (a la Signal private messenger)? The verification in the readme suggests that the certificate used for signing is the same but is this the same thing as the build being the same? Perhaps I'm missing something?

[alexbakker]

The section in the README you're referring to explains how you can verify that Aegis APK's were signed by us. Reproducible builds are something completely different and we don't support that currently.

[AmirGamilDev]

That's what I had understood. Is there a plan to include it on the roadmap? This would greatly increase the trust in the product.

[alexbakker]

Not currently. I'd first like to see a more detailed proposal and perhaps a proof of concept for this. Maintaining reproducible builds can be painful and it'd be good to have a general impression of what the impact on Aegis' build process would be.

[AmirGamilDev]

An excellent example is here:
https://github.com/signalapp/Signal-Android/tree/main/reproducible-builds

From the looks of things, I think it could be done with minimal impact to the build process once the work is carried out.

I believe this is truly important to be able to implement a TNO (Trust No One) solution.

[IzzySoft]

At IzzyOnDroid we support Reproducible Builds (see: Reproducible Builds, special client support and more at IzzyOnDroid). We're just trying to establish Aegis there as well (see the linked issue). @alexbakker asked me to outline how we do that and what tools we use. Luckily, we've already set up that blog post I've just linked, explaining it – and meanwhile also a wiki page on how you can set up your own verification server with the very same tools.

As for Aegis, it's almost reproducible, only some PNG files differ (again, please see the linked issue for details and hints on how to solve this). As we didn't yet succeed, the recipe is not yet online, so let me post it here in case one of you wants to try. These are the contents of recipes/com.beemdevelopment.aegis.yml:

---
repository: https://github.com/beemdevelopment/Aegis.git
updates: releases
versions:
  - tag: v3.2
    apks:
      - apk_pattern: aegis-v.*\.apk
        apk_url: https://github.com/beemdevelopment/Aegis/releases/download/v3.2/aegis-v3.2.apk
        build:
          - chmod +x gradlew
          - ./gradlew assembleRelease
          - mv app/build/outputs/apk/release/*unsigned.apk /outputs/unsigned.apk
        build_cpus:
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_timeout:
        build_user: build
        provisioning:
          android_home: /opt/sdk
          build_tools:
          cmake:
          cmdline_tools:
            version: '12.0'
            url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
            sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
          extra_packages: []
          image: debian:bookworm-slim
          jdk: openjdk-17-jdk-headless
          ndk:
          platform:
          platform_tools:
          tools:
          verify_gradle_wrapper: true

And the diff in the linked issue is the only difference between the APK provided here and the one we've built using this recipe.

[IzzySoft]

@alexbakker any plans on this yet? Are the details I provided on your request in #1528 helping you with this?

[IzzySoft]

@alexbakker the issue is still the very same with v3.3.4:

  -rw----     0.0 fat      216 b-      216 stor 1981-01-01 01:01:02 4f558efa res/K_.9.png
- -rw----     0.0 fat      640 b-      640 stor 1981-01-01 01:01:02 9de3b6c4 res/Kd.png
+ -rw----     0.0 fat      641 b-      641 stor 1981-01-01 01:01:02 b188a002 res/Kd.png
  -rw----     0.0 fat     2360 b-      660 defN 1981-01-01 01:01:02 49b8c223 res/Ke.xml
  -rw----     0.0 fat     2800 b-     1005 defN 1981-01-01 01:01:02 5802d65c res/L-.xml
  -rw----     0.0 fat      492 b-      280 defN 1981-01-01 01:01:02 96478447 res/LJ.xml
@@ -514,7 +514,7 @@
  -rw----     0.0 fat      684 b-      359 defN 1981-01-01 01:01:02 c57ac8cc res/VN.xml
  -rw----     0.0 fat      640 b-      352 defN 1981-01-01 01:01:02 7d16fbad res/VT.xml
  -rw----     0.0 fat     2348 b-      847 defN 1981-01-01 01:01:02 7ee08d4e res/VT1.xml
- -rw----     0.0 fat      987 b-      987 stor 1981-01-01 01:01:02 cc396cc9 res/VV.png
+ -rw----     0.0 fat      995 b-      995 stor 1981-01-01 01:01:02 e2d002c8 res/VV.png
  -rw----     0.0 fat      217 b-      217 stor 1981-01-01 01:01:02 ecd8775b res/W4.9.png

Did you consider trying the settings I recommended in November last year?

android {
    // disable PNG crunching:
    aaptOptions {
        cruncherEnabled = false
    }
    // disable generating PNGs from vector drawables:
    defaultConfig {
        vectorDrawables.generatedDensities = []
    }
}

Would be an easy try. Unless of course you're not interested in RBs – in which case please tell us so we drop the topic and stop nagging you.

Maintaining reproducible builds can be painful

It sometimes indeed is. But in case of Aegis, the only "diff" for multiple builds now is always the very same (those 2 PNG files – and even the checksums are still the same; my guess is it's the ic_launcher.xml from the res/mipmap-anydpi* directories, so the vectorDrawables.generatedDensities = [] should fix that) – and I'd expect with this fixed, there won't be much pain left 😉

[ave9858]

@IzzySoft I actually managed to reproduce the VV.png and Kd.png from the official apk release, though i did have some other small diffs due to building on Windows (wrong newline on META-INF/services/kotlinx.coroutines.android.AndroidExceptionPreHandler and META-INF/services/kotlinx.coroutines.android.AndroidDispatcherFactory). I might try and see if I can reproduce them again on Linux

[IzzySoft]

Cool! Those few files are no big issue and can easily be dealt with on our builders. If you give me an APK and name its commit, I can cross-check; until then, just going by your comment:

        build:
          - chmod +x gradlew
          - ./gradlew assembleRelease
          - find . -name '*.apk'
          - git clone -b v0.3.0 https://github.com/obfusk/reproducible-apk-tools.git
          - OUT=app/build/outputs/apk/release/app-release-unsigned.apk
          - reproducible-apk-tools/inplace-fix.py --internal --zipalign fix-newlines "$OUT" 'META-INF/services/*'
          - mv "$OUT" /outputs/unsigned.apk

should do the trick. Oh, and don't forget to add python3 as extra_package further down the YAML:

          extra_packages:
            - python3

(needed by reproducible-apk-tools/inplace-fix.py 😉)

[ave9858]

I was testing the v3.3.4 apk (i just checked out that tag on git), just using Android Studio on Windows which ofc wouldn't be suitable for your builders. I can upload the apk output I have or I can try again on Linux w/ CLI tools and upload that, I'm not sure exactly what information you'd like. I attached the diffoscope output below of what I currently have.

diff.txt

There's a lot related to signatures and keys but I think that would be solved if I just copied the signature from official or compared after stripping signatures

[IzzySoft]

I was testing the v3.3.4 apk (i just checked out that tag on git), just using Android Studio on Windows which ofc wouldn't be suitable for your builders.

Strange. 3.3.4 was the last release I've tested here: built from the tag, and matched against the APK you've attached to the tag. You wrote you "managed to reproduce the VV.png and Kd.png from the official apk release" – so did you change some code for that, or how did you achieve this? These two files were the only differences shown on my end.

Btw, for the diffing (and other tasks on debugging failed RBs), you might find rbhelper useful – for the above diff, that would be the rbtest command. If you want to check those, please take a look at the open PRs there: PR#1 provides an overhaul of the functions that makes working with them easier (waiting for review/test so we can merge it). Installation of our framework also became much easier ("a breeze", as I've been told by someone who tried), with the new rbuilder_setup scripts. Full output of the rbtest command on v3.3.4, to give you an example:

Error: APK Signing Block offset < central directory offset.

Not reproducible, going for the zipInfo diff of the APKs.
If the following shows the signature files as the only difference, try the 'zipalignment' helper to look for alignment, padding etc.

(perms: file permissions; ver: zipinfo create-version; tt: type (text/binary) and whether it's encrypted ('t-' = text, not encrypted; bx = binary, encrypted))

  perms       ver FSys Size    tt CompSize Algo Timestamp           CRC      FileName
  ----------+----+----+-------+--+--------+----+-------------------+--------+-----------
--- /dev/fd/63  2025-03-25 23:05:27.643840849 +0100
+++ /dev/fd/62  2025-03-25 23:05:27.644840845 +0100
@@ -1,5 +1,5 @@
- Archive:  ./b17c51815f7569c8e1c9726b153b37421c005f69cb55bd289fa27c22956238b1-com.beemdevelopment.aegis-v3.3.4-upstream.apk
- Zip file size: 6031546 bytes, number of entries: 1081
+ Archive:  ./b4a207ee34a9047633d007ba4dacdb75b4ff1fe60039f717c7fb50c5975b53be-com.beemdevelopment.aegis-v3.3.4-unsigned.apk
+ Zip file size: 5930067 bytes, number of entries: 1078
  -rw-r--r--  0.0 unx       56 b-       52 defN 1981-01-01 01:01:02 32137644 META-INF/com/android/build/gradle/app-metadata.properties
  -rw-r--r--  0.0 unx      120 b-      116 defN 1981-01-01 01:01:02 6ec53c0f META-INF/version-control-info.textproto
  -rw-r--r--  0.0 unx     2197 b-     2197 stor 1981-01-01 01:01:02 19d7d786 assets/dexopt/baseline.prof
@@ -386,7 +386,7 @@
  -rw----     0.0 fat      415 b-      415 stor 1981-01-01 01:01:02 ec058f9b res/KM.png
  -rw----     0.0 fat      592 b-      315 defN 1981-01-01 01:01:02 ebb93e66 res/KT.xml
  -rw----     0.0 fat      216 b-      216 stor 1981-01-01 01:01:02 4f558efa res/K_.9.png
- -rw----     0.0 fat      640 b-      640 stor 1981-01-01 01:01:02 9de3b6c4 res/Kd.png
+ -rw----     0.0 fat      641 b-      641 stor 1981-01-01 01:01:02 b188a002 res/Kd.png
  -rw----     0.0 fat     2360 b-      660 defN 1981-01-01 01:01:02 49b8c223 res/Ke.xml
  -rw----     0.0 fat     2800 b-     1005 defN 1981-01-01 01:01:02 5802d65c res/L-.xml
  -rw----     0.0 fat      492 b-      280 defN 1981-01-01 01:01:02 96478447 res/LJ.xml
@@ -514,7 +514,7 @@
  -rw----     0.0 fat      684 b-      359 defN 1981-01-01 01:01:02 c57ac8cc res/VN.xml
  -rw----     0.0 fat      640 b-      352 defN 1981-01-01 01:01:02 7d16fbad res/VT.xml
  -rw----     0.0 fat     2348 b-      847 defN 1981-01-01 01:01:02 7ee08d4e res/VT1.xml
- -rw----     0.0 fat      987 b-      987 stor 1981-01-01 01:01:02 cc396cc9 res/VV.png
+ -rw----     0.0 fat      995 b-      995 stor 1981-01-01 01:01:02 e2d002c8 res/VV.png
  -rw----     0.0 fat      217 b-      217 stor 1981-01-01 01:01:02 ecd8775b res/W4.9.png
  -rw----     0.0 fat     1200 b-      544 defN 1981-01-01 01:01:02 006f020a res/WH.xml
  -rw----     0.0 fat      712 b-      374 defN 1981-01-01 01:01:02 e321b65c res/WK.xml
@@ -1078,7 +1078,4 @@
  -rw----     0.0 fat      464 b-      269 defN 1981-01-01 01:01:02 3a79ca15 res/zq.xml
  -rw----     0.0 fat     3080 b-     3080 stor 1981-01-01 01:01:02 dac728d2 res/zr.png
  -rw----     0.0 fat  2595796 b-  2595796 stor 1981-01-01 01:01:02 638e1e8d resources.arsc
- -rw-r--r--  0.0 unx   100551 b-    45163 defN 1981-01-01 01:01:02 a169d85e META-INF/CERT.SF
- -rw-r--r--  0.0 unx     1127 b-     1049 defN 1981-01-01 01:01:02 83a3ed67 META-INF/CERT.RSA
- -rw-r--r--  0.0 unx   100477 b-    42653 defN 1981-01-01 01:01:02 83a16cff META-INF/MANIFEST.MF
- 1081 files, 9958612 bytes uncompressed, 5891157 bytes compressed:  40.8%
+ 1078 files, 9756466 bytes uncompressed, 5802301 bytes compressed:  40.5%
also see: [HOWTO: diff & fix APKs for Reproducible Builds](https://gist.github.com/obfusk/f0460dcdb21396ba3d2cc9db56bb40ae)

(now you see the "table header", you probably get a better idea to read the excerpt in a previous comment of mine 😉)

There's a lot related to signatures and keys

Which can be ignored here: the diff compares against our unsigned APK. Certificates are still shown, also because sometimes devs set up their builds in a way they end up being signed with a debug key, and have that part not in their build.gradle (where it would be obvious), but somewhere deep inside the source tree 🙈

Btw, to get a "binary identical APK", in a last step we use apksigcopier to copy the signature from the upstream APK to the one we've built. Which works perfectly on a reproducible build – but of course fails on failed RBs.

just using Android Studio on Windows which ofc wouldn't be suitable for your builders.

Most of those "Windows extras" are just line endings, which we can fix easily by "adding them to our APKs". It just gets a bit messy when developers build alternating, one release on Windows and the next on Linux – as then we have to remove/add/remove our adjustments… The only Windows things we cannot fix on our end are embedded paths (Flutter loves that, for example).

[ave9858]

Strange. 3.3.4 was the last release I've tested here: built from the tag, and matched against the APK you've attached to the tag. You wrote you "managed to reproduce the VV.png and Kd.png from the official apk release" – so did you change some code for that, or how did you achieve this? These two files were the only differences shown on my end.

No code changes, I just built with the latest Android Studio release.

Also, to be clear I'm not associated with this project at all, I'm just a user interested in reproducible and I wanted to try to get to the bottom of those image diffs. Since whatever toolchain I used managed to reproduce those pngs maybe whatever you're using to rebuild is slightly off, I'll try more variations later if I have time.