Add verified reproducible builds, maybe as SLSA 5

[david-a-wheeler]

The SLSA proposal omits reproducible builds, and I think that's a mistake.

Currently the proposal assumes that there are magical build systems that cannot be subverted. No such systems exist. You could use build systems that are protected/hardened, and that obviously reduces risk, but subverting a build environment is a known attack that just happened to SolarWinds' Orion & is one of the main reasons people are calling for improvements in supply chain integrity. Ignoring a well-known attack vector, especially one that is specifically the kind of attack people are trying to prevent, is a mistake.

Implementing reproducible builds is usually not that hard. Typical challenges are enforcing order where things were in arbitrary order (e.g., by sorting) & by forcing timestamps to fixed values. It typically requires changes because historically no one tried to make builds reproducible, but it's doable.

I recommend that verified reproducible builds. It's fine if they're added as a new SLSA level (say, "level 4").

[david-a-wheeler]

It's true that many organizations won't want to make their closed source public, but that's easily solved. It would be relatively easy to sign a contract to have some external party verify a reproducible build and provide many mechanisms to keep it from going public.

Even Oracle, which is very reticent about releasing source code for some of its products, has released their closed source code to an external lab to get a higher-level Common Criteria evaluation. A reproducible build is in many ways easier, because very few people in the rebuilder need access to the source code.

[Hixie]

In #34 I argue that verified reproducible builds isn't really a solution to the Solarwinds attack either. It arguably increases the cost of such an attack — you have to hit two systems instead of one — but is that enough?

There's an argument to be made that if you have reproducible builds it's actually easier to attack your build system. With one CI/CD system, there's one target to attack, and all the efforts can go into securing it. With two CI/CD systems plus some logic to verify they are doing the same thing, there's three systems to attack, and you need to break two of them (either both build systems, or one of the build systems plus the verifier), and the effort for securing these systems is now split in three, so isn't it possible they're net easier to attack than the one system?

[david-a-wheeler]

so isn't it possible they're net easier to attack than the one system?

NO.

First, in many setups the "separate verifier" is likely (in part) to be the recipient, who by definition must receive something to be a recipient. This isn't "voting"; if there's any difference then by definition there's a problem to be investigated.

You also CANNOT eliminate many vulnerabilities at any cost when you put all your trust in one basket. For example, a physical attack on a particular location can only be fully ruled out by having multiple locations, which necessitates reproducibility.

Even economically trying to put total trust in one place doesn't make much sense. The costs of "putting all your efforts into securing a single build" go up substantially at the high end of assurance. Getting that last little bit is extraordinarily expensive. Even if you're going to pay those prices anyway, the costs of reproducibility are in the noise, and they provide stronger level of assurance than trusting that a system is perfectly secure in the past, present, and all future time, which is what you're assuming when you don't support reproducibility.

There's no reason it has to be only 2 builds. You could have many more. If you want strong confidence, rebuild in a dozen places. Rebuilding is so cheap that it's typically not worth metering.

More importantly, reproducibility can be rechecked after release. A key advantage in reproducible builds is that you can easily re-verify after-the-fact (if you have access to the source code). So if you have doubts, you can rebuild, and rebuild, and rebuild. If you put all your eggs in one basket, and the attacker breaks in even for an instant, it's game over, typically it won't be detected, and determination of it is both costly & time-consuming.

The security world has generally given up on protection as the sole security mechanism. Attackers are smart, and they sometimes win. Reproducible builds provides a detection mechanism, so that even if an attacker gets in, they're still detected. That's a far stronger measure than hoping and praying for perfection. Protect, yes, but don't depend solely on protection.

[Hixie]

Regarding the verifier being the recipient, that's an interesting approach. I was imagining a world where after you have many builds, you then check them before uploading the final build to storage for users to download. But I guess all the systems could upload their (signed) binaries to storage, along with some cryptographic digest of the output they created, and when downloading these binaries the end user could first verify that the digests all match, then pick one at random and verify that it matched the digest. Then so long as nobody has adulterated the verification code, we could be confident that mismatches would get caught quickly. (Adulterating the verification code would just be like adulterating any code, so other SLSA practices take care of that.)

I'm not sure I really buy that building is so cheap, however. As an open source project, each new CI/CD system we depend on introduces significant new complexity: every time we make a change to our build system we now have to update each CI/CD system we use, each new dependency is one more system that can arbitrarily change its APIs in ways that break us, each new system is a new set of complexities that the whole team has to learn and deal with. Plus build systems are inherently flaky so each new dependency increases the odd that a specific commit ends up failing for bogus reasons (e.g. Travis happened to have bad connectivity today so failed to pull all the files for the build, or Cirrus ran out of Mac hosts so our build task didn't get scheduled and timed out, or...).