This code was used to triage and audit Nixpkgs activity that could have posed a risk of compromise due to the leaked GitHub token of a committer described in GHSA-67f2-674w-6g63.
The leaked token (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) was intended for
use with the Renovate automatic update bot as part of
@Scrumplex’s personal infrastructure, and leaked
on Codeberg through a
temporary unencrypted Kubernetes Secret file
over a year before write access to Nixpkgs was granted. These files contain
Base64‐encoded data that looks similar to the corresponding encrypted Sealed
Secret files, so the compromise would not necessarily have been immediately
obvious even if the file had been noticed.
Thankfully, the partially‐automated, partially‐manual audit comprehensively establishes that there was no compromise of the Nixpkgs source code as a result of this inadvertently‐leaked credential.
We thank @Scrumplex for his cooperation throughout the incident response. He has been contributing to Nixpkgs for years and gained sufficient trust from the community to be granted commit access in the first place. We believe this was a legitimate and understandable mistake, and have no reason to suspect there was any ill intent or unusual carelessness involved.
We also thank @bzzimmy for finding the leaked token and bringing it to our attention.
The code is written in TypeScript, using Deno (deno in
Nixpkgs) and GitHub’s official Octokit JavaScript API client library. You can
reproduce the audit with:
$ deno task audit
This runs offline, using the data in facts.json that was fetched from the
GitHub API ahead of time. If you want to recreate that data yourself, you can
first run:
$ deno task facts
This will take roughly an hour, can sometimes fail due to GitHub rate limits or
backend issues, and is likely to require an API token with a high API request
cap. If you are a member of the NixOS GitHub organization, then you can use the
GitHub CLI (gh in Nixpkgs) to obtain an OAuth token
that
inherits the GitHub Enterprise Cloud rate limit
with gh auth login; the program will automatically use it if present. (As this
token is highly sensitive, you may want to gh auth logout afterwards!)
Note that the data is expected to change over time: when the Nixpkgs fork corresponding to a pull request is deleted, we lose the ability to query it for push activity, which could cause the auditing script to fail when the data is refreshed. Additionally, due to GitHub API limitations, the fetching code uses a conservative, coarse‐grained criterion to pick up pull requests based on their time of last update; pull requests that were opened before the end of the vulnerability period but get updated after the time of publication will be added to any future fetches.
As a result, the audit will likely not be fully reproducible with fresh data in
future, and deno task audit may fail after the fetch. In this case, you will
have to trust the data already in this repository, or else manually check the
differences in the JSON file and record audits for anything that can no longer
be automatically checked in axioms.ts.
GitHub’s API logs only go back 180 days, and we were unable to obtain fully comprehensive data for activity in forks of Nixpkgs even for that period. Therefore, we needed to comprehensively enumerate potential sources of risk, automatically audit as much of it as possible by checking the activity against other sources of trust, and manually audit whatever was left.
The code audits the following things that could have affected, or could affect in future, the source code of Nixpkgs and the other NixOS organization repositories that Nixpkgs committers have access to, as a result of the compromise:
-
Pushes directly to affected repository branches by the affected user during the period between commit access being granted and the cut‐off date of the GitHub audit logs for the NixOS organization.
These pushes could have been done by an attacker with the leaked token. In practice, there were no pushes to our main development branches, and only a small number of pushes to non‐fork pull request branches that were manually audited.
-
Pull requests that were merged by the affected user during the period between commit access being granted and the cut‐off date of the GitHub audit logs for the NixOS organization.
These merges could have been done by an attacker with the leaked token.
-
Pull requests that were merged by the merge bot after commit access was granted, and opened before the revocation of the compromised token.
The merge bot allows arbitrary package maintainers – who do not necessarily have the degree of trust committers do – to merge pull requests to their packages that have another source of trust: being authored by trusted update or backport automation, or being authored or approved by a committer.
An attacker could have used the leaked token to create a pull request with the affected committer’s account as author, and then get it merged by a non‐privileged package maintainer (potentially a sockpuppet).
As committers with write access to a repository can push to pull request branches in forks, an attacker could also have pushed to the branch of an existing pull request that is eligible for the merge bot. Unfortunately, these pushes do not show up in the NixOS organization audit logs. We were able to obtain logs from GitHub showing that the token was not used for Git pushes to any forks of NixOS organization repositories in the 180‐day period up to the end of the logs, but sadly this does not include “web commits” done through the API.
Therefore, we audit pull requests merged by the merge bot for the entire period in which they could have potentially been affected by the compromise.
-
Pull requests that have not been merged and were updated before the revocation of the compromised token.
As above, these could have been pushed to by an attacker. This would not be a problem by itself, as pull requests would ordinarily only be merged after review by a trusted committer. However, some of them could be eligible for a merge by the merge bot, and the trust criteria used for the merge bot could be expanded in future.
To guard against any risk of this, we audit open and closed pull requests for the entire risk period. This constitutes the large majority of the pull requests in
facts.json.
The strategies used for auditing are:
-
Establishing lack of risk directly. If there were no pushes by the affected user to a pull request branch during the risk period, then we can be confident that any trust in its authorship was not compromised. If the fork repository of an unmerged pull request fork has been deleted entirely, then although its push activity cannot be audited, it also can never be merged.
-
Tying trust back to other sources. If a pull request was authored, merged, or approved at the merged commit by another committer, then there is no additional risk from the compromised token. If the affected committer opened a pull request or pushed to another pull request’s branch during the compromise period, but those commits are signed by the affected committer’s (presumed non‐compromised) OpenPGP keys, then there is no additional risk from the compromised token.
-
Manual audit of any activity that can’t be established as safe by the previous checks, as recorded in
axioms.ts.
You can read axioms.ts, facts.ts, facts.graphql, and audit.ts to see the
full details of the auditing strategy and the record of the manual audits.
The vast majority of activities and pull requests were able to be automatically audited. There were 194 pull requests that required manual audit, split between myself (@emilazy), @infinisil, and @LeSuisse; we found no evidence of anything suspicious or any security risk, nor did @mdaniels5757 after re‐reviewing all of them. The audit methodology and code were reviewed by @LeSuisse and @mdaniels5757.
Although we have concluded that this incident ultimately had no impact on Nixpkgs, it points to several systemic risks and areas where our GitHub security posture could use improvement. Efforts to address these are being tracked at NixOS/org#246.
I have endeavoured to cover all the edge cases I can think of while writing this code, but it is possible that I missed something. The results of the audit depend on the following assumptions:
-
Nothing other than merges of pull requests and pushes to pull request branches poses meaningful risk. We cannot rule out that the compromised token was used to edit a comment in Nixpkgs more than half a year ago, but the practical risk from this seems very minimal.
-
I have correctly identified the conditions for such activity to pose a risk, and the conditions for which we can automatically gain assurance in that activity.
-
The auditing code implements those conditions sufficiently correctly that any mistakes are not load‐bearing.
-
The GitHub API returns accurate results for the queries we run, and the GitHub audit logs we were able to obtain do not omit any relevant information.
-
There were no tricky race conditions between merge queue entry and the state of pull requests that were actually merged. It is not clear if these are possible or how the GitHub API would report them, and they would seem to be very difficult to exploit under the conditions of the audit logic. If there is an effective exploit along these lines, it would be unlikely to need a compromised token to work.
-
No other committer and no trusted automation was compromised during the risk period – we trust these by default, and if we find reason to believe that this is not true, that would be a matter for a separate audit.
-
The leaked token was not used to escalate permissions further in some way, such as by creating other access tokens that have not been noticed. The screenshots of the token’s permissions given to us by the compromised committer do not suggest it had access to do such a thing, and there is no evidence in the token access logs we were able to obtain from GitHub that there was any unusual activity of this nature in the 180‐day period of the logs.
-
There was no advanced “Trusting Trust” supply chain attack that invalidates this audit. For example, it is theoretically possible that an attacker could have compromised packages of Nix implementations to thwart audits of forged fixed‐output derivation hashes, compromised Git and browser packages to thwart audits of pull request changes, or even compromised software running on other committers’ systems to take over their accounts. (Perhaps the attacker wrote these very words!)
While this audit cannot definitively rule out the possibility, it would require an unprecedented amount of effort and skill to pull off an attack that could comprehensively fool an unknown set of incident responders using an unknown set of tools across multiple platforms.
It is very unlikely that an attacker of that level of resources and acumen would need to use a leaked GitHub token to carry such an operation out; building up trust and reputation in the community with legitimate contributions and gaining the commit bit would be comparatively trivial, and using a leaked token would only risk drawing unwanted attention if it is discovered.
As the audit logic is public, results can be reproduced without the use of Nixpkgs, and the manually‐audited pull requests are recorded in
axioms.ts, it should be possible for someone unaffected by the Nixpkgs software supply chain to rule this out. -
There are no unknown unknowns that make the conclusions of this audit invalid in any other way.