Description

A race condition vulnerability has been identified in the conda-forge-webservices component used within the shared build infrastructure. This vulnerability, categorized as a Time-of-Check to Time-of-Use (TOCTOU) issue, can be exploited to introduce unauthorized modifications to build artifacts stored in the cf-staging Anaconda channel. Exploitation may result in the unauthorized publication of malicious artifacts to the production conda-forge channel.

Affected Resource

https://github.com/conda-forge/conda-forge-webservices

The conda-forge-webservices service, deployed on Heroku, is a critical component responsible for validating artifacts in the cf-staging channel before promoting them to the production conda-forge channel. This validation is triggered at the end of CI/CD build processes (e.g., Azure Pipelines) through an HTTP POST request made by the build agent to the /feedstock-outputs/copy endpoint.

The request includes parameters such as a SHA-256 hash of the artifact, its location in the cf-staging channel, and a FEEDSTOCK_TOKEN used for authentication. This token ensures that only authorized feedstock maintainers or core team members can trigger the copy operation. Consequently, the attacker must wait for a legitimate build event to occur.

The request is processed by the OutputsCopyHandler, which performs the following validation steps:

1. Verifies the FEEDSTOCK_TOKEN to confirm that the request is authorized and originates from a legitimate CI/CD build;
2. Determines which outputs (artifacts) are permitted to be copied;
3. Retrieves metadata for the relevant artifact from the cf-staging channel, including the SHA-256 hash;
4. Compares the provided hash against the retrieved hash to verify integrity;
5. Initiates the copy to the production channel using the Anaconda API, provided all validations succeed.

Relevant Code Snippets

Entry point function signature

class OutputsCopyHandler(tornado.web.RequestHandler):
    async def post(self):
        headers = self.request.headers
        feedstock_token = headers.get("FEEDSTOCK_TOKEN", None)
        data = tornado.escape.json_decode(self.request.body)
        feedstock = data.get("feedstock", None)
        outputs = data.get("outputs", None)
        channel = data.get("channel", None)
        git_sha = data.get("git_sha", None)
        hash_type = data.get("hash_type", "md5")
        provider = data.get("provider", None)
		[...]

Function validating hashes

https://github.com/conda-forge/conda-forge-webservices/blob/35e25f61cfb57a70cc97ffa9f7f112efcefc3743/conda_forge_webservices/feedstock_outputs.py#L175C1-L175C47

def _is_valid_output_hash(outputs, hash_type):
		[...]
		ac = get_server_api()
		[...]

        try:
            data = ac.distribution(
                STAGING,
                name,
                version,
                basename=urllib.parse.quote(dist, safe=""),
            )
            valid[dist] = hmac.compare_digest(data[hash_type], hashsum)
            LOGGER.info("    did hash comp: %s", dist)
        except BinstarError:
            LOGGER.info("    did not do hash comp: %s", dist)
            pass
		[...]

Function initiating Anaconda API copy

def copy_feedstock_outputs(outputs, channel, delete=True):
	    [...]
    ac_prod = _get_ac_api_prod()
    ac_staging = _get_ac_api_staging()
	    [...]
    for dist in outputs:
		  	 [...]
                ac_prod.copy(
                    STAGING,
                    name,
                    version,
                    basename=urllib.parse.quote(dist, safe=""),
                    to_owner=PROD,
                    from_label=channel,
                    to_label=channel,
                    update=True,
                )
                copied[dist] = True
                LOGGER.info("    copied: %s", dist)
            except BinstarError as e:
                LOGGER.info("    did not copy: %s (%s)", dist, repr(e))
                pass
			[...]

The core vulnerability results from the absence of atomicity between the hash validation and the artifact copy operation. This gap allows an attacker, with access to the cf-staging token, to overwrite the validated artifact with a malicious version immediately after hash verification, but before the copy action is executed. As the cf-staging channel permits artifact overwrites, such an operation can be carried out using the anaconda upload --force command.

Attack Scenario:

1. The attacker, in possession of the cf-staging token, prepares a malicious package (e.g., package-A-ver1.conda) and has all parameters to upload the package later on to the cf-staging channel using the --force flag.
2. The attacker monitors for a legitimate build and waits for the conda-forge-webservices copy process to be triggered.
3. The web service receives the copy request and performs all required validation checks, including hash comparison.
4. Immediately following the hash comparison, but before the copy action is invoked, the attacker overwrites the artifact with the malicious version.
5. The conda-forge-webservices component proceeds to copy the now-modified artifact to the production conda-forge channel.
6. The malicious artifact is propagated via the Anaconda CDN and may be installed by unsuspecting users.

Despite the brief exploitation window, repeated attempts may yield success. A targeted attack on a widely used package or internal dependency could trigger a cascading supply chain compromise, potentially leading to privilege escalation or widespread artifact poisoning.

Potential Mitigations

Several potential mitigations were discussed including

1. Checking the package sha256 again after the copy and deleting any mismatches.
2. Moving the package to a temporary staging area where it cannot be replaced before initiating the upload.
3. Moving builds of packages from feedstocks to a secured build queue in order to eliminate the multistage upload process.

We've implemented items 1 and 2.