What to do about clobbering of packages in the prefix

[CJ-Wright]

What should we do when packages ship packages beyond themselves in site-packages. For instance if I have a package that ships itself and requests in site-packages that means that conda may no longer control what requests is installed, which could be problematic.

@conda-forge/core

edit: this effects the entire prefix and we should address all of that at once

[beckermr]

Here are two ideas for this.

Only Critical packages

1. Determine a list of packages that considered critical infrastructure for security or other reasons.
2. build up the mapping from files in the prefix to feedstocks
3. add checks of this mapping somehow

All Packages

1. Use libcfgraph to build up a mapping of possible files in the PREFIX to feedstocks
2. have CI jobs compute any conflicts
3. have CI jobs use their feedstock token to get the heroku server to post a lint comment on the PR

We could do both.

[beckermr]

On the technical side for the second thing, we store the mappings in a github repo and then make http requests to github to pull down json blobs with the data.

For the security-based one, we can use similar mechanisms but we need to do all of the unpacking and checking ourselves. The heroku server cannot support this load, but we can use cf-staging as a place to store validated packages before some other server pulls them to do the check.

[wolfv]

We could also make it easier for people in the test section of the recipe. Currently we often test for files using the commands test section which is really cumbersome and hard to get right. I am thinking about adding some macros to boa to check for the existence of a shared library (cross-platform), a python module in site packages, a cmake file etc.

If we design these macros nicely enough we could then warn about superfluous files ...

[isuruf]

warning is not enough. it's a security issue as well.

Also, adding stuff to boa is not going to be enough unless you would contribute those to conda-build.

[wolfv]

Sure, these could be good things to upstream. I hope to add package testing to boa next week

[beckermr]

We actually need this check in quetz to reject uploads of things that clobber. Anyone could simply remove a check in boa when building on a CI service.

[CJ-Wright]

I think that removal depends on how it is implemented, if it is a check in the recipe then yes that is removable. If it is a check applied via the boa/conda-build source code that would be more difficult to skirt.

[isuruf]

How will conda-build know that a package is clobbering another?

[CJ-Wright]

We can provide it with an API which returns if a file clobbers something already on conda-forge. I presume it has all the files in the prefix handy

[beckermr]

If it is a check applied via the boa/conda-build source code that would be more difficult to skirt.

more difficult is not good enough here if we are calling this a security issue.

[CJ-Wright]

You would have to edit the source code of conda-build/boa itself. I'm not certain if there are schemes to do that in a recipe but we could (and most likely should) have the build system hash itself when performing sensitive tasks. That way we can check if the build system was modified in some way during the build.

[wolfv]

Ok there are two aspects to this: one is securing against malicious packages and the other is to make it easier for packagers to not unintentionally clobber. But yeah, I would be happy to help with development wrt to quetz as well! We're working really hard on it right now and with the plugin system in place we could totally think about rejecting uploads like this, and indexing all files in a database.

[beckermr]

We're working really hard on it right now and with the plugin system in place we could totally think about rejecting uploads like this, and indexing all files in a database.

Yup. This coupled with package signing and a way for us to administer the outputs clobbering checks we already do would help us centralize our infrastructure and possibly replace cf-staging with something less hacky and more secure.