Enforce standard disclosure policy strongly, don't allow misuse of experimental bit

[evverx]

In #1757 (comment) we started to discuss how for some projects the standard disclosure policy has been optional all this time. It was partially corrected by flipping the experimental bit (without going through the disclosure retrospectively as far as I can tell) but the maintainers of OSS-Fuzz seem to be reluctant to always enforce the standard disclosure policy. One loophole was suggested in #1757 (comment)

In theory, it means that it's possible for a person integrating a project to mark it "experimental" to hide it from the standard disclosure policy and be a liaison between OSS-Fuzz and the developers who can fix bugs as they see fit. Sounds good to me too.

Basically as long as there are loopholes like that the standard disclosure policy is really optional.

[evverx]

Since the comment about "experimental: true" I based my point on in #1757 (comment) was silently removed I'll copy it from the mailbox:

"experimental: true" is used when:

• we just want to see the bug and see if there are false positives. usually done with msan.
• we dont have a maintainer contact and dont want to start bug filing

[Dor1s]

I'd propose / vote for the following solution:

• External contributors (i.e. not Googlers) cannot add experimental flag, and we will not accept projects without having a commitment from an upstream maintainer. This will slow down the onboarding process, but should result in fewer abandoned / incomplete integrations.
• Googlers can add projects with experimental flag, mainly for two reasons:

1. The integration might be literally experimental (new fuzzing engine, new fuzzing technique, whatever else "experimental").
2. We formally won't be disclosing bugs to the general public, until we notify a maintainer and give them 90 days. I agree that the fact that fuzz targets are public makes this point less valid. However, thinking this way, one could say "these bugs exist in open source software no matter whether you find them or not, just forward the bug reports to the full disclosure", which doesn't sound wise either.

[inferno-chromium]

I agree with Max, but for point 2). I propose stricter rules on vanilla experimental bit. For new projects, we would be monitoring this bit and not allowing things beyond 2 weeks or something. In many cases, we should just add maintainer contact from page - e.g. #2606. If developer does not want to maintain, we can look at removing the project altogether. Keeping this experimental bit forever is bad, even incorrect/forgotten, e.g. de5c2b5.

[evverx]

I'll try to reply to the comments concerning new projects later.

In the meantime, I'm more interested in what's going to happen to jsonnet, for example, which I suspect is Fudge-generated. I've just finished reading https://ai.google/research/pubs/pub48314. According to the paper

In November 2018, the Google Security Team announced an effort to “admit as many OSS projects as possible [into OSS-Fuzz]
and ensure that they are continuously fuzzed” [28]. As a part of this
effort, we have integrated more than 10 new open source projects
with OSS-Fuzz using Fudge targets. These targets are fully open
sourced and the bugs that they find are eventually publicly disclosed [10].

[inferno-chromium]

@evverx - jsonnet experimental bit is removed and developer is added - https://github.com/google/oss-fuzz/blob/master/projects/jsonnet/project.yaml

[evverx]

@inferno-chromium I'm aware of that but that doesn't mean all the bugs that were fixed silently there were disclosed.

[inferno-chromium]

@evverx - the project is brought back again (cl you cited was reverted and project is now running again). so, all current unfixed bugs will be disclosed. for fixed bugs, i dont have cycles to create them manually in monorail. you can file a bug in jsonnet repo to make sure it gets in their release notes/gets cve/etc.

[evverx]

you can file a bug in jsonnet repo to make sure it gets in their release notes/gets cve/etc.

Sure but I thought the point of monorail was to make sure bugs OSS-Fuzz finds are filed somewhere and disclosed eventually. If you need someone else to keep track of whether projects disclose their bugs you don't really enforce the standard disclosure policy.

[inferno-chromium]

you can file a bug in jsonnet repo to make sure it gets in their release notes/gets cve/etc.

Sure but I thought the point of monorail was to make sure bugs OSS-Fuzz finds are filed somewhere and disclosed eventually. If you need someone else to keep track of whether projects disclose their bugs you don't really enforce the standard disclosure policy.

Ok, filed manually for fixed security ones - https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=label%3AProj-jsonnet. All unfixed bugs are filed as well, but open accessible to developer.

[evverx]

I think it would be better to file all of them and let the community decide what is relevant to security. Generally the "security" label is just kind of guess and sometimes juicy bugs manage to slip through without being tagged. What about the other projects where the experimental bit was/is used?

More generally, who is going to do it manually if you decide to allow googlers to set the experimental bit?

[inferno-chromium]

I think it would be better to file all of them and let the community decide what is relevant to security. Generally the "security" label is just kind of guess and sometimes juicy bugs manage to slip through without being tagged. What about the other projects where the experimental bit was/is used?

Ok filed those for git and jsonnet. Other projects didn't have stuff fixed, so new bugs are opened, but not public.

More generally, who is going to do it manually if you decide to allow googlers to set the experimental bit?

We are still discussing this.

[evverx]

@Dor1s

Googlers can add projects with experimental flag, mainly for two reasons

I think googlers are capable of setting up an internal instance of ClusterFuzz. This was they can keep experimenting internally with whatever they need without rolling out fuzz targets for unprepared projects publicly (apparently at scale with Fudge). Once projects are ready they can be moved to OSS-Fuzz where the same rules are applied to everyone (theoretically at least).

[oliverchang]

As @inferno-chromium mentioned already, we're human and try to enforce this policy as best as we can. It's not a loophole, as we review every PR.

I think googlers are capable of setting up an internal instance of ClusterFuzz. This was they can keep experimenting internally with whatever they need without rolling out fuzz targets for unprepared projects publicly (apparently at scale with Fudge).

This is exactly what the "experimental" bit is for. We'll certainly be more mindful of how long we keep projects as experimental from now on, but it does not make sense to spend more time and effort on this when it's a non-issue for the vast majority of projects. You can just treat "experimental: True" as if they didn't exist on OSS-Fuzz and it's exactly the same.

I think we should just explicitly enforce that only Googlers can add this, as @Dor1s proposed, and think about some way to track how long projects can stay as experimental for, so we remember to follow up.

[evverx]

we're human and try to enforce this policy as best as we can

That's why I think people should be generally excluded from the process that can (and should) be fully automated.

It's not a loophole, as we review every PR.

What criteria do you use reviewing PRs? How come libsass was integrated in December 2018 and the maintainers were only just notified in #2605?

You can just treat "experimental: True" as if they didn't exist on OSS-Fuzz and it's exactly the same.

While we're discussing monorail and the standard disclosure policy here, I think we all here realize that they generally don't matter and sometimes rolling out fuzz targets publicly is indistinguishable from dropping zero days to who knows whom which is kind of conflicting with "OSS-Fuzz aims to make common open source software more secure".