HTTP Cookie Detector

[alessandro-Doyensec]

This PR adds the logic to detect HTTP cookies in:

• HTTP dumps
• Log dumps

[alessandro-Doyensec]

Hi @hanqiuzh

Thanks for the review.

To reduce the risk of false positives even further, I could introduce (as suggested) the detection of a nearby HTTP-related keyword. However, that would not necessarily improve the confidence: from my research if a piece of code contains valid HTTP Cookie payloads, it also probably contains HTTP-related keywords.

The best option here would be to have a way to understand which file extension is currently being processed, but that is not possible inside Veles.

Feel free to ask/suggest anything.

[hanqiuzh]

@alessandro-Doyensec ,

However, that would not necessarily improve the confidence: from my research if a piece of code contains valid HTTP Cookie payloads, it also probably contains HTTP-related keywords.

Sorry, I'm confused 😂 , doesn't that mean HTTP keywords is a good indicator if valid cookie payloads always co-exist with the HTTP keywords? it's only a problem if HTTP-related keywords can always be found when we have invalid cookie payloads if I'm understanding right.

The best option here would be to have a way to understand which file extension is currently being processed, but that is not possible inside Veles.

There are some extractors to detect only certain files (e.g. https://github.com/google/osv-scalibr/blob/main/extractor/filesystem/secrets/mariadb/mariadb.go you can find FileRequired()). However, I'm not quite sure how that'll increase the confidence, the file can just have no extension?

[alessandro-Doyensec]

Hello @hanqiuzh

Sorry for the miscommunication!

doesn't that mean HTTP keywords is a good indicator if valid cookie payloads always co-exist with the HTTP keywords

What I meant was that a payload respecting the Cookie format is usually a good enough indicator of a true positive on its own. During my initial research, I found that requiring a match for HTTP-related keywords resulted in some false negatives (for example, in HTTP logs) without improving precision by a lot.

To be 100% sure, I developed a "fuzz test" that runs the detector on files returned by a GitHub code search (e.g., https://github.com/search?type=code&q=Cookie%3A+). I noticed that the false positive rate was already low, but not zero. I verified that all of those remaining false positives could be removed by adding the context keyword, which is why I implemented the context search.

However, I'm not quite sure how that'll increase the confidence, the file can just have no extension?

Regarding the file extensions, one good heuristic to reduce false positives would be to exclude .md or source code files, as they are less likely to contain real Cookies. However, I think the detector is in good enough shape as it is.

[hanqiuzh]

Thanks for the research & update. I think that might be the best we can do for now. Left one comment about how to handle pos.

[hanqiuzh]

Detection LGTM. Added some comments about test cases regarding potential license problems I didn't notice before. Please take a look. Thanks