Support to extract pragmas for detect-secrets-server

[francisluz]

🗒️ Description

As a detect-secrets-server user, I would like to extract all pragmas in my codebase where I can validate the use of pragma: allowlist secrets.

💡 Solution

Using the current process that loop through the code and plugin, one more step was added to extract the pragmas. This is an optional feature where on the server side will be triggered through a new flag called --extract-pragmas which will be pushed in its proper PR on detect-secrets-server repo.

⚠️ Caveats

As scan_diff is called only by detect-secrets-server this feature will be available only there.

⌛ Time complexity

The performance impact on the scan if this feature is enabled, will be O(n+1) where extract-pragmas is considered an extra plugin run.

😎 Hope you guys like this feature, we're already using it in our internal CI process.
👍 Feel free to reach me out.

[domanchi]

Whoops! I realize this was never merged.

I'm going to go ahead and port this change to our pre-v1-launch branch, given that:

1. This should have been merged before the effort to rearchitect the tool
2. This is the feature that we needed, but never got around to it.

@francisluz : mind being the reviewer for the change, so we can verify that it works as designed?

[francisluz]

Hey @domanchi,

I don't mind at all 👍

I also need to create another PR to server to include some changes there.

[domanchi]

Actually, now that I think about this more @francisluz, what's the difference between this feature, and:

grep -rE '(allow|white)list secret'

[francisluz]

Actually, now that I think about this more @francisluz, what's the difference between this feature, and:

grep -rE '(allow|white)list secret'

Hey @domanchi ,

Well, I actually started this by using the grep which is not much different, but I had some issues because our Jenkins runs some sensitive data masking on it, then I couldn't generate the output list with the real values.

Then I had to implement it inside of the code where it was out of Jenkins scope.

[domanchi]

Huh. Very interesting. I'm always curious to learn more about how different teams are using this tool in their various workflows.

Be on the look out for a PR for this early next week.

[francisluz]

Awesome, definitely make sense as you guys have the big picture of the tool to understand more about its use.

Btw, just open the detect-secrets-server PR #68 for this feature, I was delaying it because there's a bunch of unit tests that is not passing and seems that they were there before.

Thanks @domanchi,

[domanchi]

Apparently, it was easier to implement than I expected. @francisluz , be sure to check out the linked PR and verify it on your end manually, to see if I missed anything in the port. You can do so via:

$ pip install git+https://github.com/Yelp/detect-secrets.git@pre-v1-launch
$ detect-secrets scan --only-allowlisted

If you wanted to do it specifically with detect-secrets-server, you may need to do a bunch of tinkering around. I'd recommend just using detect-secrets as a package, and testing that way too. e.g.

from detect_secrets.core.scan import scan_for_allowlisted_secrets_in_diff

with open('filename') as f:
    diff = f.read()

for secret in scan_for_allowlisted_secrets_in_diff(diff):
    print(secret)

[domanchi]

Closing as done, since I've already ported it in the linked review.

[francisluz]

@domanchi Man, I would like to get some thoughts about how are you guys handling simples passwords, Like This list here.

I know that DS doesn't handle this, do you guys use a custom plugin? do you handle this at all?

Thanks a mil