Support exclude regexes, excludewords, and entropy filters for custom detectors

[zricethezav]

Description:

Support exclude regexes, excludewords, and entropy filters for custom detectors. I went with a simple implementation for a few reasons:

1. I didn't want to introduce a breaking change for custom detectors (ideally we could custom detector regex entries would be more than just a map where we could add these filters).
2. Maintainability
3. We plan on supporting "detectors as config" this year which will replace the custom detector system.

ex config:

detectors:
- name: generic-password
  keywords:
  - pass
  - access
  - auth
  - credential
  - cred
  - secret
  - token
  regex:
    secret: |-
      (?i)[\w.-]{0,50}?(?:access|auth|(?-i:[Aa]pi|API)|credential|creds|key|passw(?:or)?d|secret|token)(?:[ \t\w.-]{0,20})[\s'"]{0,3}(?:=|>|:{1,3}=|\|\||:|=>|\?=|,)[\x60'"\s=]{0,5}([\w.=-]{10,150}|[a-z0-9][a-z0-9+/]{11,}={0,3})(?:[\x60'"\s;]|\\[nr]|$)
  entropy: 3
  exclude_regexes_capture:
    - |-
      (?i)(?:ignore)
  exclude_regexes_match:
    - |-
      (?i)(?:access(?:ibility|or)|access[_.-]?id|random[_.-]?access|api[_.-]?(?:id|name|version)|rapid|capital|[a-z0-9-]*?api[a-z0-9-]*?:jar:|author|X-MS-Exchange-Organization-Auth|Authentication-Results|(?:credentials?[_.-]?id|withCredentials)|(?:bucket|foreign|hot|idx|natural|primary|pub(?:lic)?|schema|sequence)[_.-]?key|key[_.-]?(?:alias|board|code|frame|id|length|mesh|name|pair|ring|selector|signature|size|stone|storetype|word|up|down|left|right)|key[_.-]?vault[_.-]?(?:id|name)|keyVaultToStoreSecrets|key(?:store|tab)[_.-]?(?:file|path)|issuerkeyhash|(?-i:[DdMm]onkey|[DM]ONKEY)|keying|(?:secret)[_.-]?(?:length|name|size)|UserSecretsId|(?:io\.jsonwebtoken[ \t]?:[ \t]?[\w-]+)|(?:api|credentials|token)[_.-]?(?:endpoint|ur[il])|public[_.-]?token|(?:key|token)[_.-]?file|(?-i:(?:[A-Z_]+=\n[A-Z_]+=|[a-z_]+=\n[a-z_]+=)(?:\n|\z))|(?-i:(?:[A-Z.]+=\n[A-Z.]+=|[a-z.]+=\n[a-z.]+=)(?:\n|\z)))
  exclude_words:
    - exclude

Against this dummy file:

access_id = "3188ad60055da792f5dd2bba21a88f4d5be80f96"

password = "8dyfuiRyq=vVignorec3RRr_edRk-fK__JItpZ"

password = "exclude_wordpresent123456789"

password = "lowEaaaaaaaaaaaaaaaaaaaa"

# real looking secret
password = "8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ"

yields one finding:

Found unverified result 🐷🔑❓
Detector Type: CustomRegex
Decoder Type: PLAIN
Raw result: 8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ
Name: generic-password
File: trash.txt
Line: 10

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[rosecodym]

Seems pretty straightforward. Hopefully the documentation for this doesn't end up too hairy.

[zricethezav]

Seems pretty straightforward. Hopefully the documentation for this doesn't end up too hairy

Added a little blurb in the readme w/ an example