Fix check bypass via square brackets Issue #857 by tsigouris007 · Pull Request #860 · Yelp/detect-secrets

tsigouris007 · 2024-06-20T14:21:27Z

Please check if the PR fulfills these requirements

Tests for the changes have been added

~~Docs have been added / updated~~

All CI checks are green

Tests pass

What kind of change does this PR introduce?

Fixes behavior on missed secrets that have square brackets before their default value.
Example:

"<%= ENV['CLIENT_ACCESS_KEY_ID'].presence || 'AKIA123456789ABCDEF1' %>"

What is the current behavior?

Issue: #857

What is the new behavior (if this is a feature change)?

Escapes the square brackets and catches secrets properly.

Does this PR introduce a breaking change?

No breaking changes.

lorenzodb1

Hi @tsigouris007, thank you for your PR!

Could you explain what the bug was?

tsigouris007 · 2024-06-20T14:45:44Z

Hi @tsigouris007, thank you for your PR!

Could you explain what the bug was?

Hi,
You can visit the issue for the compete write-up.

lorenzodb1 · 2024-06-21T09:41:50Z

Hi @tsigouris007, #857 describes the issue, I was looking for an explanation around what's causing said issue in the code

tsigouris007 · 2024-06-21T10:43:53Z

Hi @tsigouris007, #857 describes the issue, I was looking for an explanation around what's causing said issue in the code

I haven't quite had the time to find the exact location that brakes this.
The combined opening/closing square brackets [] seem to be causing the lines to not be fetched at all for checking.

For example I got AWSKeyDetector's code into a separate standalone python file and the following secret was caught properly by the regex:

access_key_id: "<%= ENV['CLIENT_ACCESS_KEY_ID'].presence || 'AKIA123456789ABCDEF1' %>"

I am taking wild a guess here and I most probably think the problem is how the checked file lines are fetched (and that's why I thought of "escaping" the square brackets earlier).
As far as I can help atm you can go to detect_secrets/core/scan.py method _process_line_based_plugins and the following condition is returning True and is quitting early:

if _is_filtered_out(
            required_filter_parameters=['line'],
            filename=filename,
            line=line,
            context=code_snippet,
):

If you remove it or add a pass instead of continue these secrets are caught because it hits the else clause in _is_filtered_out loop returning a debug message of:

Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.

I can take a closer look but it will take me at least a couple of days.

lorenzodb1 · 2024-06-21T11:09:30Z

Understanding what exactly is causing the bug will help us finding a good solution moving forward rather than a workaround that could potentially add another set of issues, hence why I'm asking this. Take your time and please let me know what you'll find!

tsigouris007 · 2024-06-21T11:16:56Z

Understanding what exactly is causing the bug will help us finding a good solution moving forward rather than a workaround that could potentially add another set of issues, hence why I'm asking this. Take your time and please let me know what you'll find!

It's understandable.
I 'll try to get back to this asap.

tsigouris007 · 2024-06-25T15:40:30Z

Hi @lorenzodb1 ,
so I found where the problem occurs.
In file detect_secrets/filters/heuristic.py there is _get_indirect_reference_regex with a regex of:

return re.compile(r'([^\v=!:]*)\s*(:=?|[!=]{1,3})\s*([\w.-]+[\[\(][^\v]*[\]\)])')

So the choice is to either change this regex a bit or escape earlier the square brackets.
The latter could be done as is in my PR or we could do it also in is_indirect_reference function inside the heuristic.py file right before the boolean return as shown:

def is_indirect_reference(line: str) -> bool:
    """
    Filters secrets that take the form of:

        secret = get_secret_key()

    or

        secret = request.headers['apikey']
    """
    # Constrain line length as the heuristic's intention is to target lines that resemble
    # function calls. The constraint avoids catastrophic backtracking failures of the regex.
    if len(line) > 1000:
        return False
    line = line.replace('[', '\[').replace(']', '\]')
    return bool(_get_indirect_reference_regex().search(line))

This produces the correct results.

After playing around with the regex I came up with the following that seems to be working (tested on a huge repo with ~1.2k secrets):

def _get_indirect_reference_regex() -> Pattern:
    # Regex details:
    #   ([^\v=!:]*)     ->  Something before the assignment or comparison
    #   \s*             ->  Some optional whitespaces
    #   (:=?|[!=]{1,3}) ->  Assignment or comparison: :=, =, ==, ===, !=, !==
    #   \s*             ->  Some optional whitespaces
    #   (
    #       [\w.-]+     ->  Some alphanumeric character, dot or -
    #       [\[\(]      ->  Start of indirect reference: [ or (
    #       [^\v]*      ->  Something except line breaks
    #       [\]\)]      ->  End of indirect reference: ] or )
    #   )
    # return re.compile(r'([^\v=!:]*)\s*(:=?|[!=]{1,3})\s*([\w.-]+[\[\(][^\v]*[\]\)])') # Left the old one for reference
    return re.compile(r'([^\v=!:"<%>]*)\s*(:=?|[!=]{1,3})\s*([\w.-]+[\[\(][^\v]*[\]\)])')

How do you want to go through with this?
I have no strong opinion.

lorenzodb1 · 2024-07-16T12:59:14Z

Hi @tsigouris007, sorry for the delay in replying. I personally prefer the second option. Thanks for investigating this and coming up with a solution!

tsigouris007 · 2024-07-16T15:38:23Z

Hi @tsigouris007, sorry for the delay in replying. I personally prefer the second option. Thanks for investigating this and coming up with a solution!

Hi @lorenzodb1 ,
I have updated the PR.
Give it a test if you want, for sanity reasons.

lorenzodb1 · 2024-07-29T09:57:25Z

    #       [\]\)]      ->  End of indirect reference: ] or )
    #   )
-    return re.compile(r'([^\v=!:]*)\s*(:=?|[!=]{1,3})\s*([\w.-]+[\[\(][^\v]*[\]\)])')
+    return re.compile(r'([^\v=!:"<%>]*)\s*(:=?|[!=]{1,3})\s*([\w.-]+[\[\(][^\v]*[\]\)])')


@tsigouris007 could you add a test for this change?

Sorry for the delay.
I added two test cases for this change.
Are we good to go?

Isn't that what you were trying to fix with this PR?

I was catching the first case as that was my initial issue as described above.
So, I beefed up the regex and now it catches both cases (broke it in multiple lines for readability, it's not that far from the original one).
I also added a few more test cases that should be caught following this logic and added a brief explanation for each case.
Let me know if this works for you now.

tsigouris007 · 2025-01-15T16:33:46Z

Hi @lorenzodb1 ,
it's been quite some time.
Is the latest update what we're looking for?

lorenzodb1 · 2025-04-13T10:04:20Z

Hi @tsigouris007, I'm sorry for the late reply. I'm no longer involved in this project as a maintainer, but I'll tag @jpdakran so he can take a look when it's most convenient for him. Changes do lgtm anyway 👍🏼

tsigouris007 · 2025-04-13T10:22:13Z

Hi @tsigouris007, I'm sorry for the late reply. I'm no longer involved in this project as a maintainer, but I'll tag @jpdakran so he can take a look when it's most convenient for him. Changes do lgtm anyway 👍🏼

Thank you very much!

lorenzodb1 reviewed Jun 20, 2024

View reviewed changes

tsigouris007 force-pushed the fix_issue_857 branch from f5be7c0 to 23e5097 Compare July 16, 2024 15:35

lorenzodb1 reviewed Jul 29, 2024

View reviewed changes

lorenzodb1 reviewed Oct 11, 2024

View reviewed changes

Comment thread tests/filters/heuristic_filter_test.py Outdated

tsigouris007 force-pushed the fix_issue_857 branch from b7c79e0 to c4210a1 Compare November 7, 2024 13:40

tsigouris007 requested a review from lorenzodb1 November 7, 2024 13:43

tsigouris007 added 2 commits July 2, 2025 12:49

[Fix] Square brackets bypass Issue Yelp#857

35f94b8

[Test] Add test case for issue Yelp#857

74b7a92

tsigouris007 force-pushed the fix_issue_857 branch from c4210a1 to 74b7a92 Compare July 2, 2025 09:49

Conversation

tsigouris007 commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorenzodb1 left a comment

Choose a reason for hiding this comment

Uh oh!

tsigouris007 commented Jun 20, 2024

Uh oh!

lorenzodb1 commented Jun 21, 2024

Uh oh!

tsigouris007 commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorenzodb1 commented Jun 21, 2024

Uh oh!

tsigouris007 commented Jun 21, 2024

Uh oh!

tsigouris007 commented Jun 25, 2024

Uh oh!

lorenzodb1 commented Jul 16, 2024

Uh oh!

tsigouris007 commented Jul 16, 2024

Uh oh!

lorenzodb1 Jul 29, 2024

Choose a reason for hiding this comment

Uh oh!

tsigouris007 Oct 1, 2024

Choose a reason for hiding this comment

Uh oh!

lorenzodb1 Nov 4, 2024

Choose a reason for hiding this comment

Uh oh!

tsigouris007 Nov 7, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tsigouris007 commented Jan 15, 2025

Uh oh!

lorenzodb1 commented Apr 13, 2025

Uh oh!

tsigouris007 commented Apr 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tsigouris007 commented Jun 20, 2024 •

edited

Loading

tsigouris007 commented Jun 21, 2024 •

edited

Loading