deferred expressions not applied in correlation rules

[arblade]

Description

When a correlation rule has, in one of its "sub rule" or "referenced rule" a deferred expression, like a regex (converted to | regex) or an regex oring, these are removed from the query.

Example :

title: Base rule
name: base_rule
status: test
logsource:
    category: test
detection:
    selection:
        fieldA: value1
        fieldB: value2
    condition: selection
---
title: Base rule 2
name: base_rule2
status: test
logsource:
    category: test
detection:
    selection:
        fieldA: value1
        fieldB|re: value2
    condition: selection
---
title: Multiple occurrences of base event
status: test
correlation:
    type: event_count
    rules:
        - base_rule
        - base_rule2
    group-by:
        - fieldC
        - fieldD
    timespan: 15m
    condition:
        gte: 10

will currently be converted into:

| multisearch
[ search fieldA="value1" fieldB="value2" | eval event_type="base_rule" ]
[ search fieldA="value1" | eval event_type="base_rule2" ]

| bin _time span=15m
| stats count as event_count by _time fieldC fieldD

| search event_count >= 10

We note that fieldB disappeared from the second sub search.

Explanation

This is because of this line commited following this issue. The thing is, we need to get deferred expressions in the search part.
And as deferred are handled in the finalize_query step, in my opinion, we need to reactivate the finalization for rules that are part of a correlation rule, and let the responsibility of the finalization to the backends (ie, let them select which part need to be in the search part of the query, for instance, here, we want deferred expressions to be in the search part, but we dont want this for fields which are converted to | table field1, field2 ...).

When I tried to include fields (so the | table) in the subsearch, splunk fired the error multisearch subsearches might only contain purely streaming operations)

So, in the splunk backend this could be done by handling this in the finalize_query_default:

def finalize_query_default(
        self, rule: SigmaRule, query: str, index: int, state: ConversionState
    ) -> str:
        if not rule._backreferences: # checking if rule is not part of a correlation rule
            table_fields = " | table " + ",".join(rule.fields) if rule.fields else ""
            return query + table_fields
        else :
            return query

I opened this issue as I was working on the #51, as I need to create new deferred expressions which are missing in the correlation rules.

Summary

So, in a nutshell, I propose:

• making a PR on pysigma and change this line by removing the if condition, allowing all rules to be finalized, even those which are part of a correlation rule.
• making a PR on the pysigma splunk backend to update the finalize_query_default, as mentioned previously.