Conditions added by processing pipelines are deferred with OR-ed regex

[thomaspatzke]

Problem

Regular expressions logically linked with OR are deferred

Reproduction

Processing pipeline:

name: Example Sigma Pipeline Config
priority: 100
transformations:
  - id: prefix_source_and_index
    type: add_condition
    conditions:
      index: test
      source: test

Rule:

title: Example Sigma Rule
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    EventID: 4688
    CommandLine|re:
      - "suspicious_command"
  selection2:
    Image|re:
      - "suspicious_command"
  condition: selection or selection2

Result:

| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search index="test" source="test" (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

Expected result:

index="test" source="test"
| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

[arblade]

Proposed Solution

Summary

The problem is that deferred expressions (those beginning with a pipe |) have no search terms in front of them. Splunk requires at least one term before a pipe, so the query fails.

A simple fix is to treat all necessary terms as deferred, then prepend them to the final search. By “necessary”, I mean any term whose only parent operators are AND (i.e., it never appears under an OR). This approach produces exactly the result described in your comment.

Details

Consider this slightly modified example:

detection:
  selection:
    EventID: 4688
    CommandLine|re:
      - "suspicious_command"
  selection2:
    Image|re:
      - "suspicious_command"
  selection3:
    User: null
  condition: (selection OR selection2) AND selection3

After adding an index via add_condition, pysigma generates:

| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search index=* (EventID=4688 CommandLineCondition="true") OR ImageCondition="true" AND NOT User=*

Because AND is commutative, you can move any clause joined only by AND to the front without changing the result:

index=* AND NOT User=*
| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

More generally, any term whose entire path in the condition tree consists only of AND operators can be prepended. For example:

(sel1 OR sel2) AND (sel3 AND (sel4 OR sel5))

is equivalent to:

sel3 AND (sel1 OR sel2) AND (sel4 OR sel5)

More

A cleaner way would be to get all parts that are not linked with "complex" deferred expressions (understand with deferred expressions converted with a | rex ... | eval), and add them to the front of the query, but it seems way more complex to handle, we would first need to identify the "complex" deferred expressions, then target parts that are not linked to them with OR operator, to add them in front of the query, while keeping the logic between them, like :

(sel1 OR sel_regex) AND sel4 AND (sel2 OR sel3) AND sel5

would become:

sel5 AND sel4 AND (sel2 OR sel3) AND (sel1 OR sel_regex)

with sel1 OR sel_regex which is a complex deferred expression.

My current solution is the following one:

sel5 AND sel4 AND (sel1 OR sel_regex) AND (sel2 OR sel3)

This is not perfect, but it is making things work, with little changes in the splunk.py file.

Here is my solution: https://github.com/arblade/pySigma-backend-splunk/blob/main/sigma/backends/splunk/splunk.py
If the concept is ok, I can work on a clean PR to add these changes.

[thomaspatzke]

Looks good to me, would be great to have it as PR!