Duplicate query when using multiple pipelines

[defensivedepth]

Ok, so if I run the following with no pipelines:

sigma convert -t eql --without-pipeline sigma/rules/windows/powershell/powershell_module/posh_pm_get_clipboard.yml                                  

Parsing Sigma rules  [####################################]  100%

any where Payload:"*Get-Clipboard*"

Then, I add the windows-logsources pipeline:

sigma convert -t eql -p windows-logsources sigma/rules/windows/powershell/powershell_module/posh_pm_get_clipboard.yml                               

Parsing Sigma rules  [####################################]  100%

any where (Channel like~ ("Microsoft-Windows-PowerShell/Operational", "PowerShellCore/Operational")) and (EventID:4103 and Payload:"*Get-Clipboard*")

Finally, I add the ecs_windows pipeline:

sigma convert -t eql -p ecs_windows -p windows-logsources sigma/rules/windows/powershell/powershell_module/posh_pm_get_clipboard.yml                

Parsing Sigma rules  [####################################]  100%

any where (winlog.channel like~ ("Microsoft-Windows-PowerShell/Operational", "PowerShellCore/Operational")) and ((winlog.channel like~ ("Microsoft-Windows-PowerShell/Operational", "PowerShellCore/Operational")) and (event.code:4103 and winlog.event_data.Payload:"*Get-Clipboard*"))

In that last one, is it expected that there is a duplicate search for winlog.channel?

[andurin]

@thomaspatzke Do you have an idea how to solve this?

[thomaspatzke]

This is caused because both pipelines map generic event types to log sources here in the ecs_windows and here in the windows-logsources pipeline. There are different ways to do this:

• Adding processing items missed in ecs_windows pipeline from windows-logsources, such that only the ECS pipeline must be used. The dict's used in the windows-logsources pipeline can still be used to generate the items, such that no code duplication between both projects occur.
• Removing stuff that is already done in the windows-logsources pipeline from the ecs_windows pipeline. Then both pipelines must be applied by the user.

I prefer the first alternative because it's more user-friendly to use a single pipeline instead of requiring that multiple pipelines are sticked together manually.