Question: how to adapt logs to expected format

[stefhak]

Hi, I don't know where to ask so trying here. I have a system where windows hosts collect logs using Loki, and a part of the config.alloy looks like:

loki.source.windowsevent "defender" {
	eventlog_name = "Microsoft-Windows-Windows Defender/Operational"
	forward_to    = [loki.write.defender.receiver]
}
-----
loki.write "defender" {
	external_labels = {
		env         = env("env"),
		service     = "windows_server",
		type        = "security",
		location    = env("location"),
		computer    = string.to_lower(env("COMPUTERNAME")),
		server_uuid = env("uuid"),
	}

	endpoint {
		url = "https://...../push"
	}
}

that produces logs which can look something like (filtering on event_id=5007 just as an example)

	
{"source":"Microsoft-Windows-Windows Defender","channel":"Microsoft-Windows-Windows Defender/Operational","computer":"some_name","event_id":5007,"level":4,"levelText":"Information","opCodeText":"Info","keywords":"0x8000000000000000","timeCreated":"2025-06-24T07:13:18.6155840Z","eventRecordID":14332,"execution":{"processId":3568,"threadId":17564,"processName":"MsMpEng.exe"},"security":{"userId":"S-1-5-18","userName":"NT AUTHORITY\\SYSTEM"},"event_data":"\u003cData Name='Product Name'\u003eMicrosoft Defender Antivirus\u003c/Data\u003e\u003cData Name='Product Version'\u003e4.18.25050.5\u003c/Data\u003e\u003cData Name='Old Value'\u003eHKLM\\SOFTWARE\\Microsoft\\Windows Defender\\UX Configuration\\ToastOrSsoTrigger = 0x0\u003c/Data\u003e\u003cData Name='New Value'\u003eHKLM\\SOFTWARE\\Microsoft\\Windows Defender\\UX Configuration\\ToastOrSsoTrigger = 0x1\u003c/Data\u003e","message":"Microsoft Defender Antivirus Configuration has changed. If this is an unexpected event you should review the settings as this may be the result of malware.\r\n \tOld value: HKLM\\SOFTWARE\\Microsoft\\Windows Defender\\UX Configuration\\ToastOrSsoTrigger = 0x0\r\n \tNew value: HKLM\\SOFTWARE\\Microsoft\\Windows Defender\\UX Configuration\\ToastOrSsoTrigger = 0x1"}

However, doing sigma -t loki on files in rules/windows/builtin/windefend produces things like

{job=~"eventlog|winlog|windows|fluentbit.*"} | json | EventID=5007 and NewValue=~`(?i).*\\Real\-Time\ Protection\\SubmitSamplesConsent\ =\ 0x0.*`

There is a mismatch. I have to start my grafana query by {channel=~"Microsoft-Windows-Windows Defender/Operational"} (instead of job=~"eventlog|winlog|windows|fluentbit.*"}, I have to use event_id rather than EventID, and NewValue does not exist (but there are New Value and New value hidden deep in the log data).

It feels like I am missing some (likely obvious) step or conversion, can someone point at a place where I could understand what I should add (and where)?

Thank you.

[kelnage]

Hey @stefhak! Thanks for the question - it's a very good one, that isn't at all obvious to newcomers to Sigma :)

You're entirely right that many Sigma rules include field names that don't map directly to the field names that actually appear in the log data. Sigma's solution to this is called pipelines. Pipelines allow you to map from rule field names to the log field names (among other things). We created a custom pipeline to allow you to set the stream selector and parser to be more accurate.

To solve your specific issue, you'll want to create a pipeline file windefender.yml that looks something like this:

name: windefender
priority: 100
transformations:
  - id: log_source
    type: set_custom_attribute
    attribute: logsource_loki_selection
    value: '{channel=~"Microsoft-Windows-Windows Defender/Operational"}'
  - id: custom_parser
    type: set_custom_attribute
    attribute: loki_parser
    value: '| json'
  - id: field_mapping
    type: field_name_mapping
    mapping:
      EventID: event_id
      NewValue: message
      # Add more custom rule field name -> log field name mappings here

And then you can use it in Sigma CLI with the following command: sigma convert -t loki -p windefender.yml rules/windows/builtin/windefend

Let me know if that works! We should absolutely update our docs to make this more obvious :)

[stefhak]

Hi @kelnage thanks a lot for the input! I will try out the pipeline. I did some experimenting with pipelines and was able to convert EventID into event_id but I wanted to ask before spending too much effort in that direction.

The custom pipeline you mention, is that part of the code that is being run when doing sigma -t loki? And is the best way forward to modify there or is it better to have a separate pipeline (as the one you provided above)?

Also, I saw some proposing that processing stages are added in config.alloy instead to convert already there. Is that something you would recommend?

Finally, the get-started documentation proposes to use Promtail (instead of Alloy I presume). Would that sort things automatically? (Though I noted that Promtail seems to become deprecated).

Again, thanks a lot for your input!

[stefhak]

PS the example pipeline you provided works perfectly, thanks a lot! I will now look at extending it.