Should be able to monitor log files like the original Fail2Ban, not just Windows Events

[JennaScvl]

I spent 3 hours trying to figure out why it wasn't loading when it turned out to be that it just can't monitor logs.
For example I'd been trying to use this as one of my rules
{
"logName": "F:/xampp/apache/logs/error.log",
"eventId": 0,
"ipAddressPattern": "\[client (?\d+\.\d+\.\d+\.\d+):\d+\]",
"failurePattern": "AH00124"
}
and it just wouldn't load. The idea was to block an exploit attempt I keep seeing pop up in my Apache error.log
Also
{
"logName": "F:/xampp/apache/logs/modsec_audit.log",
"eventId": 0,
"ipAddressPattern": ""client_ip":"(?\d+\.\d+\.\d+\.\d+)"",
"failurePattern": ""status":"218""
}
This one is to monitor modsecurity's audit log to block certain common exploit attempts I've seen come in.
But it just can't monitor log files like the original can, apparently.

[jmoeller-ua]

Just wanted to +1 this request. I have a similar challenge with IIS logs. I can have the IIS put its logs into the Event Viewer, but the IP address and the rest of the filtering criteria I'd need to account for are in different EventData Data fields, so I can't use Fail2Ban4Win for this particular challenge. It'd work if I could trigger off of additional fields in an EventLogSelector, or if I could work with the logfile directly, like @JennaScvl is requesting.

[Aldaviva]

Hi @jmoeller-ua,
Feel free to export and upload an .evtx file for the IIS event(s), and I will try to take a look at them and see if they can be selected.
Thanks.

[jmoeller-ua]

An example entry looks like this:

date 2024-07-10 time 23:18:54 s-sitename W3SVC1 s-computername <my_host> s-ip <my_ip> cs-method POST cs-uri-stem <my_url> cs-uri-query - s-port 443 cs-username - c-ip <remote_ip> cs-version - cs(User-Agent) Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/115.0.0.0+Safari/537.36+Edg/115.0.1901.203 cs(Cookie) - cs(Referer) - cs-host - sc-status 200 sc-substatus 0 sc-win32-status 0 sc-bytes 15322 cs-bytes 422 time-taken 351

The Event Viewer puts each element neatly and correctly into the EventData hashtable, but since I need to reference elements other than just c-ip to know if it's actually a block-worthy event (cs(User-Agent), cs-method, etc). If I could work with the entire message via regexp, just like a line in a logfile, I could be off and running.

Unhelpfully, the IIS logger puts in all events with the same event ID of 6200. But I guess it makes sense for a web log that it wouldn't be able to discern outcomes.

[Aldaviva]

That wasn't an .evtx file, but I figured out how to generate my own:

iis-access-log.evtx

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name="Microsoft-Windows-IIS-Logging" Guid="{7e8ad27f-b271-4ea2-a783-a47bde29143b}" />
        <EventID>6200</EventID>
        <Version>0</Version>
        <Level>4</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2024-07-10T23:39:12.6648021Z" />
        <EventRecordID>1</EventRecordID>
        <Correlation />
        <Execution ProcessID="10412" ThreadID="10788" />
        <Channel>Microsoft-IIS-Logging/Logs</Channel>
        <Computer>Thor.aldaviva.com</Computer>
        <Security UserID="S-1-5-18" />
    </System>
    <EventData>
        <Data Name="EnabledFieldsFlags">2478079</Data>
        <Data Name="date">2024-07-10</Data>
        <Data Name="time">23:39:09</Data>
        <Data Name="c-ip">73.202.30.161</Data>
        <Data Name="cs-username">-</Data>
        <Data Name="s-sitename">W3SVC1</Data>
        <Data Name="s-computername">Thor</Data>
        <Data Name="s-ip">192.168.1.17</Data>
        <Data Name="cs-method">GET</Data>
        <Data Name="cs-uri-stem">/health-check.txt</Data>
        <Data Name="cs-uri-query">-</Data>
        <Data Name="sc-status">304</Data>
        <Data Name="sc-win32-status">0</Data>
        <Data Name="sc-bytes">0</Data>
        <Data Name="cs-bytes">766</Data>
        <Data Name="time-taken">103</Data>
        <Data Name="s-port">443</Data>
        <Data Name="csUser-Agent">Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/126.0.0.0+Safari/537.36</Data>
        <Data Name="csCookie">-</Data>
        <Data Name="csReferer">-</Data>
        <Data Name="cs-version">-</Data>
        <Data Name="cs-host">-</Data>
        <Data Name="sc-substatus">0</Data>
        <Data Name="CustomFields" />
    </EventData>
</Event>

This can be enabled by going to the Logging feature page for a site in IIS Manager, configuring it to use W3C format and log ETW events, and enabling and watching Event Viewer > Application and Service Logs > Microsoft > Windows > IIS-Logging > Logs.

---

Given this structure, it seems like allowing an additional optional configuration property to be specified in the EventLogSelector to act as a predicate in the XPath query generated by selectorToQuery would allow you to filter based on the user agent string, request method, response status code, and other data element values.

[jmoeller-ua]

That wasn't an .evtx file,

Indeed, I just wanted to strip out the identifying information. I had set it up just as you had mentioned.

Given this structure, it seems like allowing an additional optional configuration property to be specified in the EventLogSelector to act as a predicate in the XPath query generated by selectorToQuery would allow you to filter based on the user agent string, request method, response status code, and other data element values.

Yes, that's sort of what I was imagining, additional optional configuration items that I could refer to put a specific event in scope of Fail2Ban4Win. Would love to see it if you think it'd be a reasonable feature to implement.

[Aldaviva]

Right, those ETW instructions were for me and anyone else who looks at this issue in the future, because I had to figure it out and was worried I'd forget next time I have to do it (when updating unit tests, adding new dependent features, etc).

I believe adding the additional XPath predicate to the configuration is a reasonable feature to add. I will try to find the time soon to try it out and send you a development snapshot build to see if it fits your use case too before releasing it.

[Aldaviva]

@jmoeller-ua: give this a try and let me know how well it fits your use case

Developer snapshot executable

Fail2Ban4Win.zip
(minimally tested so far)

Configuration

There is now a new optional property of EventLogSelector configuration objects with the key eventPredicate and the value of a string. The value is an XPath 1.0 predicate expression, which must be surrounded by square brackets and is evaluated by ETW against the top-level Event object of the ETW log record element.

Multiple predicates can be constructed with XPath and and or operators, or by concatenating multiple predicates like [predicate1][predicate2] instead of using the and operator. Note that functions like contains() and starts-with() are unfortunately not supported by Windows ETW.

Example

This selector should select only IIS access logs where the response status code is 403.

{
    "logName": "Microsoft-Windows-IIS-Logging/Logs",
    "source": "IIS-Logging",
    "eventId": 6200,
    "ipAddressEventDataName": "c-ip",
    "eventPredicate": "[EventData/Data[@Name='sc-status']=403]"
}

This will result in Fail2Ban4Win using an effective XPath expression of

*[System/EventID=6200][System/Provider/@Name=\"IIS-Logging\"][EventData/Data[@Name='sc-status']=403]

Here is another example that matches requests with both status code 304 and request method GET.

{
    "eventPredicate": "[EventData[Data[@Name='sc-status']='304'][Data[@Name='cs-method']='GET']]"
    /* other properties are the same */
}

Here is an example of an or expression to match two different status codes.

{
    "eventPredicate": "[EventData/Data[@Name='sc-status']='304' or EventData/Data[@Name='sc-status']='403']"
    /* other properties are the same */
}

[jmoeller-ua]

That would be fantastic. I'd be happy to try it out.

…

________________________________ From: Ben Hutchison ***@***.***> Sent: Wednesday, July 10, 2024 5:42:11 PM To: Aldaviva/Fail2Ban4Win ***@***.***> Cc: Moeller, John D - (jmoeller) ***@***.***>; Mention ***@***.***> Subject: [EXT] Re: [Aldaviva/Fail2Ban4Win] Should be able to monitor log files like the original Fail2Ban, not just Windows Events (Issue #33) External Email

________________________________ Right, those ETW instructions were for me and anyone else who looks at this issue in the future, because I had to figure it out and was worried I'd forget next time I have to do it (when updating unit tests, adding new dependent features, etc). I believe adding the additional XPath predicate to the configuration is a reasonable feature to add. I will try to find the time soon to try it out and send you a development snapshot build to see if it fits your use case too before releasing it. — Reply to this email directly, view it on GitHub<#33 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APM4GNXCFXGYFIEAQ5THIBLZLXIGHAVCNFSM6AAAAABJG57YP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRG43TINJYGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jmoeller-ua]

@jmoeller-ua: give this a try and let me know how well it fits your use case

Developer snapshot executable

Fail2Ban4Win.zip (minimally tested so far)

It works great, this is amazing. I can't believe how fast you did this!

[Aldaviva]

Thanks! Great to hear it's working for you.

This was a pretty straightforward change because it leveraged Windows' existing ETW XPath filtering, which this project was already using. By adding the one new optional, backwards-compatible configuration property, it augmented the existing Event ID and Source filtering that this project had already been set up to use. So it was a small change that did not reinvent the wheel.

[Aldaviva]

I'm splitting the eventPredicate feature that we've been discussing here into a different issue, #34, because it resulted in a different implementation than the topic of this issue, tailing text log files. Any further eventPredicate discussions can happen in #34, while this issue can remain for text log files.