Vague BigDecimal() errors related to mutate filters

[landon-lengyel]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)
   9.2.1
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
   DEB
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
   systemd
4. Plugins installed: (bin/logstash-plugin list --verbose)

JVM (e.g. java -version): (Bundled JDK)

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (java -version)
   openjdk 21.0.9 2025-10-21 LTS
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
   Bundled
3. Value of the LS_JAVA_HOME environment variable if set.
   N/A

OS version (uname -a if on a Unix-like system):
Debian 13
Linux SYSLOG03 6.12.38+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.38-1 (2025-07-16) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I'm not sure if you want this as a bug report since it's a behavior change from an update, or feature request.

Ever since this commit which added bigdecimal, our pipelines have not worked properly and have gotten spammed with BigDemical() errors.

I've resolved one of the biggest issues, turns out it was caused by logs being sent by WinLogBeat using inconsistent LogLevel. Sometimes it's an integer, sometimes it's a string, etc. So if there's a mutate filter to attempt to convert it to an integer, it doesn't properly check that the data is convertible and fails.

My expected behavior here is that the log should as least tell us which pipeline the error occurred in, and ideally which line the failed mutate filter occurred. With multiple pipelines, just saying that some error occurred in a mutate somewhere is extremely difficult to troubleshoot.

Steps to reproduce:

1. Setup a WinLogBeat connection to a Logstash server (config below).
2. Setup Logstash pipeline (configs below).
3. Watch vague errors occur.

WinLogBeat config:

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
    ignore_older: 72h
    # event_id: 4624-4630, -4701
  - name: System
    ignore_older: 72h
  - name: Microsoft-Windows-GroupPolicy/Operational
    ignore_older: 72h
    level: critical, error, warning
  - name: Microsoft-Windows-TaskScheduler/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Defender/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Defender/WHC
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurity
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurityVerbose
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/FirewallDiagnostics
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/FirewallVerbose
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Network Isolation Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Remote Management/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-WMI-Activity/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-WindowsUpdateClient/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Dhcp-Server/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Dhcp-Server/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Dhcp-Server/FilterNotifications
    ignore_older: 72h    
  - name: Microsoft-Windows-PrintService/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-PrintService/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-Compute/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-Config/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-StorageVSP/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-VMMS/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-VMMS/Networking
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-VMMS/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-VMMS/Storage
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-Hypervisor-Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-VmSwitch/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-Worker/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-Hyper-V-Hypervisor-Operational
    ignore_older: 72h
  - name: OpenSSH/Admin
    ignore_older: 72h
  - name: OpenSSH/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-CertificateServicesClient-Lifecycle-User/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-CertificateServicesClient-Lifecycle-System/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-CertificateServicesClient-CredentialRoaming/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-CertificateServices-Deployment/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Crypto-DPAPI/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Crypto-DPAPI/BackUpKeySvc
    ignore_older: 72h
  - name: Microsoft-Windows-PowerShell/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-PowerShell/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-SMBServer/Audit
    ignore_older: 72h
  - name: Microsoft-Windows-SMBServer/Connectivity
    ignore_older: 72h
  - name: Microsoft-Windows-SMBServer/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-SMBServer/Security
    ignore_older: 72h
  - name: Microsoft-Windows-NTLM/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-LocalSessionManager/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-LocalSessionManager/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-RemoteConnectionManager/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-SessionBroker/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-SessionBroker/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-SessionBroker-Client/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-SessionBroker-Client/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-Gateway/Admin
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-Gateway/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational
    ignore_older: 72h
    level: critical, error, warning

processors:

output:
  logstash:
    encoding: utf-8
    hosts: [ "{HOST}:10070" ]
    ssl.ssl_enabled: true
    ssl.certificate_authorities: [ "C:\\Program Files\\Elastic\\Beats\\9.2.1\\winlogbeat\\CA_Certificate_Chain.pem" ]

Logstash pipeline conf.d:

input {
    beats {
        ecs_compatibility => "disabled"
        enrich => [ "ssl_peer_metadata", "source_metadata" ]
        host => "0.0.0.0"
        port => "10070"
        ssl_enabled => true
        ssl_certificate => "/etc/logstash/http-certs/syslog.pem"
        ssl_key => "/etc/logstash/http-certs/syslog.key"
        ssl_certificate_authorities => "/etc/logstash/http-certs/OLCA01-TrustChain.pem"
        ssl_client_authentication => "optional"
    }
}

filter {
        if [log][level]{
                mutate { convert => { "[log][level]" => "integer" } }
        }
}

output {
  file {
          path => "/opt/output.json"
          codec => json
  }
}

Provide logs (if relevant):

Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695072,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695072,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695072,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695073,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695073,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695073,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695073,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695074,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695074,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695074,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695074,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exception caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}
Nov 17 16:58:15 SYSLOG03 logstash[6779]: {"level":"WARN","loggerName":"logstash.filters.mutate","timeMillis":1763398695075,"thread":"[sys_win_server]>worker59","logEvent":{"message":"Exce
ption caught while applying mutate filter","exception":"invalid value for BigDecimal(): \"error\""}}

[donoghuc]

This is certainly new behavior in the mutate filter. One option would be to downgrade the version in production while you sort out an alternative or add conditional logic to identify if a message is coercible. Note that the coerce behavior before had flaws (which is why that bug fix was included).

I'm trying to understand your logging request. Currently the "thread":"[sys_win_server] bit in the provided logs is the link back to which logstash pipeline the error is emanating from. I see you shared a snippet from presumably that pipeline, are there others as well? If you can share a rough idea of how many pipelines are involved and your log4j2.properties we can take a closer look at making sure all the information you need for debugging which pipelines errors are originating from.

For example, using default logging we see a message like:

logstash-1  | [2025-11-17T22:06:31,388][WARN ][logstash.filters.mutate  ][pipeline1][b801eb4536ebedf6dffba66f748c482646d373b6e624f613789bb50303ada97d] Exception caught while applying mutate filter {:exception=>"invalid value for BigDecimal(): \"error\""}

Which is much more explicity about which pipeline the error originated from.

[landon-lengyel]

Ah I see. I didn't realize that the log content is so different from the default log4j format to the JSON format. That's very good to know. I also didn't know about the coerce function, I will experiment with that to see if that can help with this situation. Thank you for that info!

That does tell me which pipeline it is coming from, which is a huge help. It would be really nice if it could also include which line in the pipeline file the error was thrown at, but I can't imagine I'm the first person to request that so I suspect that's a big ask (unless there's some log4j voodoo I'm unaware of).

As far as my log4j file, it is completely default. I can share it if you still want it. There are about 7 pipelines but changing that log format back helps clear up which one(s) it's coming from.