json configuration not recognized anymore

[vaol]

Hello,

I updated from 4.11 to 4.14 with the following input definition:

filebeat::input { 'syslog':
  paths          => ['/var/log/syslog.json'],
  ignore_older   => '10m',
  close_older    => '10m',
  scan_frequency => '5s',
  fields         => { logtype => 'syslog', },
  json           => {
    keys_under_root => true,
    add_error_key   => true,
    overwrite_keys  => true,
  },
}

and the corresponding line example from /var/log/syslog.json
{"@timestamp":"2023-07-17T21:02:55.407481+00:00","@version":"1","message":"Applied catalog in 24.84 seconds","host":"code-acceptance-db-1","severity":"notice","facility":"daemon","programname":"puppet-agent","procid":"9130"}

My issue being the programname no longer recognized as input field after the update.

After checking /etc/filebeat/conf.d/syslog.yml and the corresponding documentation I changed the current json into parsers.0.ndjson and I got the field back.

Is there another way to have the same behaviour than previously with the 4.14 version ?

Thank you !

Best regards,
OV

[psychonaut]

It's actually not a module fault but a filebeat upgrade fault I suppose, the newer version of filebeat supports only ndjson parsers. I checking my options with this module since supporting parsers would mean some rewrites of the input template. BTW. did you manage using this parsers.0.ndjson trick with puppet module or outside of the module? I don't think the current version support injecting these parsers.0.ndjson trick.

[vaol]

The current version of the module does not support anything else than json as it is hardcoded. I changed manually to parsers.0.ndjson to make it work.
From my point of view there is a incompatibility between the default input type = filestream and the hardcoded json entry in the configuration.

[psychonaut]

Actually I've end up with wrapper:

#
# this is resource used when upstream filebeat module don't support configuration quirks which filebeat could have
#
define filebeat_wrapper::input (
  Array[Hash]               $config,
  Enum['absent', 'present'] $ensure       = present,
  # this resource has hash with sensible default values but it could be switched off
  Boolean                   $use_defaults = true,
) {

  $defaults = {
    'encoding' => 'plain',
    'document_type' => 'log',
    'prospector' => {
      'scanner' => {
        'check_interval' => '10s',
      }
    },
    'harvester_buffer_size' => 16384,
    'message_max_bytes' => 10485760,
    'tail_files' => false,
    'backoff' => {
      'init' => '1s',
      'max' => '10s',
    },
    'backoff_factor' => 2,
    'close_inactive' => '5m',
    'close_renamed' => false,
    'close_removed' => true,
    'close_eof' => false,
    'clean_inactive' => 0,
    'close_timeout' => 0
  }

  #
  # merge with default settings
  #
  if $use_defaults {
    $merged_config  = $config.map |$entry| {
      deep_merge($defaults, $entry)
    }
  } else {
    $merged_config = $config
  }

  $validate_cmd = "\"${filebeat::filebeat_path}\" -c \"${filebeat::config_file}\" test config"
  file { "filebeat-${name}":
    ensure       => $ensure,
    path         => "${filebeat::config_dir}/${name}.yml",
    owner        => 'root',
    group        => 'root',
    mode         => $filebeat::config_file_mode,
    content      => to_yaml($merged_config),
    validate_cmd => $validate_cmd,
    notify       => Service['filebeat'],
    require      => File['filebeat.yml'],
  }
}

But I think the problem is that when you want to support many filebeat versions you just can't cover it with a fancy template. I think using to_yaml with config validation is the only option. I can also think that the ruby function which would use JSON schema for validation would be helpful here, similarly as in helm and JSON schema validation: https://www.arthurkoziel.com/validate-helm-chart-values-with-json-schemas/

[vaol]

Well, this sounds like a solution indeed, but makes the usage of such input more complex though.
Don't you believe that it is realistic to use either json or parsers.0.ndjson depending on the filebeat version.
After all this was the idea behind choosing filestream as the default depending on this fact.

[psychonaut]

The idea is great but every time Elastic changes something in configuration format you have to wait for the module author to release a new version.