beats 7.x

[Raboo]

Hi,

Do you have any plans to upgrade beats to 7.x?
Currently it doesn't work with a 7.x cluster.

2020/03/23 23:18:02.991238 client.go:447: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [varnishlogbeat-2020.03.23] as the final mapping would have more than 1 type: [_doc, doc]"}

It is trying to create documents with multiple types.
I tried bumping the beats version in Gopkg.toml, but then I couldn't compile.

[phenomenes]

I'm planning to but I don't have the bandwidth at the moment.
Would you like to send a PR?

[Raboo]

I can update Gopkg.toml and submit a PR. But I don't know any go, so I wouldn't know what to do when I hit the compile error.. I would need someone that know go to take it on from there.

[Mathsterk]

A quick workaround: set template.enabled: false in your config, then it'll push to elasticsearch 7.x without problems 😄

[Raboo]

@Mathsterk are you sure? Did you actually try it?
Cause I think I tried that before and it failed. But I could be wrong since this was like a year ago.
Have you tried with 7.6 and newer?

[Mathsterk]

Yeah, with 7.9.x It was unstable (random crashes at random intervals) with the amount of logs (about 1k req/s) (raw varnishlog is about 500k lines/s). I am however looking at alternative ways of accomplishing the same task. Most likely a mix of fluentbit and fluentd

[Raboo]

@Mathsterk Ok, if you find a decent solution, please ping me with some details :-)