Add support for metadata in Fluentd event structure

[cosmo0920]

Is your feature request related to a problem? Please describe.

Currently, Fluentd handles the following events structure:

[TAG, TIMESTAMP, {RECORD}]

We wanted to change to handle metadata in Fluentd event structure:

[TAG, [TIMESTAMP, {METADATA}], {RECORD}]

Note that, in the above structure, timestamp is already one of the metadata structure and other metadata should be handled in msgpack map that is represented Hash in Ruby.

Describe the solution you'd like

Handle metadata requirements in Fluentd.

Describe alternatives you've considered

The minimum solution is just handled as normal Forwarded events with metadata from fluent-bit:

From fluent-bit, events metadata should be sent as:

[TAG, TIMESTAMP, {RECORD}, {metadata key/val in OPTION}]

And metadata part should be stored in a special key. However, supporting metadata in the second part of the event should be simpler for long term perspective.

Additional context

We'd discussed this RFC in fluent-bit issues:
fluent/fluent-bit#6666

UPDATE:

Also, Fluent Bit with out_forward just uses options to represent their metadata. We don't expect any breakage for sending logs type of Fluent Bit events from Fluent Bit to Fluentd.

[ashie]

It seems that we need to revise the forward protocol of fluentd to realize it:
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#entry

[cosmo0920]

Yes. But there is no breakage for sending with forward protocol from Fluent Bit to Fluentd for now. Not in a hurry to handle this RFC.

[ashie]

BTW this protocol has the year 2038 problem since TIMESTAMP is only 32bit length.
We also need to modify it to allow timestamp extension type of MessagePack when we revise the specification.

I didn't notice this issue until I read Fluentd実践入門.