Description
Description
The introduction of the new logs index mode uses two fields for index sorting, host.name and timestamp.
Overriding index sorting is still possible using using Elasticsearch index sort options.
While the timestamp field is always expected to be present, different integrations might miss the host.name field or just require sorting on multiple (keyword) fields other than the timestamp.
We need to support a more flexible index sorting configuration in LogsDB which allows integrations to decide a set of fields to use for sorting other than the defaults. We should explore the idea of using a meta field, something like logs_id, which is used as a meta filed to collect values for fields to sort on. Users and integrations would be able to specify that such a field is a combination of one or more fields whose values are used for sorting. As an example, integration might specify that they would like to sort on service.name and host.name other than on timestamp. By default, such field should just use host.name and timestamp.
NOTE: such meta field is not expected to be used for logs deduplication.
Also, such feature is not necessarily required for LogsDB General Availability.