LogsDB: Add synthetic_source_keep = none to arrays where order/duplicates do not matter

[andrewkroh]

For array fields treated as unordered sets, we should add synthetic_source_keep: "none" to the mappings to optimize storage under LogsDB. Fields like host.ip and related.ip would be candidates because order and duplicates are irrelevant.

Adding this option prevents the array field from being stored in _source.

Support for this is in-progress in Elasticsearch and will be first available in 8.16.

References

• https://github.com/elastic/elasticsearch/blob/a2df7e7229e772eddc9a8aba2ecfdbd162810c78/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java#L33
• https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#synthetic-source-keep

Related

• LogsDB compatibility: specify if the ordering of arrays needs to be preserved #2372 (no longer relevant as we switched to an opt-in model for array optimization in logsdb)

Tasks

Preview Give feedback

1. Add support for synthetic_source_keep = none #2422
   backport-8.x +1
   
2. Update ecs@mappings dynamic template in elastic/elasticsearch
3. [Change Proposal] Support 'synthetic_source_keep' in field mapping attributes package-spec#861
   discuss +1
4. [8.18 testing] Validate synthetic_source_keep: "none" is added to mappings for ECS tags integrations#12485
   Team:Security-Deployment and Devices +1

[andrewkroh]

A first step that can be taken here is to add support into the ECS repo to allow expressing which fields are unordered sets. This can be done before Elasticsearch has the synthetic_source_keep: "none" mapping parameter. Once Elasticsearch has it then we can update the generators to output Elasticsearch mappings containing the parameter.

I would like to begin the process of annotating the fields that can receive this optimization, but we need support in the schema/*.yml files first.

[andrewkroh]

This work can proceed. LogsDB is enabled by default in Serverless and will be on by default in ES 9.0. So now is a great time to implement the optimizations for fields like tags and related.ip.

The changes should also be added to the ecs@mappings dynamic template that is bundled into Elasticsearch.