Open
Description
Ingest into indices with index.mode set to logsdb skips validation because index.mapping.ignore_malformed is set to true for those, and we skip validation if index.mapping.ignore_malformed is true (at least for numbers).
This behavior works as intended for ingest and simulate ingest, as long as no overrides are given.
Add a template for a logsdb data stream:
PUT _index_template/datastream-logsdb-template
{
"index_patterns": [
"datastream-logsdb-*"
],
"data_stream": {},
"template": {
"mappings": {
"properties": {
"my_field": {
"type": "long"
}
}
},
"settings": {
"index": {
"mode": "logsdb"
}
}
},
"priority": 500
}
Create the data stream:
PUT /_data_stream/datastream-logsdb-1
Now when we simulate ingest, it correctly (for logsdb) skips validation:
POST /_ingest/_simulate
{
"docs": [
{
"_index": "datastream-logsdb-1",
"_id": "0",
"_source": {
"my_field": "asdf",
"@timestamp" : "2027-03-30T12:02:01.789Z"
}
}
]
}
And when we do normal ingest, it correctly skips validation:
POST datastream-logsdb-1/_doc
{
"my_field": "asdf",
"@timestamp" : "2027-03-30T12:02:01.789Z"
}
But if we add a completely unrelated and unused component_template_override, it incorrectly (for logsdb) validates the outputs:
POST /_ingest/_simulate
{
"docs": [
{
"_index": "datastream-logsdb-1",
"_id": "0",
"_source": {
"my_field": "asdf",
"@timestamp" : "2027-03-30T12:02:01.789Z"
}
}
],
"component_template_substitutions": {
"template123": {
"template": {
"mappings": {
"properties": {
"my_field": {
"type": "long"
}
}
}
}
}
}
}
Gives output:
{
"docs": [
{
"doc": {
"_id": "0",
"_index": "datastream-logsdb-1",
"_version": -3,
"_source": {
"@timestamp": "2027-03-30T12:02:01.789Z",
"my_field": "asdf"
},
"executed_pipelines": [],
"error": {
"type": "document_parsing_exception",
"reason": "[1:53] failed to parse field [my_field] of type [long] in document with id '0'. Preview of field's value: 'asdf'",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "For input string: \"asdf\""
}
}
}
}
]
}