opentelemetry-demo failed to index metrics

[carsonip]

APM Server version (apm-server version): 8.15

Description of the problem including expected versus actual behavior:

Sending opentelemetry demo data to apm-server causes an error in apm-server.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. run open-telemtery/opentelemetry-demo, add exporter to export otlphttp to apm-server 8.15
2. check apm-server logs
3. see apm-server error logs

Provide logs (if relevant):

{"log.level":"error","@timestamp":"2024-09-10T13:16:56.049+0100","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.1/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2024.09.10-000001' (document_parsing_exception): [1:824] failed to parse field [redis.commands] of type [double] in document with id 'Grbd25EB293cGPB_h0Uh'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

[simitt]

It would be good to get this fixed soon but I am not aware of this issue blocking anything, so don't see it critical for this iteration.

[raultorrecilla]

thanks @simitt, moving it forward then to it-108

[ericywl]

Update: Managed to reproduce this issue in 8.15.5 and 8.17.2, will look into the root cause.

8.15.5

2025-02-24 16:49:53 {"log.level":"error","@timestamp":"2025-02-24T08:49:53.200Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.1/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000001' (document_parsing_exception): [1:779] failed to parse field [redis.commands] of type [double] in document with id 'obwmN5UBXPKckzJvBMQr'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

8.17.2

2025-02-24 16:43:53 {"log.level":"error","@timestamp":"2025-02-24T08:43:53.375Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.3.1/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000003' (document_parsing_exception): ","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

[ericywl]

@carsonip I have found the issue, it seems to be due to some incompatibility with redisreceiver and Elasticsearch.

redisreceiver has these 2 metrics:

• redis.commands (Gauge)
• redis.commands.processed (Sum)

When ES receives redis.commands.processed, it constructs mapping as:

{
    "redis": {
        "commands": {
            "processed": 123
        }
    }
}

but this is incompatible with redis.commands which is a double, hence the error. There are no more errors when either redis.commands or redis.commands.processed is disabled.

Side note: One other perculiar thing is that redis.version is a resource label and shows up in ES as labels.redis_version instead of redis.version because apm-data converts it to redis_version, causing ES to not recognize it as under redis.

[inge4pres]

When ES receives redis.commands.processed, it constructs mapping as:

A question: is the mapping dynamic? You mention that the field is sent as an integer (or long) but then we expect a double in the mapping?

i dont understand if the error is originating from the mapping expecting a double and receiving an integer or vice versa.

[carsonip]

but this is incompatible with redis.commands which is a double, hence the error. There are no more errors when either redis.commands or redis.commands.processed is disabled.

Seems to be our old friend subobjects: https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html

APM data streams don't set subobjects: false but it is required for metric names like redis.commands vs redis.commands.processed. Attributes, on the other hand, are handled differently as you've noticed - we replace all dots . with underscores _ in labels

@simitt should we just ignore it and say that metrics from redisreceiver is not compatible with apm server? I believe it is not worth fixing at this point of time, and it is more like a known limitation. I suggest that we document in public docs using this issue, adjacent to #15509

1. the logged error e.g. failed to parse field [redis.commands] of type [double] in document with id
2. how to workaround it by adding a processor to rename the metric in a collector.

Side note: One other perculiar thing is that redis.version is a resource label and shows up in ES as labels.redis_version instead of redis.version because apm-data converts it to redis_version, causing ES to not recognize it as under redis

As explained, this is expected. The task to document it is #15509

[ericywl]

@inge4pres My bad, let me clarify. The mapping in ES is dynamic. Both redis.commands and redis.commands.processed are integers on the receiver side, but I believe ES treats them as doubles.

Anyways, ES receives indexing request for redis.commands.processed = 123 and indexes it as

{
    "redis": {
        "commands": {
            "processed": 123
        }
    }
}

But it also receives redis.commands = 456 and tries to index it

{
    "redis": {
        "commands": 456
    }
}

Since, redis.commands is already an object from the first step, it cannot index the integer / double 456 from the second step.

EDIT: Seems like Carson replied about ES subobjects above while I was typing this.

[simitt]

@simitt should we just ignore it and say that metrics from redisreceiver is not compatible with apm server? I believe it is not worth fixing at this point of time, and it is more like a known limitation. I suggest that we document in public docs using this issue, adjacent to #15509

Users should be able to work around it by creating a dynamic custom component template with a dedicated mapping and setting subobjects: false.

The apm-data templates are defined in https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/apm-data/src/main/resources/index-templates. The base index templates link to many component templates. Some of these are defined by the elastic apm solution, but some are custom component templates, which do not exist by default. These can be leveraged by customers to define their own custom mappings, and are what we usually suggest as solution.

The problem here is that the otel demo ideally should work out of the box, without any customization. @carsonip @ericywl is this only a problem when pointing to the apm-server or MIS or also when pointing the demo to EDOT/managed Otel on serverless?

@mlunadia @akhileshpok for visibility and input on this topic and prioritization.

[ericywl]

The problem here is that the otel demo ideally should work out of the box, without any customization. @carsonip @ericywl is this only a problem when pointing to the apm-server or MIS or also when pointing the demo to EDOT/managed Otel on serverless?

@simitt I tried running the demo with serverless from https://github.com/elastic/opentelemetry-demo (instead of the original OTel one with local APM server). It works fine because it does not use redisreceiver.

[simitt]

Thanks @ericywl ; IMO we can close this issue then as a won't fix.
But still want to hear @akhileshpok and @mlunadia 's feedback if they are ok with this approach.