[cmd/telemetrygen] Flag `--allow-export-failures` has inconsistent behavior across signal types.

[bogdan-st]

Component(s)

cmd/telemetrygen

What happened?

Description

For traces, --allow-export-failures is accepted but has no effect. Traces use the SDK's SpanProcessor which handles export errors internally, so the worker never sees them. Traces always continue on failure regardless of the flag.

For logs and metrics in non-batch mode, it works as expected. When an export fails, logger.Fatal is called and the process crashes, unless --allow-export-failures is set.

For logs and metrics in batch mode, the flushBuffer() function always logs the error and continues. It never calls logger.Fatal, so the flag is effectively ignored.

Steps to Reproduce

Run telemetrygen logs in batch mode against a non-existent endpoint:

telemetrygen logs --otlp-insecure --otlp-endpoint localhost:19999 --logs 3 --batch=true --batch-size 2

The process continues running after the export fails (only logs an ERROR). Compare with non-batch mode:

telemetrygen logs --otlp-insecure --otlp-endpoint localhost:19999 --logs 3 --batch=false

This one crashes with FATAL as expected.

Expected Result

When --allow-export-failures is not set and an export fails.

Actual Result

The process should crash regardless of whether batching is enabled or not when sending to a non-existing endpoint.

Collector version

v0.148.0

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• cmd/telemetrygen: @mx-psi @codeboten @Erog38 @bogdan-st

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bogdan-st]

@mx-psi @codeboten @Erog38 Heya, any thoughts on refactoring telemetrygen to use the collector's otlpexporter and have the workers build the pdata directly? Seems like there are some inconsistencies in the sdk for each signal type (for our usecase mainly batching) and i would love to have all signals be the same, for the most part. I can draft a pr for just one signal and work from there but i wanted your opinion.

[iw4p]

Hi @bogdan-st ! I’m interested in helping with this.

Before I start, should this be fixed as a targeted behavior-consistency bug in the current implementation, or do you want to address it only as part of a broader refactor? If a scoped fix is acceptable, I’d be happy to take a look..

[bogdan-st]

Hi @iw4p , I think we should stick to a targeted approach(if there is one) until the other maintainers agree with any kind of refactor, and it would be nice to have a fix.
Buuut since we re using the SDK's spanprocessor that handles exports for tracecs, im not sure there is an easy fix here.

[mx-psi]

@mx-psi @codeboten @Erog38 Heya, any thoughts on refactoring telemetrygen to use the collector's otlpexporter and have the workers build the pdata directly? Seems like there are some inconsistencies in the sdk for each signal type (for our usecase mainly batching) and i would love to have all signals be the same, for the most part. I can draft a pr for just one signal and work from there but i wanted your opinion.

My main concern with this would be that the data we generate would be less 'realistic'. I wonder what @open-telemetry/go-maintainers think about these inconsistencies.

[dmathieu]

I agree that keeping data realistic is better, and therefore using an SDK's exporter makes more sense.

One counterpoint is the support for profiles in telemetrygen, which hasn't been done so far because there is no profiling SDK.
I don't think switching all signals just for the sake of profiles makes sense though.

[alliasgher]

Opened #47651 for this.