New component: Pipeline processor

[jmacd]

The purpose and use-cases of the new component

The Pipeline processor incorporates a number of standard pipeline management features. These features are typically associated with exporters via the core exporterhelper package, however there are occasional use-cases for incorporating these as features in a processor. The yaml field names are identical to the OTLP exporter in core:

• Queue function (queue_sender)
• Batch function (batcher)
• Retry function (retry_on_failure)
• Timeout (timeout, squashed)

Example configuration for the component

The features are already documented. These fields are identical to the OTLP exporter.

type Config struct {
	exporterhelper.TimeoutConfig `mapstructure:",squash"`
	exporterhelper.QueueConfig   `mapstructure:"sending_queue"`
	RetryConfig                  configretry.BackOffConfig `mapstructure:"retry_on_failure"`
	BatcherConfig exporterbatcher.Config `mapstructure:"batcher"`
}

Default settings will be disabled for queue, retry, and batcher. Timeout functionality, like exporterhelper, will be unconditional.

Telemetry data types supported

We wish to use the exporterhelper's observability sending logic, however this deserves a little investigation. The processorhelper also supports metrics, and we could suppress the exporterhelper's metrics in favor of the processorhelper's variations.

Code Owner(s)

jmacd,drewrelmas

Sponsor (optional)

No response

Additional context

Supporting evidence

This feature is meant to provide a path to removing the processor/batchprocessor component from the core, while replacing it for users that would like batching, queuing, retry, and timeout inside a pipeline as opposed to in an exporter.

Here are some associated issues and PRs that provide motivation for this new component.

Describing how batching support moved to the exporterhelper, because it enables protocol-specific batching: open-telemetry/opentelemetry-collector#4646

Noting that the batch processor does not meet many requirements that are now satisfied by the exporterhelper-based batcher, the new component will use better logic than the former batchprocessor: open-telemetry/opentelemetry-collector#11308

More detail on this topic, in a PR that I plan to re-open after the recent dust settles: open-telemetry/opentelemetry-collector#11947

A similar request presented in a recent Collector SIG: #35803

Implementation approach

The exporterhelper code base will be re-used, making this component relatively simple. The only pieces that will be needed are the logic for batching from pdata object into pdata object, which is logic that exists in a least one other place: the original batch processor (in core) includes pdata-to-pdata batching/splitting logic which should be re-used.

Summarizing, this component should amount to little more than exporterhelper pieces and batching/splitting logic copied from batchprocessor with new tests and some attention to the question about metrics/observability stated above.

[akats7]

This works for me, to confirm for the connector use case is there a general consensus that we want to go with this approach over modifying the connector interface. I do like that it helps keep connectors simple.

If this has the sponsorship required I’d be happy to help contribute this.