Implement receiver/processor/exporter "group" components

[tigrannajaryan]

Is your feature request related to a problem? Please describe.

Collector configuration requires listing receivers at least in 2 places: once in the top-level receivers section and one more time the receivers setting of the pipeline.

This gives flexibility to use the same receiver in more than one pipeline, but complicates the most common use case when the receiver is used in one pipeline only.

Every time a new receiver needs to be added, it needs to be added in 2 places in the config, which is annoying and error prone (and prevents an interesting use case described below).

Describe the solution you'd like

I would like to have a new group receiver component that can instantiate other receivers. The config for example can look like this:

receivers:
  group/all:
    hostmetrics: ...
    otlp: ....
    redis: ...

service:
  pipelines:
    metrics:
      receivers: [group/all]
      processors: ...
      exporters: ...

With a config like this if I need to add or remove a receiver I only need to do it in one place, under the group/all section.

Functionality the group component will be a significantly simplified version of the existing receivercreator component.

The exact same problem applies to processors and exporters for which equivalent processor group and exporter group component would be also highly desirable.

Additional context

Here is a use case.

We can break down Collector config file into smaller, more manageable config files, where receivers, processors and exporters are in their own files. For example:

# this is the main config file

receivers:
  group/all: ${file: receivers.yaml}

processors:
  group/all: ${file: processors.yaml}

exporters:
  group/all: ${file: exporters.yaml}

service:
  pipelines:
    metrics:
      receivers: [group/all]
      processors: [group/all]
      exporters: [group/all]

# this is receivers.yaml

hostmetrics: ...

otlp: ...

redis: ...

# this is exporters.yaml

otlphttp:
  endpoint: ...

[Aneurysm9]

I like the concept. Would it be possible to achieve this as a config convertor that rewrites the pipeline configuration to expand a set of components? That could make it a zero-cost abstraction at runtime with low startup overhead.

[tigrannajaryan]

I like the concept. Would it be possible to achieve this as a config convertor that rewrites the pipeline configuration to expand a set of components? That could make it a zero-cost abstraction at runtime with low startup overhead.

This is zero-cost abstraction for the receivergroup and a tiny cost for processorgroup and exportergroup (just one for loop going over the list of nested component and a single extra function call per nested component - I think it will be negligible). We can benchmark, but my gut feeling is that the a straightforward, trivial implementation should be good enough.

[atoulme]

I think this is a great idea. I believe @rmfitzpatrick implemented this approach in our config to allow to create config modules. We can port over/adopt this approach.

[djaglowski]

I like the idea a lot. What do you think about using group as the component name?

receivers:
  group/all:
    hostmetrics: ...
    otlp: ....
    redis: ...

service:
  pipelines:
    metrics:
      receivers: [group/all]
      processors: ...
      exporters: ...

[tigrannajaryan]

I like the idea a lot. What do you think about using group as the component name?

I like the shorter name.

Can we have a component named group which is a receiver, a processor or an exporter depending on the context it is used in?

[djaglowski]

Can we have a component named group which is a receiver, a processor or an exporter depending on the context it is used in?

Yes, all are always in context of one type (receiver/processor/exporter). We have existing examples of overlap such as otlp, though of course this is only receiver/exporter, but the same concept applies to processors.

[tigrannajaryan]

I updated the PR title/description to use group as the component name.

[bogdandrutu]

Lets avoiding jumping to the implementation details Converter/Proper Component/etc. I think the idea is very good and helpful. Let'a agree for the configuration.

For some reasons I don't like the name "group". It does not signal to me that this is just a "config grouping". I have some questions, maybe that will clarify:

• Can we reference a receiver from a group in a pipeline or only the whole group?
• For processors, if we don't allow individual references then what is the order of execution?