Support multiple collection intervals within a receiver

[sv-splunk]

Component(s)

No response

Is your feature request related to a problem? Please describe.

Currently, OpenTelemetry receivers support configuring only a single collection interval per receiver instance. All configured data categories—such as infrastructure metrics, events, and similar data—are collected using this same interval.

receivers:
  sqlserver:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s
    metrics:
      sqlserver.database.count:
        enabled: true
    query_sample_collection:
      max_rows_per_query: 100
    top_query_collection:
      top_query_count: 1000
    events:
      db.server.top_query:
        enabled: true
      db.server.query_sample:
        enabled: true

In practice, considering the database receivers for example, have different categories often with very different optimal collection frequencies.

• Infra structure metrics - 60 Seconds
• top_query_collection - 60 Seconds
• query_sample_collection - 10 Seconds

With the current configuration model, each receiver instance supports only a single collection_interval, which forces all data categories to be collected at the same frequency. While it is possible to configure an additional receiver instance—for example, solely for query_sample_collection with a shorter collection interval—this results in significant configuration duplication and added complexity.

At present, there is an alternative approach implemented specifically for top_query_collection, where a separate collection_interval can be configured and the top query collection is skipped until that interval has elapsed. However, this approach still does not allow configuring a shorter interval than the global collection interval (Please see below config as an example).

receivers:
  sqlserver:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s     <------- global collection_interval
    metrics:
      sqlserver.database.count:
        enabled: true
    query_sample_collection:
      max_rows_per_query: 100
    top_query_collection:
      collection_interval: 60s       <------- separate collection interval for top_query
      top_query_count: 1000
    events:
      db.server.top_query:
        enabled: true
      db.server.query_sample:
        enabled: true

Describe the solution you'd like

We need to be able to define separate collection intervals for each type. I would propose a solution like below, where introduce a new section for collection_intervals to configure the values for each category,

receivers:
  sqlserver:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s                     << global collection_interval
    metrics:
      sqlserver.database.count:
        enabled: true
    query_sample_collection:
      max_rows_per_query: 100
    top_query_collection:
      top_query_count: 1000
    collection_intervals:                  <<
      metrics: 30s                         <<
      query_sample_collection: 10s         <<
      top_query_collection: 60s            <<
    events:
      db.server.top_query:
        enabled: true
      db.server.query_sample:
        enabled: true

Describe alternatives you've considered

Alternate solution that currently works is to define separate receiver instances with collection_interval that suite each type as below. This is a lot of duplication and added complexity.

receivers:
  sqlserver/metrics:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s
    metrics:
      sqlserver.database.count:
        enabled: true
 sqlserver/top_query:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 60s
    top_query_collection:
      top_query_count: 1000
    events:
      db.server.top_query:
        enabled: true
  sqlserver/samples:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 10s    
    query_sample_collection:
      max_rows_per_query: 100
    events:
       db.server.query_sample:
         enabled: true

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[iblancasa]

I think this can make sense depending on the receiver.

Also, receivers are not sharing this setting. So we would need to have one issue per "affected component" to study it individually. I will add the tag for the sqlserver receiver since you referred to that one in your examples.

[github-actions]

Pinging code owners for receiver/sqlserver: @sincejune @crobert-1. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triage' to set the priority and remove the needs-triage label.

[dmitryax]

I think this can be beneficial for all scrapers. It potentially can be implemented with mdatagen which generates configuration interface for each metric/event. We can add collection_interval as an additional optional field for every metric/event. The interface would look like this:

receivers:
  sqlserver:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s                     << global collection_interval
    metrics:
      sqlserver.database.count:
        enabled: true
    query_sample_collection:
      max_rows_per_query: 100
    top_query_collection:
      top_query_count: 1000
    events:
      db.server.top_query:
        enabled: true
      db.server.query_sample:
        enabled: true
        collection_interval: 60s            << override

[sv-splunk]

I think this can be beneficial for all scrapers. It potentially can be implemented with mdatagen which generates configuration interface for each metric/event. We can add collection_interval as an additional optional field for every metric/event. The interface would look like this:

receivers:
  sqlserver:
    username: {username}
    password: {password}
    port: 1433
    server: localhost
    collection_interval: 30s                     << global collection_interval
    metrics:
      sqlserver.database.count:
        enabled: true
    query_sample_collection:
      max_rows_per_query: 100
    top_query_collection:
      top_query_count: 1000
    events:
      db.server.top_query:
        enabled: true
      db.server.query_sample:
        enabled: true
        collection_interval: 60s            << override

@dmitryax agree 💯, whatever fits better with the general pattern for implementation.