Add resource attributes to Datadog OtelCollector payload #5287
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…open-telemetry#43738) This PR contains the following updates: | Package | Change | Age | Confidence | |---|---|---|---| | [github.com/twmb/franz-go/pkg/kadm](https://redirect.github.com/twmb/franz-go) | `v1.17.0` -> `v1.17.1` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>twmb/franz-go (github.com/twmb/franz-go/pkg/kadm)</summary> ### [`v1.17.1`](https://redirect.github.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v1171) [Compare Source](https://redirect.github.com/twmb/franz-go/compare/v1.17.0...v1.17.1) \=== This patch release fixes four bugs (two are fixed in one commit), contains two internal improvements, and adds two other minor changes. #### Bug fixes - If you were using the `MaxBufferedBytes` option and ever hit the max, odds are likely that you would experience a deadlock eventually. That has been fixed. - If you ever produced a record with no topic field and without using `DefaultProduceTopic`, or if you produced a transactional record while not in a transaction, AND if the client was at the maximum buffered records, odds are you would eventually deadlock. This has been fixed. - It was previously not possible to set lz4 compression levels. - There was a data race on a boolean field if a produce request was being written at the same time a metadata update happened, *and* if the metadata update has an error on the topic or partition that is actively being written. Note that the race was unlikely and if you experienced it, you would have noticed an OutOfOrderSequenceNumber error. See [this comment](https://redirect.github.com/twmb/franz-go/pull/781#issuecomment-2248563542) for more details. #### Improvements - Canceling the context you pass to `Produce` now propagates in two more areas: the initial `InitProducerID` request that occurs the first time you produce, and if the client is internally backing off due to a produce request failure. Note that there is no guarantee on *which* context is used for cancelation if you produce many records, and the client does *not* allow canceling if it is currently unsafe to do so. However, this does mean that if your cluster is somewhat down such that `InitProducerID` is failing on your new client, you can now actually cause the `Produce` to quit. See [this comment](https://redirect.github.com/twmb/franz-go/issues/769#issuecomment-2235707417) for what it means for a record to be "safe" to fail. - The client now ignores aborted records while consuming only if you have configured `FetchIsolationLevel(ReadCommitted())`. Previously, the client relied entirely on the `FetchResponse` `AbortedTransactions` field, but it's possible that brokers could send aborted transactions even when not using read committed. Specifically, this was a behavior difference in Redpanda, and the KIP that introduced transactions and all relevant documents do not mention what the broker behavior actually should be here. Redpanda itself was also changed to not send aborted transactions when using read committed, but we may as well improve franz-go as well. - Decompression now better reuses buffers under the hood, reducing allocations. - Brokers that return preferred replicas to fetch from now causes an info level log in the client. #### Relevant commits - [`305d8dc`](https://redirect.github.com/twmb/franz-go/commit/305d8dc) kgo: allow record ctx cancelation to propagate a bit more - [`24fbb0f`](https://redirect.github.com/twmb/franz-go/commit/24fbb0f) **bugfix** kgo: fix deadlock in Produce when using MaxBufferedBytes - [`1827add`](https://redirect.github.com/twmb/franz-go/commit/1827add) **bugfix** kgo sink: fix read/write race for recBatch.canFailFromLoadErrs - [`d7ea2c3`](https://redirect.github.com/twmb/franz-go/commit/d7ea2c3) **bugfix** fix setting lz4 compression levels (thanks [@&open-telemetry#8203;asg0451](https://redirect.github.com/asg0451)!) - [`5809dec`](https://redirect.github.com/twmb/franz-go/commit/5809dec) optimise: use byteBuffer pool in decompression (thanks [@&open-telemetry#8203;kalbhor](https://redirect.github.com/kalbhor)!) - [`cda897d`](https://redirect.github.com/twmb/franz-go/commit/cda897d) kgo: add log for preferred replicas - [`e62b402`](https://redirect.github.com/twmb/franz-go/commit/e62b402) **improvement** kgo sink: do not back off on certain edge case - [`9e32bf9`](https://redirect.github.com/twmb/franz-go/commit/9e32bf9) kgo: ignore aborted txns if using `READ_UNCOMMITTED` </details> --- ### Configuration 📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xNTYuMSIsInVwZGF0ZWRJblZlciI6IjQxLjE1Ni4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==--> --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: otelbot <[email protected]>
…-telemetry#43721) <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43719 Signed-off-by: Israel Blancas <[email protected]>
…ibutes (open-telemetry#41859) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description The current count connector can only set attributes from span, log record, or similar sources. This PR enhances that functionality. With this update, the count connector will set attributes in the following precedence order: - Span (log record, data point, profile) attributes - Scope attributes - Resource attributes
The package contained a single function and was triggering lint failures due to its name. Moved the func into the metrics package instead. Signed-off-by: alex boten <[email protected]>
…try#43652) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Adds integration test for ICMP Check Receiver that pings container host and collects metrics. Test setup lets any group open the socket so the integration test can send real ICMP probes. If the `sysctl` can’t be applied, the test skips to avoid CI failures. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue [open-telemetry#29009](open-telemetry#29009) <!--Describe what testing was performed and which tests were added.--> #### Testing - New integration test for ICMP Check Receiver <!--Please delete paragraphs that you did not use before submitting.-->
…s log events. (open-telemetry#43616) #### Description Add a support to ingest AWS Network Firewall Logs. Log types are: alert, flow (`netflow`) and TLS. #### Link to tracking issue Fixes open-telemetry#43642 #### Testing For now unit tests with: ``` cd extension/encoding/awslogsencodingextension && go test -v ./... ``` #### Documentation
…pen-telemetry#43410) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This is another fix for open-telemetry#43044 (oh, never ending). This change fixes the following error: ``` --- FAIL: BenchmarkFromPdataConverter/worker_count=1 frompdataconverter_test.go:153: Error Trace: /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/adapter/frompdataconverter_test.go:153 /opt/homebrew/Cellar/go/1.24.4/libexec/src/testing/benchmark.go:219 /opt/homebrew/Cellar/go/1.24.4/libexec/src/testing/benchmark.go:245 /opt/homebrew/Cellar/go/1.24.4/libexec/src/runtime/asm_arm64.s:1223 Error: "[0x140f228bec0]" should have 250000 item(s), but has 1 Test: BenchmarkFromPdataConverter/worker_count=1 ``` - Removes `require.Len(b, entries, 250_000)`, line 153, as the test fails here. There's nothing related to `250_000` in the test cases, and it looks like outdated. The channel receives entry slices that only contain a single element, and it's because the inputs(`plog.Logs`) have only a single log record for each. Maybe we should add more tests that cover multi-log-records cases separately if needed - Fixes misuse of`ResetTimer` as it should not be called in`b.Loop()` - Eliminates a timeout check since I think someone who runs the benchmark should notice something's going wrong if It takes too long. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43044 <!--Describe what testing was performed and which tests were added.--> #### Testing ``` cd pkg/stanza/adapter go test -bench='^BenchmarkFromPdata' ```
…#43733) Fix `nil` dereference panic when using the clickhouse exporter with the `clickhouse.json` feature gate enabled. #### Description The recent refactoring in open-telemetry#43518 introduced a variable shadowing bug: ```go exp := newLogsExporter(set.Logger, c) startOpt := exporterhelper.WithStart(exp.start) if featureGateJSON.IsEnabled() { // here is the bug, below line creates a new variable, in the scope of the if // the outer `exp` variable still points to the newLogsExporter exp := newLogsJSONExporter(set.Logger, c) // without this line here we even would have gotten a compiler error that exp is unused startOpt = exporterhelper.WithStart(exp.start) } return exporterhelper.NewLogs( ctx, set, cfg, // this always points to the newLogsExporter, and not as intended to the // JSON exporter in case the feature gate is set exp.pushLogsData, // ... ``` Since `exp.start` is never called for that, this results in the following panic during runtime, because `exp.db` is `nil`. ``` "}, "otelcol.component.id": "otlp/hyperdx", "otelcol.component.kind": "receiver", "endpoint": "[::]:4318"} 2025-10-22T13:53:30.234Z info [email protected]/service.go:245 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "1e79a701-9f1d-4434-94f8-5154dfe70543", "service.name": "otelcol-contrib", "service.version": "0.138.0"}} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1deb510] goroutine 199 [running]: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhouseexporter.(*tracesExporter).pushTraceData(0x4000fc5ef0, {0xbac9cc8?, 0x4000a12150?}, {0x4000565a28?, 0x40016c8e68?}) github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter_traces.go:71 +0x50 go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces(0x4000a12160?, {0xbac9cc8?, 0x4000a12150?}, {0x4000565a28?, 0x40016c8e68?}) go.opentelemetry.io/collector/[email protected]/traces.go:27 +0x44 ``` My fix here was to define a common interface for both exporters, so that the `exp` variable can be re-assigned from within the `if` scopes. Alternatively, also the refactoring could be reverted, or both `exp.pushLogsData` and `exp.shutdown` could be assigned to a helper variable similar to what is done currently already with `exp.start` --------- Co-authored-by: Christos Markou <[email protected]>
…pen-telemetry#43703) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description From [issue](open-telemetry#41289) the AspNet OTEL instrumentation was upgraded to use semantic version 1.24.0. This became a breaking change for the azure monitor exporter that was stuck on 1.12.0/1.27.0/1.7.0 semantic versions. This PR updates the azure monitor exporter to 1.34.0 <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#41289 <!--Describe what testing was performed and which tests were added.--> #### Testing * Updated conventions_test.go to handle client and server / breaking api changes * Updated trace_to_envelope_test.go to handle the split of client and server / breaking api changes * Followed the local [testing steps](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/d97fdbfc606ed2a24450261c8f4df9b585a28f23/CONTRIBUTING.md) * Ran the collector local with my config to confirm data was flowing into azure app insights <img width="1108" height="89" alt="image" src="https://github.com/user-attachments/assets/9e9161f5-98d5-43a3-b7a4-31a1422a11fb" />
…ry#43191) This PR contains the following updates: | Package | Change | Age | Confidence | |---|---|---|---| | [github.com/leodido/go-syslog/v4](https://redirect.github.com/leodido/go-syslog) | `v4.2.0` -> `v4.3.0` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>leodido/go-syslog (github.com/leodido/go-syslog/v4)</summary> ### [`v4.3.0`](https://redirect.github.com/leodido/go-syslog/releases/tag/v4.3.0) [Compare Source](https://redirect.github.com/leodido/go-syslog/compare/v4.2.0...v4.3.0) <!-- Release notes generated using configuration in .github/release.yml at v4.3.0 --> #### What's Changed ##### Features & Enhancements 🎉 - feat: optionally accept time second fractions for rfc3164 timestamps by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;25](https://redirect.github.com/leodido/go-syslog/pull/25) - feat(octetcounting): improve handling (safer) of max length by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;29](https://redirect.github.com/leodido/go-syslog/pull/29) - feat(rfc3164): extend RFC3164 parser to support Cisco IOS logs by [@&open-telemetry#8203;alistairking](https://redirect.github.com/alistairking) and [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;22](https://redirect.github.com/leodido/go-syslog/pull/22) - thanks to [@&open-telemetry#8203;go3ranh](https://redirect.github.com/go3ranh) too - feat(octetcounting,nontransparent): allow passing down machine options from nontransparent/octetcounting parsers by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;23](https://redirect.github.com/leodido/go-syslog/pull/23) - feat(octetcounting,nontransparent): restore backward compatibility while keeping `WithMachineOptions` by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;36](https://redirect.github.com/leodido/go-syslog/pull/36) ##### Bug Fixes 🪤 - fix(common): properly escape multi-byte Unicode characters in [#&open-telemetry#8203;20](https://redirect.github.com/leodido/go-syslog/pull/20) ##### Other Changes 🫶 - ci: pin gh action versions by [@&open-telemetry#8203;step-security-bot](https://redirect.github.com/step-security-bot) in [#&open-telemetry#8203;26](https://redirect.github.com/leodido/go-syslog/pull/26) - ci: fine-tune release gh token by [@&open-telemetry#8203;step-security-bot](https://redirect.github.com/step-security-bot) in [#&open-telemetry#8203;27](https://redirect.github.com/leodido/go-syslog/pull/27) - ci: create sast workflow by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;28](https://redirect.github.com/leodido/go-syslog/pull/28) - ci: read-only permissions for sast workflow by [@&open-telemetry#8203;step-security-bot](https://redirect.github.com/step-security-bot) in [#&open-telemetry#8203;30](https://redirect.github.com/leodido/go-syslog/pull/30) - docs(rfc3164): document Cisco IOS extensions for RFC 3164 by [@&open-telemetry#8203;leodido](https://redirect.github.com/leodido) in [#&open-telemetry#8203;34](https://redirect.github.com/leodido/go-syslog/pull/34) #### New Contributors - [@&open-telemetry#8203;step-security-bot](https://redirect.github.com/step-security-bot) made their first contribution in [#&open-telemetry#8203;26](https://redirect.github.com/leodido/go-syslog/pull/26) - [@&open-telemetry#8203;alistairking](https://redirect.github.com/alistairking) made their first contribution in [#&open-telemetry#8203;22](https://redirect.github.com/leodido/go-syslog/pull/22) **Full Changelog**: <leodido/go-syslog@v4.2.0...v4.3.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xMzEuOSIsInVwZGF0ZWRJblZlciI6IjQxLjEzMS45IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==--> --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: otelbot <[email protected]> Co-authored-by: Yang Song <[email protected]> Co-authored-by: Andrzej Stencel <[email protected]>
…f datapoint attributes (open-telemetry#43751) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description - Changed `net.peer.ip` and `net.peer.name` to resource attributes - Included previously omitted metric for RTT Standard Deviation (`ping.rtt.stddev`) <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue open-telemetry#29009 <!--Describe what testing was performed and which tests were added.--> #### Testing - adjusted scraper unit tests - adjust integration test <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.-->
…les (open-telemetry#42843) Follow up to open-telemetry#40548 FYI: @atoulme @felixge @aalexand TODO: - [ ] verify result - [ ] more tests --------- Signed-off-by: Florian Lehner <[email protected]> Co-authored-by: Roger Coll <[email protected]>
…emetry#43745) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR fixes the issue described at open-telemetry#43730 where nested configurations are not unmarshaled properly and hence they cannot be loaded later by the components: ``` 2025-10-22T14:11:01.323Z info [email protected]/observerhandler.go:206 starting receiver {"resource": {"service.instance.id": "a282b5ed-2088-41ab-8b31-e4c38209791d", "service.name": "otelcol-contrib", "service.version": "0.138.0"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080", "endpoint": "10.1.1.86:8080", "endpoint_id": "k8s_observer/49541bff-1b96-443d-a3fa-6190d23609a7/http-metrics(8080)", "config": {"config":{"scrape_configs":[{"job_name":"prometheus_metrics","metrics_path":"/metrics","scrape_interval":"10s","static_configs":[{"targets":["`endpoint`"]}]}]},"trim_metric_suffixes":true}} 2025-10-22T14:11:01.324Z error [email protected]/observerhandler.go:222 failed to start receiver {"resource": {"service.instance.id": "a282b5ed-2088-41ab-8b31-e4c38209791d", "service.name": "otelcol-contrib", "service.version": "0.138.0"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "receiver": "prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080", "error": "failed to load \"prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080\" template config: decoding failed due to the following error(s):\n\n'config' has invalid keys: scrape_configs"} github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).startReceiver github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/observerhandler.go:222 github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).OnAdd github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/observerhandler.go:93 github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/endpointswatcher.(*EndpointsWatcher).updateAndNotifyOfEndpoints github.com/open-telemetry/opentelemetry-collector-contrib/extension/[email protected]/endpointswatcher/endpointswatcher.go:115 ``` The fix loads the string map coming from the yaml into a `confmap.Conf`, which is later unmarshaled properly into a `userConfigMap{}` which is `map[string]any`. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43730 <!--Describe what testing was performed and which tests were added.--> #### Testing 1. Tuned unit-tests 2. Tested manually with the steps of the reported issue **Deploy the helm chart using the a custom image** (https://hub.docker.com/r/christosmarkou788/otelcontribcol-dev_rc_annotation_fix) ```yaml mode: daemonset image: repository: christosmarkou788/otelcontribcol-dev_rc_annotation_fix tag: "latest" pullPolicy: IfNotPresent clusterRole: create: true rules: - apiGroups: - '' resources: - 'pods' - 'nodes' verbs: - 'get' - 'list' - 'watch' - apiGroups: [ "" ] resources: [ "nodes/proxy"] verbs: [ "get" ] - apiGroups: - "" resources: - nodes/stats verbs: - get - nonResourceURLs: - "/metrics" verbs: - get config: extensions: k8s_observer: auth_type: serviceAccount node: ${env:K8S_NODE_NAME} observe_nodes: true health_check: endpoint: ${env:MY_POD_IP}:13133 exporters: debug: verbosity: detailed receivers: receiver_creator/metrics: watch_observers: [k8s_observer] discovery: enabled: true receivers: service: extensions: [health_check, k8s_observer] telemetry: logs: level: INFO pipelines: metrics: receivers: [ receiver_creator/metrics ] processors: [ batch ] exporters: [ debug ] ``` **deploy a prometheus target** ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: node-exporter name: node-exporter spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter annotations: io.opentelemetry.discovery.metrics.9100/enabled: "true" io.opentelemetry.discovery.metrics.9100/scraper: prometheus io.opentelemetry.discovery.metrics.9100/config: | trim_metric_suffixes: true config: scrape_configs: - job_name: 'prometheus_metrics' scrape_interval: 10s metrics_path: /metrics static_configs: - targets: - '`endpoint`' spec: containers: - args: - --path.sysfs=/host/sys - --path.rootfs=/host/root - --no-collector.wifi - --no-collector.hwmon - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/) - --collector.netclass.ignored-devices=^(veth.*)$ name: node-exporter image: prom/node-exporter ports: - containerPort: 9100 protocol: TCP resources: limits: cpu: 250m memory: 180Mi requests: cpu: 102m memory: 180Mi volumeMounts: - mountPath: /host/sys mountPropagation: HostToContainer name: sys readOnly: true - mountPath: /host/root mountPropagation: HostToContainer name: root readOnly: true volumes: - hostPath: path: /sys name: sys - hostPath: path: / name: root ``` **Verify that discovery succeeds and the prometheus receiver starts collecting metrics** ```console 2025-10-22T13:57:09.644Z info [email protected]/observerhandler.go:206 starting receiver {"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100", "endpoint": "10.244.0.46:9100", "endpoint_id": "k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)", "config": {"config":{"scrape_configs":[{"job_name":"prometheus_metrics","metrics_path":"/metrics","scrape_interval":"10s","static_configs":[{"targets":["10.244.0.46:9100"]}]}]},"trim_metric_suffixes":true}} 2025-10-22T13:57:09.649Z info [email protected]/metrics_receiver.go:157 Starting discovery manager {"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)"} 2025-10-22T13:57:09.650Z info targetallocator/manager.go:219 Scrape job added {"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)", "jobName": "prometheus_metrics"} 2025-10-22T13:57:09.650Z info [email protected]/metrics_receiver.go:216 Starting scrape manager {"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)"} ``` Signed-off-by: ChrsMark <[email protected]>
…iver oauth2 flow (open-telemetry#43596) #### Description As part of our project we needed Pulsar Exporter to work with oauth2 based IAM. But during configuration it was found that there is no proper way to provide `client_secret` to the exporter. Going through the Pulsar Client code, I found the keys which are needed to add and updated the config of Pulsar Exporter and Receiver accordingly. Changes - Added `scope` config field to accept Oauth2 scope value - Added `private_key` config field to accept path to a private client credentials json file #### Testing Tested with Pulsar Cluster which is configured to work with OIDC Auth only in Kubernetes. Created the following k8s secret and mounted it inside the pod and configured the `private_key` path to it. ```json { "type": "client_credentials", "client_id": "pulsar-client", "client_secret": "<REDACTED>", "issuer_url": "https://example.com/oauth2" } ``` Config ```yaml pulsar: auth: oauth2: audience: default client_id: pulsar-client issuer_url: https://example.com/oauth2 scope: default private_key: /etc/secrets/auth.json ``` Able to successfully connect with Pulsar Cluster with this. NOTE: If there is any error in Oauth2 flow, then the error is ignored. This is because Pulsar Client drops the error and does not return it during the `NewAuthenticationOAuth2` flow. --------- Co-authored-by: Antoine Toulme <[email protected]>
…lemetry#43568) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add system scraper implementation for the Cisco OS receiver to collect device availability and connection metrics from Cisco devices. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue open-telemetry#42647 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests for system scraper config, factory, and scraping logic - All tests passing in receiver/ciscoosreceiver module - Repository checks passing: make golint, make gotest - Live SSH testing validated against real Cisco IOS-XE device <!--Describe the documentation added.--> #### Documentation - Updated main receiver metadata.yaml with new metrics - Generated metrics builders and config code via mdatagen - Factory integration with proper scraper registration <!--Please delete paragraphs that you did not use before submitting.-->
…en-telemetry#43760) I'm not involved in development of those components, so removing myself
This PR updates the opentelemetry-collector modules to open-telemetry/opentelemetry-collector@8038c69 --------- Signed-off-by: otelbot <[email protected]> Co-authored-by: otelbot <[email protected]>
…ry#43767) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Fixes typo in the OpAMP Extension docs for configuration options. Co-authored-by: Alex Boten <[email protected]>
#### Description open-telemetry#42805 moved Datadog's E2E tests into `internal/e2e`, and the example configurations used as part of the tests into `internal/e2e/examples`. The intent was to move the tests into `internal/datadog/e2e` instead, but it looks like this change was accidentally reverted during a force push. Moreover, it seems that moving the example configs created some broken links (eg. [here, "collector.yaml"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datadogexporter#datadog-exporter), [here, "example configuration"](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter/deploy/?tab=gateway#kubernetes)). This PR: - moves `internal/e2e` to `internal/datadog/e2e` - moves the example configs back to their original location to restore these links
#### Description Adds support to the parsing of CloudTrail digest log [^1] through CloudTrail unmarshaler. Table below highlight the digest fields to OTel log attributed mapping (extracted from updated documentation), | CloudTrail digest field | Attribute in OpenTelemetry log | |--------------------------|---------------------------------------------------| | awsAccountId | cloud.account.id | | digestEndTime | aws.cloudtrail.digest.end_time | | digestS3Bucket | aws.cloudtrail.digest.s3_bucket | | digestS3Object | aws.cloudtrail.digest.s3_object | | newestEventTime | aws.cloudtrail.digest.newest_event | | oldestEventTime | aws.cloudtrail.digest.oldest_event | | previousDigestS3Bucket | aws.cloudtrail.digest.previous_s3_bucket | | previousDigestS3Object | aws.cloudtrail.digest.previous_s3_object | | logFiles.s3Bucket | aws.cloudtrail.digest.log_files.s3_bucket | | logFiles.s3Object | aws.cloudtrail.digest.log_files.s3_bucket | | logFiles.newestEventTime | aws.cloudtrail.digest.log_files.newest_event_time | | logFiles.oldestEventTime | aws.cloudtrail.digest.log_files.oldest_event_time | Note that I have not included the digest signature, algorithm, and related hash values, given usage of these requires extra metadata that's only present at the S3 object [^2]. But having unmarshaled digest file information (ex:- bucket, object and timing) allows consuming system to implement custom validation workflows. `DigestStartTime` is used as the log record timestamp. #### Link to tracking issue Fixes open-telemetry#43403 #### Testing I have added unit tests for the digest file parsing #### Documentation I have updated the documentation with the resulting fields [^1]: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-validation-digest-file-structure.html [^2]: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-custom-validation.html --------- Signed-off-by: Kavindu Dodanduwa <[email protected]>
…GrokPatterns functions (open-telemetry#43728) <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Part of open-telemetry#43555 Signed-off-by: odubajDT <[email protected]>
…lace_all_matches() and replace_match() functions (open-telemetry#43747) <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Part of open-telemetry#43555 Signed-off-by: odubajDT <[email protected]>
… replicaset name (open-telemetry#42534) Fixes open-telemetry#42530 <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Extract deployment name from replicaset name in the pod owner reference. Doesn't need to start the informer for the replicaset, reduce traffic to api server. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#42530 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit test, and test in kind cluster that trace still has container name attribute. <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Tyler Helmuth <[email protected]> Co-authored-by: Jina Jain <[email protected]>
…-telemetry#43764) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Sets the agent capabilities on the OpAMP client with `client.SetCapabilities()` instead of the deprecated `settings.Capabilities`. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43763 <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.-->
Removes the internal utils package that only included a func used in one place. Signed-off-by: alex boten <[email protected]>
…emetry#43762) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Updates the OpAMP [Supervisor Configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor/specification#supervisor-configuration) for `capabilities` to the currently available options and their default values. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43761 <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation Updated OpAMP Supervisor Configuration docs <!--Please delete paragraphs that you did not use before submitting.-->
…ults (open-telemetry#42686) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Kafkaexporter uses the [default backoff config](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/1573d65f57dc36222b51d7440ede200d32ed8403/exporter/kafkaexporter/factory.go#L52) from the `configretry` module; however, the [defaults for config retry](https://github.com/open-telemetry/opentelemetry-collector/blob/d87c6ce3a89614fd7d30719b22682586b18796c7/config/configretry/backoff.go#L14-L23) don't match what is documented in kafkaexporter. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue N/A <!--Describe what testing was performed and which tests were added.--> #### Testing N/A <!--Describe the documentation added.--> #### Documentation Updated
…emetry#43718) #### Description Update the tests to explicitly set otelcol.Factories.Telemetry. This will soon be a required field. #### Link to tracking issue Relates to open-telemetry/opentelemetry-collector#14003 #### Testing Tests pass. #### Documentation N/A
…pen-telemetry#43789) Reduce the API of the component, hiding most of the structs. This is required for checkapi upgrade v0.28.1 as it picks up on the structs.
Both k8s and prometheus libraries have to be updated because they depend on each other and both have breaking changes. The following files changed to adopt to the API breaking changes: - processor/k8sattributesprocessor/internal/kube/fake_informer.go: - kubernetes/kubernetes#126387 - receiver/prometheusreceiver/metrics_receiver.go: - prometheus/prometheus#16257 - receiver/prometheusremotewritereceiver/receiver.go: - prometheus/prometheus#17160 I haven't touched any usages of deprecated k8s API to keep the changeset small. Created a separate issue for that instead open-telemetry#43891 Some prometheus tests are failing due to behavioral change in prometheus library. I skipped them for now and reported open-telemetry#43893
…y#43909) I was trying to understand why we get failures in open-telemetry#42947 and it is unclear so far. I am enabling debug logs to see if it can help when the failure happens next time.
open-telemetry#43914) #### Description Makes the behavior of strategies consistent: they do not adjust metrics which already have a valid start time set (and aren't a true-reset point). #### Link to tracking issue Part of open-telemetry#43739 #### Testing Added and updated unit tests. cc @eriksywu @ridwanmsharif @braydonk
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Fixed a concurrency bug in the Prometheus remote write receiver where concurrent requests with identical job/instance labels would return empty responses after the first successful request. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#42159 <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Signed-off-by: perebaj <[email protected]> Co-authored-by: Vihas Makwana <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]>
<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43778 Signed-off-by: Israel Blancas <[email protected]>
to unblock collector core dependency upgrade
Swap args in assertions. Signed-off-by: alex boten <[email protected]>
Add more configuration to prepare for the upgrade.
Signed-off-by: alex boten <[email protected]>
Both functions are used for benchmarks in the module. They do not need to be exposed as API.
…-telemetry#43946) Add missing stability level to splunk.license.expiration.seconds_remaining metric to unblock core upgrade
#### Description Move the metricstarttime processor to beta. We have been using it in production, and have fixed issues that have come up. At this point, we don't see needing any breaking chagnes, so it is time to move it to beta, and replace the adjustment in the prometheus receiver. #### Link to tracking issue Part of open-telemetry#43656 @ridwanmsharif
…3925) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description In the dependency upgrade open-telemetry#43890 there was the PR prometheus/prometheus#16429 which changed the provisions for staleness tracking. Now the code only does the tracking if the series was successfully appended in storage. This is indicated by a non zero storage reference returned by the appender. Since we used to return 0 in all cases, the staleness tracking is now broken. Solution is to return a fake reference of 1, just to indicate success. Alternative solution would be to make a unique reference for each series label set, but that incurs an overhead which we can avoid. Performance impact: probably small as the cache should be stable over time - unless targets churn a lot between receiver instances. Resource utilization: +1 hashmap with series identifier (name+labels) pointing to cache entries. Size related to number of series. `addRef` here: https://github.com/prometheus/prometheus/blob/c8f1de18a7870e5dc99ed50119c194691c41e41c/scrape/scrape.go#L1089 <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#43893 <!--Describe what testing was performed and which tests were added.--> #### Testing Un-skip related tests. --------- Signed-off-by: György Krajcsovits <[email protected]>
…43906) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Some tests set the global registry to enable native histograms, which is not safe for t.Parallel(). Fix this by only using the feature gate in the receiver factory. This makes TestNativeVsClassicHistogramScrapeViaProtobuf 4x faster on my laptop. Signed-off-by: György Krajcsovits <[email protected]>
…#43920) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Refactors the Cisco OS receiver connection handling: <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue open-telemetry#42647 Fixes - Centralized connection establishment logic into shared connection factory - Removed duplicate validation checks across receiver and connection layers - Simplified scraper nil checks and updated related tests - Cleaned up linting issues (unused imports, context usage) <!--Describe what testing was performed and which tests were added.--> #### Testing - Connection factory tests for password and key file authentication - Scraper initialization and shutdown tests added <!--Describe the documentation added.--> #### Documentation - Simplified inline documentation throughout connection handling code <!--Please delete paragraphs that you did not use before submitting.-->
…metry#43953) to unblock core collector dependency upgrade
…-map (open-telemetry#43949) The problems happens when assume inside "attributes" there is a key called "tags" that is a pcommon.Map. Here is an example: ``` delete_key(attributes[tags], "test_key") where metric.name = "foo" ``` Because of the changes to always call set value on the target (see open-telemetry#42350) we will get to do a SetValue where the src and destination are the same, but because we always set the destination to emptyMap will do a full copy of the sub-map tags. Signed-off-by: Bogdan Drutu <[email protected]>
…#43879) #### Description This PR updates all dependencies on the datadog-agent repo to a recent pseudo-version past: - [this PR](DataDog/datadog-agent#41710), which should hopefully fix "ambiguous import" issues caused by googleapis/go-genproto#1015 - [this PR](DataDog/datadog-agent#42268), which should fix open-telemetry#43594
#### Description I forgot to add a release note for some of the behavior changes introduced as part of the dependency upgrade in open-telemetry#43879.
replaces open-telemetry#43955 --------- Signed-off-by: otelbot <[email protected]> Co-authored-by: otelbot <[email protected]>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Mark [ciscoosreceiver] as Alpha. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue open-telemetry#42647 Fixes <!--Describe what testing was performed and which tests were added.--> #### Testing Ran `make generate` and `make genotelcontribcol`. All tests pass. <!--Describe the documentation added.--> #### Documentation README status section auto-generated via `make generate`. <!--Please delete paragraphs that you did not use before submitting.-->
…_keys functions (open-telemetry#43729) Co-authored-by: Evan Bradley <[email protected]>
Several of the benchmarks were failing to run, this fixes them all for future use.
|
Bits AI Dev Agent Status: ✅ Done Status History (1 entries)You can ask for changes by mentioning @DataDog in a comment. |
Co-authored-by: jackgopack4 <[email protected]>
jackgopack4
force-pushed
the
dd/add-resource-attributes-to-collector
branch
from
acba2c3 to
49e675d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR by Bits for Dev Agent Session
You can ask for changes by mentioning @DataDog in a comment.
Feedback (especially what can be better) welcome in #code-gen-feedback!
Description
This change adds support for collecting and exposing resource attributes from the OpenTelemetry Collector's TelemetrySettings to the Datadog extension payload. The
collector_resource_attributesfield is now populated in the OtelCollector struct as a list of strings in the format["key:value", "key:value", ...].The implementation:
set.Resource.Attributes()in the TelemetrySettingsinfo.resourceAttributesfieldThis follows the pattern used in
extension/opampextension/opamp_agent.gofor accessing and storing resource attributes.Link to tracking issue
Fixes
Testing
The changes maintain backward compatibility. The
collector_resource_attributesfield is initialized as an empty list if no resource attributes are present. Existing tests should continue to pass, and the new field will be properly serialized in the JSON payload.Documentation
The new
CollectorResourceAttributesfield in theOtelCollectorstruct is self-documenting through its struct tagjson:"collector_resource_attributes"and follows the existing payload structure conventions.