Adds resourceVersion support to k8sObject receiver

[dhruv-shah-sumo]

Description

Implements optional resourceVersion checkpointing to prevent duplicate events on collector restart.
Resource version persistence is developed as part of watch mechanism which is being used by k8sObjects receiver and k8sEvents receiver. The feature is not enabled for pull mode in any of the receivers.

Features:

• Opt-in via declaring storage extension for persistence
• Namespace-aware checkpointing: Creates per stream checkpoint when separate namespaces are mentioned. Per namespace keys are maintained in the storage. The format is latestResourceVersion/configmaps.kube-system for per namespace watch stream. In case namespaces are not specified, a global clusterwide watch stream is created that creates a different checkpoint key with format: latestResourceVersion/pods, latestResourceVersion/nodes
• Watch mode only (validated at config time)
• If persistence is enabled, then resource_version provided in the config will be ignored. The persisted version will be used to kick-off the watch stream.
• In case of stale persisted version, a List() API will be called to get the latest resource version available which is the existing way of handling stale resource version in k8s inventory.

Configuration:

  receivers:
    k8sobjects:
      storage: file_storage
      objects:
        - name: pods
          mode: watch

How it works:

1. At the start of each watch stream, getResourceVersion() is called which checks if persistence is enabled then it retrieves the persisted version.
   • If persisted version is empty then it calls List() API to get the list of available objects and picks the latest version and persists it.
   • If persisted version is non-empty then it is supplied to kick-off the watch stream.
   • In case, the persisted version supplied to start watch stream in the step above is stale then checkpoint is deleted. This prompts the List() API call again to get the latest available resource version for the given stream.
2. After processing each watch event, saves resourceVersion to storage.
3. In case persistence operation fails for any reason, the watch stream continues.

Link to tracking issue

Fixes
#46017

Testing

• Unit tests
• Manually built the updated image using make docker-otelcontribcol and deployed in local. Tested various restarts and 410 handling scenarios for global watch streams, per namespace watch streams.

Documentation

• Updated schema yaml files for k8sobject and k8sevents receiver.

Example - how the new config would look like?

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    timeout: 10s

receivers:
  k8sobjects:
    auth_type: serviceAccount
    storage: file_storage
    include_initial_state: true
    objects:
      - name: pods
        mode: watch
        namespaces: [default]
      - name: events
        mode: watch
        namespaces: [default]

exporters:
  nop:

service:
  telemetry:
    logs:
      level: debug
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [k8sobjects]
      exporters: [nop]

[github-actions]

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

• Please review our Contributing Guidelines.
• Don't forget to sign the Contributor License Agreement (CLA) if you haven't already.

A maintainer will review your pull request soon. Thank you for helping make OpenTelemetry better!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: dhruv-shah-sumo / name: Dhruv Shah (0705e53, 09ef108, 0ad8d4b, 0dbdd04, 1c166c0, 327e60f, 3cbe3f0, 3fd1164, 4538dce, 573b40c, 5a07726, 6828119, 74a3d2b, 809b419, 84ee801, 85edef4, 8693981, 93a8187, 9e92f03, a15da3a, dab4bdd, e20ade7, e5fa5a6, ec2223c, f27a8f0)
• ✅ login: dhruv-shah-sumo / name: dhruv.shah (4c19e4b)

[jmmcorreia]

@dhruv-shah-sumo I did a first pass on the PR and had a few comments and questions. I will try to do a second pass some time this week to check if I missed something, but already left my initial comments so you can take a look.

Hoping this will make review easier for codeowners if they can take a look at the PR at a later point.

[dhruv-shah-sumo]

@dhruv-shah-sumo I did a first pass on the PR and had a few comments and questions. I will try to do a second pass some time this week to check if I missed something, but already left my initial comments so you can take a look.

Hoping this will make review easier for codeowners if they can take a look at the PR at a later point.

Thanks a lot for comprehensive review comments @jmmcorreia . I'll address them as soon as possible.

[jmmcorreia]

Left a few extra comments. Code wise, other than the remaining comments, I would say it seems to be good.

Next pass I will check the UT in a bit more detail

[ChrsMark]

LGTM
Thank's @dhruv-shah-sumo!

** Needs a rebase :)

[ChrsMark]

@dhruv-shah-sumo main is broken. Mind taking 573b40c to a standalone PR to fix it first?

[dhruv-shah-sumo]

@dhruv-shah-sumo main is broken. Mind taking 573b40c to a standalone PR to fix it first?

#47759 Have tagged you on the PR.

[otelbot]

Thank you for your contribution @dhruv-shah-sumo! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey. If you are getting started contributing, you can also join the CNCF Slack channel #opentelemetry-new-contributors to ask for guidance and get help.