Skip to content

Commit 0a9283c

Browse files
Add persist_resource_version documentation to k8sobjects receiver README
- Added persist_resource_version to example configurations - Documented persist_resource_version configuration property - Added dedicated "ResourceVersion Persistence" section with: - Key features and benefits - Complete configuration example with storage extension - Storage key format explanation - Deployment considerations with PVC example - Updated ConfigMap and Deployment examples to include persistence setup Co-Authored-By: Dhruv Shah <dhruv.shah@sumologic.com>
1 parent fd52ea9 commit 0a9283c

1 file changed

Lines changed: 118 additions & 3 deletions

File tree

receiver/k8sobjectsreceiver/README.md

Lines changed: 118 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ The following is example configuration
2626
```yaml
2727
k8sobjects:
2828
auth_type: serviceAccount
29+
storage: file_storage
2930
k8s_leader_elector: k8s_leader_elector
3031
objects:
3132
- name: pods
@@ -39,6 +40,7 @@ The following is example configuration
3940
mode: watch
4041
group: events.k8s.io
4142
namespaces: [default]
43+
persist_resource_version: true
4244
```
4345
4446
Brief description of configuration properties:
@@ -67,8 +69,8 @@ use this config to specify the group to select. By default, it will select the f
6769
For example, `events` resource is available in both `v1` and `events.k8s.io/v1` APIGroup. In
6870
this case, it will select `v1` by default.
6971
- `k8s_leader_elector` (default: none): if specified, will enable Leader Election by using `k8sleaderelector` extension
70-
- `storage` (default: none): if specified, will enable the receiver to store the last resource version seen in a file. The resource version will be
71-
persisted when the watch mode is configured for the given object.
72+
- `storage` (default: none): specifies the storage extension to use for persisting resourceVersions. Required when `persist_resource_version` is enabled.
73+
- `persist_resource_version` (default: `false`): When set to `true` (watch-mode only), the receiver persists the resourceVersion after processing each event. On restart, the receiver resumes watching from the persisted resourceVersion, preventing duplicate events. Requires a `storage` extension to be configured.
7274

7375

7476
The full list of settings exposed for this receiver are documented in [config.go](./config.go)
@@ -97,6 +99,90 @@ The `k8sobjectsreceiver` supports collecting a wide range of standard Kubernetes
9799

98100
This receiver supports both `pull` and `watch` modes, allowing for flexible and real-time monitoring of these objects. Please note that custom resources are supported only if their CRDs are available in the cluster.
99101

102+
### ResourceVersion Persistence
103+
104+
When using `watch` mode, the receiver can persist the resourceVersion after processing each event. This ensures that on collector restart, the receiver resumes from where it left off, preventing duplicate events from being collected.
105+
106+
#### Key Features
107+
- **Prevents Duplicate Events**: On restart, watches resume from the last persisted resourceVersion
108+
- **Per-Namespace Tracking**: Separate resourceVersions are maintained for each namespace
109+
- **410 Gone Error Recovery**: Automatically handles expired resourceVersions by fetching a fresh one
110+
- **Opt-in**: Disabled by default; enable via `persist_resource_version: true`
111+
112+
#### Configuration Example
113+
114+
To enable resourceVersion persistence, you need:
115+
1. A storage extension (e.g., `file_storage`)
116+
2. The `storage` field pointing to the extension
117+
3. `persist_resource_version: true` on objects you want to persist
118+
119+
```yaml
120+
extensions:
121+
file_storage:
122+
directory: /var/lib/otelcol/storage
123+
timeout: 1s
124+
125+
receivers:
126+
k8sobjects:
127+
auth_type: serviceAccount
128+
storage: file_storage # Reference to storage extension
129+
objects:
130+
- name: pods
131+
mode: watch
132+
namespaces: [default, production]
133+
persist_resource_version: true # Enable persistence
134+
- name: events
135+
mode: watch
136+
group: events.k8s.io
137+
persist_resource_version: true
138+
139+
exporters:
140+
debug:
141+
verbosity: detailed
142+
143+
service:
144+
extensions: [file_storage] # Must include the extension
145+
pipelines:
146+
logs:
147+
receivers: [k8sobjects]
148+
exporters: [debug]
149+
```
150+
151+
#### Storage Key Format
152+
153+
The receiver stores resourceVersions with the following key format:
154+
- **Namespace-specific watches**: `latestResourceVersion/{objectType}.{namespace}`
155+
- Example: `latestResourceVersion/pods.default`, `latestResourceVersion/events.production`
156+
- **Cluster-wide watches**: `latestResourceVersion/{objectType}`
157+
- Example: `latestResourceVersion/nodes`, `latestResourceVersion/namespaces`
158+
159+
#### Deployment Considerations
160+
161+
When using persistence:
162+
- Ensure the storage directory is mounted to a persistent volume so data survives pod restarts
163+
- The storage extension must be listed in the `service.extensions` array
164+
- Only applicable for `watch` mode; `pull` mode does not use persistence
165+
166+
```yaml
167+
apiVersion: apps/v1
168+
kind: Deployment
169+
metadata:
170+
name: otelcontribcol
171+
spec:
172+
template:
173+
spec:
174+
containers:
175+
- name: otelcontribcol
176+
image: otel/opentelemetry-collector-contrib:latest
177+
volumeMounts:
178+
- name: storage
179+
mountPath: /var/lib/otelcol/storage
180+
volumes:
181+
- name: storage
182+
persistentVolumeClaim:
183+
claimName: otel-storage-pvc
184+
```
185+
100186
### Configuration
101187

102188
Create a ConfigMap with the config for `otelcontribcol`. Replace `OTLP_ENDPOINT`
@@ -112,20 +198,29 @@ metadata:
112198
app: otelcontribcol
113199
data:
114200
config.yaml: |
201+
extensions:
202+
file_storage:
203+
directory: /var/lib/otelcol/storage
204+
115205
receivers:
116206
k8sobjects:
207+
auth_type: serviceAccount
208+
storage: file_storage
117209
objects:
118210
- name: pods
119211
mode: pull
120212
- name: events
121213
mode: watch
214+
persist_resource_version: true
215+
122216
exporters:
123217
otlp_grpc:
124218
endpoint: <OTLP_ENDPOINT>
125219
tls:
126220
insecure: true
127221
128222
service:
223+
extensions: [file_storage]
129224
pipelines:
130225
logs:
131226
receivers: [k8sobjects]
@@ -212,7 +307,22 @@ Create a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/
212307
Note: This receiver must be deployed as one replica, otherwise it'll be producing duplicated data.
213308

214309
```bash
215-
<<EOF | kubectl apply -f -
310+
# First, create a PersistentVolumeClaim for storage (if using persistence)
311+
cat <<EOF | kubectl apply -f -
312+
apiVersion: v1
313+
kind: PersistentVolumeClaim
314+
metadata:
315+
name: otel-storage-pvc
316+
spec:
317+
accessModes:
318+
- ReadWriteOnce
319+
resources:
320+
requests:
321+
storage: 1Gi
322+
EOF
323+
324+
# Then create the Deployment
325+
cat <<EOF | kubectl apply -f -
216326
apiVersion: apps/v1
217327
kind: Deployment
218328
metadata:
@@ -239,11 +349,16 @@ spec:
239349
volumeMounts:
240350
- name: config
241351
mountPath: /etc/config
352+
- name: storage
353+
mountPath: /var/lib/otelcol/storage
242354
imagePullPolicy: IfNotPresent
243355
volumes:
244356
- name: config
245357
configMap:
246358
name: otelcontribcol
359+
- name: storage
360+
persistentVolumeClaim:
361+
claimName: otel-storage-pvc
247362
EOF
248363
```
249364

0 commit comments

Comments
 (0)