New component: k8slog receiver

[h0cheung]

The purpose and use-cases of the new component

The existing filelog receiver can find and collect logs in files based on configured path expressions.

However, for k8s containers, we need a special path to access the files in the container. It is not convenient to match the corresponding path directly, and difficult to get its metadata or filter it.

The sidecar solution brings additional problems such as deployment troubles and wasted resources. At present, there is no solution in otel that meets our needs in all aspects.

So we decided to develop a new component: k8slog receiver, to provide full functionality to collect logs in k8s containers. It is an all-in-one solution, which supports to:

• collect logs from files inside k8s containers via daemonset
• support docker and cri-containerd
• support many docker graph drivers / snapshotters
• collect logs from stdout of k8s containers via daemonset
• filter containers by metadata
• extract metadata of containers
• collect file logs in k8s containers via sidecar

If we want to collect logs from k8s, we can use this component.

Example configuration for the component

receivers:
  k8slog:
    discovery:
      mode: 1 # choices:
              # 1 (default): collect logs from files inside containers via daemonset
              # 2: collect logs from stdout of containers via daemonset
              # 3: collect logs from files via sidecar
      host_root: /host-root # default. we need to mount / of host in the container as a subpath. Read-only is enough.
      runtime_apis: # APIs to be used for discovering pods. By default it will try to connect to both docker and cri-containerd.
        type: docker
        addr: unix:///<host_root>/var/run/docker.sock # default. address of docker api.
        containerd_addr: unix:///<host_root>/run/docker/containerd/containerd.sock # default. address of underlying containerd api.
      filter: # filters for pods. By default all containers will be included.
        annotations:
        labels:
        namespaces:
        containers:
        pods:
        uids:
        expr: # custom expression (using antonmedv/expr)
      extract: # extrace resource attributes, like k8sattributes processor.
        metadata:
        env:
        annotations:
        labels:
    # configs below are (almost) the same as filelog receiver.
    include: # In `daemonset-file` mode, these paths are paths inside the contaienrs. In `daemonset-stdout` mode, these paths is useless.
      - /var/log/*.log
    exclude:
    start_at:
    multiline:
    force_flush_period:
    encoding:
    include_file_name:
    poll_interval:
    fingerprint_size:
    max_log_size:
    max_concurrent_files:
    attributes:
    resource:
    operators:
    storage:
    preserve_leading_whitespaces:
    preserve_trailing_whitespaces:
    max_batches:
    header:
    retry_on_failure:

Telemetry data types supported

logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

@TylerHelmuth

Additional context

No response

[TylerHelmuth]

@h0cheung I will sponsor this component.

A couple comments on the proposal:

• lets not introduce expr at the start. If we need custom expression I would prefer to use OTTL, but we could introduce it later.
• for mode, lets you strings instead of integers as they make it easier for users reading configuration to understand what the component is doing.
• Do we definitely need to include configuration option if we already have the filter configuration section? Could we combine them in some way?
• I think it would be nice to nest the filelogreceiver configuration options under another section instead of placing them all in root.

[djaglowski]

I generally think this is a good idea but strongly believe we need to depend on the fileconsumer package for tracking and reading log files. It appears this is the intention, but I'd like to understand what if any differences you foresee.

configs below are (almost) the same as filelog receiver.

Can you clarify in which ways it is different? Is it only in the way the include paths are interpretted?

When reading from files, to what extent should users be able to apply operators? If there is some pre-configured set of operators that handles parsing, would a user be able to change these? Would they be able to place operators before and/or after the pre-defined operators?

[TylerHelmuth]

@djaglowski I agree that we should use fileconsumer.

I believe this component will pre-configure some operators based on kubernetes requirements. Since this is an opinionated receiver, I don't think those should be configurable (at least not the operators themselves - the proposal is using a different interface to configure how the resource attributes should be set and I like separating this config from the operator itself).

Users should be able to add operators after the component's pre-determined manipulation/setup.

[h0cheung]

@djaglowski @TylerHelmuth

As planned, I will write this component from scratch. Some of the configuration items are the same as filelog, this is to make it easier for users to get started, it does not mean that the same implementation is used. There are several reasons for this:

• fileconsumer currently has some imperfections, such as [pkg/stanza/fileconsumer] Add ability to read files asynchronously #23056 tries to address. It's not easy to make large changes for already released components, but for new components I want to do as good a job as possible from the start.
• For symbolic links, we need special handling based on the mount point mapping when looking for files.
• In daemonset-stdout mode, the file path of the logs is completely defined, no glob is needed. So we can manage the reader directly after getting the file path, no need for resource consuming mechanisms like fingerprint, just handle the log rotation.

For efficiency and flexibility, parsing the k8s standard output and splicing containerd logs (based on "T" and "F" tags) will be implemented in separate code. There are no preconfigured operators.

[djaglowski]

As planned, I will write this component from scratch. Some of the configuration items are the same as filelog, this is to make it easier for users to get started, it does not mean that the same implementation is used.

fileconsumer currently has some imperfections, such as #23056 tries to address. It's not easy to make large changes for already released components, but for new components I want to do as good a job as possible from the start.

I strongly disagree with the idea of writing a completely separate implementation for file tracking and ingestion. We should address imperfections in pkg/stanza/fileconsumer rather than maintaining two separate implementations. For the particular issue mentioned here, this is a performance enhancement for an edge case - hardly a reason to reroll a major package and maintain two implementations.

For symbolic links, we need special handling based on the mount point mapping when looking for files.

I recommend we define an interface or function that can be passed to fileconsumer to customize behavior for handling symlinks. We can have a default behavior and another that is substituted by k8slog and perhaps reused by other components in the future.

In daemonset-stdout mode, the file path of the logs is completely defined, no glob is needed. So we can manage the reader directly after getting the file path, no need for resource consuming mechanisms like fingerprint, just handle the log rotation.

Similarly, I recommend we define an interface for finding files. We already have a matcher package which fully encapsulates the finding and filtering of files. It should be possible to substitute another implementation if necessary.

[TylerHelmuth]

I agree with @djaglowski.