OSDOCS-14601 [NETOBSERV-2053] Flowlogs Flp filtering

modules/network-observability-config-flows-at-ingestion.adoc

@@ -0,0 +1,76 @@
+// Module included in the following assemblies:
+
+// * networking/network_observability/configuring-operators.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="network-observability-config-flows-at-ingestion_{context}"]
+
+= Filtering network flows at ingestion
+
+You can create filters to reduce the number of generated network flows. Filtering network flows can help reduce the resource usage of the Network Observability Operator.
+
+There are two different kinds of filters that you can configure:
+
+[id="ebpf-agent-filters_{context}"]
+== eBPF agent filters
+
+eBPF agent filters maximize performance because they take effect at the earliest stage of the Network Observability Operator process.
+
+To configure eBPF agent filters with the Network Observability Operator, see "Filtering eBPF flow data using multiple rules".
+
+[id="flowlogs-pipeline-filters_{context}"]
+== Flowlogs-pipeline filters
+
+Flowlogs-pipeline filters provide greater control over traffic selection because they take effect later in the Network Observability process. They are primarily used to improve data storage.
+
+[id="flowlogs-pipeline-filters-query-language_{context}"]
+=== Flowlogs-pipeline filters query language
+
+Flowlogs-pipeline filters use a simple query language to filter network flows:
+
+[source]
+----
+(srcnamespace="netobserv" OR (srcnamespace="ingress" AND dstnamespace="netobserv")) AND srckind!="service"
+----
+
+The query language uses the following syntax:
+
+Logical boolean operators (not case-sensitive)::
+* `and`
+* `or`
+Comparison operators::
+* equals ``=`
+* not equals `!=`
+* matches regexp `=~`
+* not matches regexp `!~`
+* less than (or equal) `<` / `<=`
+* greater than (or equal) `>` / `>=`
+Unary operations::
+* field is present: `with(field)`
+* field is absent: `without(field)`
+Parenthesis-based priority::
+{empty}
+
+[id="flowlogs-pipeline-configure_{context}"]
+=== Flowlogs-pipeline configuration
+You can configure flowlogs-pipeline filters in the `spec.processor.filters` parameter of the `FlowCollector` resource. For example:
+
+.Example YAML Flowlogs-pipeline filter
+[source,yaml]
+----
+apiVersion: flows.netobserv.io/v1beta2
+kind: FlowCollector
+metadata:
+  name: cluster
+spec:
+  namespace: netobserv
+  agent:
+  processor:
+    filters:
+      - query: |
+          (SrcK8S_Namespace="netobserv" OR (SrcK8S_Namespace="openshift-ingress" AND DstK8S_Namespace="netobserv"))
+        outputTarget: Loki  <1>
+        sampling: 10  <2>
+----
+<1> Sends matching flows to a specific output, such as Loki, Prometheus, or an external system.
+<2> Optional. Applies a sampling ratio to limit the number of flows that match the query.

observability/network_observability/configuring-operator.adoc

@@ -24,6 +24,12 @@ include::modules/network-observability-enriched-flows.adoc[leveloffset=+1]
 * xref:../../observability/network_observability/json-flows-format-reference.adoc#network-observability-flows-format_json_reference[Network flows format reference].
 
 include::modules/network-observability-configuring-FLP-sampling.adoc[leveloffset=+1]
+include::modules/network-observability-config-flows-at-ingestion.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../network_observability/observing-network-traffic.adoc#network-observability-filtering-ebpf-rule_nw-observe-network-traffic[Filtering eBPF flow data using multiple rules]
+
 include::modules/network-observability-configuring-quickfilters-flowcollector.adoc[leveloffset=+1]
 include::modules/network-observability-resource-recommendations.adoc[leveloffset=+1]
 include::modules/network-observability-resources-table.adoc[leveloffset=+2]