Skip to content

Commit 5d1479d

Browse files
mikefatmikefat
committed
Update secretfilter docs with gitleaks implementation
1 parent 4172e1c commit 5d1479d

1 file changed

Lines changed: 22 additions & 76 deletions

File tree

‎docs/sources/reference/components/loki/loki.secretfilter.md‎

Lines changed: 22 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -40,76 +40,26 @@ loki.secretfilter "<LABEL>" {
4040

4141
You can use the following arguments with `loki.secretfilter`:
4242

43-
| Name | Type | Description | Default | Required |
44-
| ----------------- | -------------------- | -------------------------------------------------------------- | ---------------------------------- | -------- |
45-
| `forward_to` | `list(LogsReceiver)` | List of receivers to send log entries to. | | yes |
46-
| `allowlist` | `list(string)` | List of regular expressions to allowlist matching secrets. | `[]` | no |
47-
| `enable_entropy` | `bool` | Enable entropy-based filtering. | `false` | no |
48-
| `gitleaks_config` | `string` | Path to the custom `gitleaks.toml` file. | Embedded Gitleaks file | no |
49-
| `include_generic` | `bool` | Include the generic API key rule. | `false` | no |
50-
| `origin_label` | `string` | Loki label to use for the `secrets_redacted_by_origin` metric. | `""` | no |
51-
| `partial_mask` | `int` | Show the first N characters of the secret. | `0` | no |
52-
| `redact_with` | `string` | String to use to redact secrets. | `"<REDACTED-SECRET:$SECRET_NAME>"` | no |
53-
| `types` | `list(string)` | List of secret types to look for. | All types | no |
54-
55-
56-
The `gitleaks_config` argument is the path to the custom `gitleaks.toml` file.
57-
If you don't provide the path to a custom configuration file, the Gitleaks configuration file [embedded in the component][embedded-config] is used.
43+
| Name | Type | Description | Default | Required |
44+
| ---------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | -------- |
45+
| `forward_to` | `list(LogsReceiver)` | List of receivers to send log entries to. | | yes |
46+
| `config_path` | `string` | Path to a custom Gitleaks TOML config file. If empty, the default Gitleaks config is used. | `""` | no |
47+
| `origin_label` | `string` | Loki label to use for the `secrets_redacted_by_origin` metric. If empty, that metric is not registered. | `""` | no |
48+
| `redact_with` | `string` | Template for the redaction placeholder. Use `$SECRET_NAME` and `$SECRET_HASH`. When set, percentage-based redaction is not used. | `"<REDACTED-SECRET:$SECRET_NAME>"` | no |
49+
| `redact_percent` | `uint` | When `redact_with` is not set: percent of the secret to redact (1–100). Shows leading (100-N)% + `"..."`; 100 = full `"REDACTED"`. 0 or unset defaults to 80. | `80` | no |
5850

59-
{{< admonition type="note" >}}
60-
This component doesn't support all the features of the Gitleaks configuration file.
61-
It only supports regular expression-based rules, `secretGroup`, and allowlist regular expressions. `regexTarget` only supports the default value `secret`.
62-
Other features such as `keywords`, `paths`, and `stopwords` aren't supported.
63-
The `extend` feature isn't supported.
64-
If you use a custom configuration file, you must include all the rules you want to use within the configuration file.
65-
Unsupported fields and values in the configuration file are ignored.
66-
{{< /admonition >}}
51+
The `config_path` argument is the path to a custom [Gitleaks TOML config file][gitleaks-config]. The file supports the standard Gitleaks structure (rules, allowlists, and `[extend]` to extend the default config). If `config_path` is empty, the component uses the default Gitleaks configuration [embedded in the component][embedded-config].
6752

6853
{{< admonition type="note" >}}
69-
The embedded configuration file may change between {{< param "PRODUCT_NAME" >}} versions.
70-
To ensure consistency, use an external configuration file.
54+
The default configuration may change between {{< param "PRODUCT_NAME" >}} versions. For consistent behavior, use an external configuration file via `config_path`.
7155
{{< /admonition >}}
7256

73-
The `types` argument is a list of secret types to look for.
74-
The values provided are used as prefixes to match rules IDs in the Gitleaks configuration.
75-
For example, providing the type `grafana` matches the rules `grafana-api-key`, `grafana-cloud-api-token`, and `grafana-service-account-token`.
76-
If you don't provide this argument, all rules are used.
57+
**Redaction behavior:**
7758

78-
{{< admonition type="note" >}}
79-
Configuring this argument with the secret types you want to look for is strongly recommended.
80-
If you don't, the component looks for all known types, which is resource-intensive.
81-
{{< /admonition >}}
82-
83-
{{< admonition type="caution" >}}
84-
Some secret types in the Gitleaks configuration file rely on regular expression patterns that don't detect the secret itself but rather the context around it.
85-
For example, the `aws-access-token` type detects AWS key IDs, not the keys themselves.
86-
This is because the keys don't have a unique pattern that can easily be detected with a regular expression.
87-
As a result, with this secret type enabled, the component redacts key IDs but not actual secret keys.
88-
This behavior is consistent with the Gitleaks redaction feature but may not be what you expect.
89-
Currently, the secret types known to have this behavior are: `aws-access-token`.
90-
{{< /admonition >}}
59+
- If `redact_with` is set, it is used as the replacement string for every detected secret. Supported placeholders: `$SECRET_NAME` (rule ID) and `$SECRET_HASH` (SHA1 hash of the secret).
60+
- If `redact_with` is not set, redaction is percentage-based (Gitleaks-style): `redact_percent` controls how much of the secret is redacted. For example, `80` shows the first 20% of the secret followed by `"..."`; `100` replaces the entire secret with `"REDACTED"`. When `redact_percent` is 0 or unset, 80% redaction is used.
9161

92-
The `redact_with` argument is a string that can use variables such as `$SECRET_NAME`, replaced with the matching secret type, and `$SECRET_HASH`, replaced with the SHA1 hash of the secret.
93-
94-
The `include_generic` argument is a boolean that enables the generic API key rule in the Gitleaks configuration file if set to `true`.
95-
It's disabled by default because it can generate false positives.
96-
Consider enabling entropy-based filtering if you enable this rule.
97-
98-
The `allowlist` argument is a list of regular expressions to allow matching secrets.
99-
A secret won't be redacted if it matches any of the regular expressions. The allowlist in the Gitleaks configuration file is also applied.
100-
101-
The `partial_mask` argument is the number of characters to show from the beginning of the secret before the redact string is added.
102-
If set to `0`, the entire secret is redacted.
103-
If a secret isn't at least 6 characters long, it's entirely redacted.
104-
For short secrets, at most half of the secret is shown.
105-
106-
The `enable_entropy` argument enables entropy-based filtering.
107-
When you set this to `true`, the entropy of the detected potential secret is calculated and compared against the threshold provided for the matching rule in the configuration file.
108-
The secret is then redacted only if the entropy is above the threshold.
109-
This can help reduce false positives.
110-
111-
The `origin_label` argument specifies which Loki label value to use for the `secrets_redacted_by_origin` metric.
112-
This metric tracks how many secrets were redacted in logs from different sources or environments.
62+
**Origin metric:** The `origin_label` argument specifies which Loki label to use for the `secrets_redacted_by_origin` metric, so you can track how many secrets were redacted per source or environment.
11363

11464
[embedded-config]: https://github.com/grafana/alloy/blob/{{< param "ALLOY_RELEASE" >}}/internal/component/loki/secretfilter/gitleaks.toml
11565

@@ -133,22 +83,16 @@ The following fields are exported and can be referenced by other components:
13383

13484
`loki.secretfilter` exposes the following Prometheus metrics:
13585

136-
| Name | Type | Description |
137-
| --------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------- |
138-
| `loki_secretfilter_processing_duration_seconds` | Summary | Summary of the time taken to process and redact logs in seconds. |
139-
| `loki_secretfilter_secrets_allowlisted_total` | Counter | Number of secrets that matched a rule but were in an allowlist, partitioned by source. |
140-
| `loki_secretfilter_secrets_redacted_by_origin` | Counter | Number of secrets redacted, partitioned by origin label value. |
141-
| `loki_secretfilter_secrets_redacted_by_rule_total` | Counter | Number of secrets redacted, partitioned by rule name. |
142-
| `loki_secretfilter_secrets_redacted_total` | Counter | Total number of secrets that have been redacted. |
143-
| `loki_secretfilter_secrets_skipped_entropy_by_rule_total` | Counter | Number of secrets that matched a rule but whose entropy was too low to be redacted, partitioned by rule name. |
144-
145-
The `origin_label` argument specifies which Loki label value to use for the `secrets_redacted_by_origin` metric.
146-
This metric tracks how many secrets were redacted in logs from different sources or environments.
86+
| Name | Type | Description |
87+
| ---------------------------------------------- | ------- | ----------------------------------------------------------------- |
88+
| `loki_secretfilter_processing_duration_seconds` | Summary | Time taken to process and redact logs, in seconds. |
89+
| `loki_secretfilter_secrets_redacted_total` | Counter | Total number of secrets redacted. |
90+
| `loki_secretfilter_secrets_redacted_by_rule_total` | Counter | Number of secrets redacted, partitioned by rule name. |
91+
| `loki_secretfilter_secrets_redacted_by_origin` | Counter | Number of secrets redacted, partitioned by origin label (when `origin_label` is set). |
14792

14893
## Example
14994

150-
This example shows how to use `loki.secretfilter` to redact secrets from log lines before forwarding them to a Loki receiver.
151-
It uses a custom redaction string that includes the secret type and its hash.
95+
This example shows how to use `loki.secretfilter` to redact secrets from log lines before forwarding them to a Loki receiver. It uses a custom redaction template with `$SECRET_NAME` and `$SECRET_HASH`. You can instead omit `redact_with` to use percentage-based redaction (default 80% redacted), set `redact_percent` (e.g. `100` for full redaction), or set `config_path` to point to a custom Gitleaks TOML file.
15296

15397
```alloy
15498
local.file_match "local_logs" {
@@ -163,6 +107,8 @@ loki.source.file "local_logs" {
163107
loki.secretfilter "secret_filter" {
164108
forward_to = [loki.write.local_loki.receiver]
165109
redact_with = "<ALLOY-REDACTED-SECRET:$SECRET_NAME:$SECRET_HASH>"
110+
// optional: config_path = "/etc/alloy/gitleaks.toml"
111+
// optional: redact_percent = 100 // use when redact_with is not set
166112
}
167113
168114
loki.write "local_loki" {

0 commit comments

Comments
 (0)