Skip to content

Commit 688930b

Browse files
committed
fix(prompt): generic matched-value precedence over downstream use
Generalise the 'judge the matched value, not the surrounding code' rule to any reference-or-expression match (bare variable, named-argument pass-through, field/attribute access, config/env/secrets lookup, call, or interpolation) across languages, and make it take precedence over how the value is used downstream. A bare-variable argument pass-through (token=token) was confirmed ~10% of the time under sampling non-determinism because the value flowed into an API call; the precedence rule dismisses it 40/40 while the confirm fixtures still confirm 30/30. Adds the kwarg_passthrough corpus fixture.
1 parent e758f85 commit 688930b

5 files changed

Lines changed: 67 additions & 15 deletions

File tree

CHANGELOG.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,24 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [S
44

55
## [Unreleased]
66

7+
### Changed
8+
9+
- Hardened the gemma4 system prompt so the "judge the matched value, not
10+
the surrounding code" rule explicitly takes precedence over downstream
11+
use, and generalised it to cover any reference-or-expression match
12+
(bare variable, named-argument pass-through, field/attribute access,
13+
config/env/secrets lookup, call, or interpolation) across languages,
14+
rather than the Python-shaped examples it carried before. The trigger
15+
was a `token=token` argument pass-through: the aggressive rule
16+
captures the bare variable `token`, and the model — acknowledging it
17+
was "a variable being passed to a function call" — still confirmed it
18+
~10% of the time (4/40 runs at conf 0.7-0.9) because the value flows
19+
into an API call. vLLM is non-deterministic even at temperature 0, so
20+
that tail surfaced as a spurious alert. With the precedence rule the
21+
same finding dismisses 40/40; the three confirm fixtures still confirm
22+
30/30, so recall is unaffected. New `kwarg_passthrough` corpus fixture
23+
guards it.
24+
725
## [0.5.0], 2026-06-18
826

927
### Fixed

internal/integration/testdata/diffs/code_expression.expect.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"description": "The aggressive gitleaks rule matches `access_token = token_incl_metadata.get(` because 'token' appears in the variable name and the captured value is a method call. The value is code, not a credential (the real token is fetched at runtime, never in the diff). The LLM should dismiss. Real-world false positive from an OIDC token-handling commit.",
2+
"description": "The aggressive gitleaks rule matches `access_token = token_incl_metadata.get(` because 'token' appears in the variable name and the captured value is a method call. The value is code, not a credential (the real token is fetched at runtime, never in the diff). The LLM should dismiss. Modelled on a real-world OIDC token-handling false positive.",
33
"min_after_dedup": 1,
44
"expectations": [
55
{ "match": "token_incl_metadata.get(", "verdict": "dismissed" }
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
diff --git a/services/objects/client.py b/services/objects/client.py
2+
index 1111111..2222222 100644
3+
--- a/services/objects/client.py
4+
+++ b/services/objects/client.py
5+
@@ -680,6 +680,14 @@ def fetch_object_properties(app, object_id):
6+
params = {"filter": build_filter(object_id)}
7+
+ token = app.settings["API_TOKEN"]
8+
+ base_url = app.settings["API_URLBASE"]
9+
+ response = http_get_with_retries(
10+
+ f"{base_url}/api/objects",
11+
+ params=params,
12+
+ token=token,
13+
+ )
14+
+ return response.json()["data"]
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"description": "The aggressive gitleaks rule fires twice on this snippet: on `token = app.settings[\"API_TOKEN\"]` (a config lookup) and on `token=token` (a keyword-argument pass-through, captured value `token,`). Neither is a literal credential. The pass-through is the dangerous one: its captured value is a bare variable reference, and before the precedence-rule prompt hardening the model confirmed it ~10% of the time under sampling non-determinism (it acknowledged the variable but confirmed on the 'used in an API call' cue). Both must dismiss.",
3+
"min_after_dedup": 2,
4+
"expectations": [
5+
{ "match": "app.settings[", "verdict": "dismissed" },
6+
{ "match": "token,", "verdict": "dismissed" }
7+
]
8+
}

prompts/gemma4_system.tmpl

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -18,20 +18,32 @@ your-key-here), documentation examples, and known vendor sentinels
1818
(AKIAIOSFODNN7EXAMPLE, ya29.dummy, sk-test-...).
1919

2020
Judge the matched value itself, not the words around it. The match must
21-
BE a literal credential to be confirmed. Dismiss when the matched value
22-
is code that references or produces a credential rather than the
23-
credential itself: a variable or attribute name, a function or method
24-
call, an environment-variable lookup, or a format/template string. For
25-
example `access_token = token_data.get("access_token")`,
26-
`password = config.get("db_password")`, `key = os.environ["API_KEY"]`,
27-
and `secret = f"{prefix}-{suffix}"` all assign a credential that is
28-
computed at runtime and never appears in the diff, so there is nothing
29-
to leak. A variable named after a secret is not a secret.
30-
31-
Confirm strings that ARE a credential of some format and appear at a
32-
call site that uses them: database connections, API client
33-
initialisation, request header construction, environment-variable
34-
defaults bound to production hostnames.
21+
BE a literal credential to be confirmed. This is language-independent.
22+
Dismiss when the matched value is a reference or expression that yields
23+
a credential at runtime rather than the literal credential itself. Such
24+
matches include: a bare variable or identifier; an argument or
25+
parameter passed through by name; a field, property, or attribute
26+
access; an index or key lookup into a config, settings, or secrets
27+
object; an environment-variable read; a function or method call; or a
28+
string interpolation, format, or template. These shapes recur across
29+
languages, for example `token=token`, `cfg.Token`, `config["TOKEN"]`,
30+
`process.env.API_KEY`, `os.environ["API_KEY"]`, `System.getenv("KEY")`,
31+
`ENV.fetch("SECRET")`, and `"${PREFIX}-${SUFFIX}"`. In every case the
32+
real credential is held or computed at runtime and never appears in the
33+
diff, so there is nothing to leak. A variable named after a secret is
34+
not a secret.
35+
36+
This takes precedence over how the value is used later. If the matched
37+
value is a reference or expression rather than a literal, dismiss it
38+
even when the resulting variable is then passed to a credential-using
39+
call site. Forwarding a variable into an API call, or a config/env
40+
lookup into an authorization header, does not turn the reference into a
41+
literal secret. Judge the matched expression, not the downstream use.
42+
43+
Confirm only when the matched value is ITSELF a literal credential of
44+
some format and appears at a call site that uses it: database
45+
connections, API client initialisation, request header construction,
46+
environment-variable defaults bound to production hostnames.
3547

3648
When in doubt, prefer confirmed with lower confidence. A missed real
3749
secret is more expensive than a noisy alert.

0 commit comments

Comments
 (0)