Skip to content

Commit 2ac093a

Browse files
fix(osvmatcher): dedupe bulk queries (#2808)
## Overview This fixes duplicate package queries being sent in the same OSV bulk request. Fixes #2654 ## Details When the scan input contains the same package more than once, `OSVMatcher` currently sends that same package/version query multiple times to OSV. That does not change the final result, but it does add unnecessary API work and makes larger scans do more work than needed. This PR deduplicates equivalent OSV queries before calling `BatchQueryPaging`, then expands the hydrated results back to the original package order. That keeps the scanner output behavior the same while reducing duplicate requests. I also updated the cached matcher query collection to use the same query key, so it does not rely on pointer identity when collecting missing package queries. ## Testing - Added a regression test that passes duplicate package inputs and verifies only unique queries are sent while all original result slots are preserved. - Ran `go test ./internal/clients/clientimpl/osvmatcher`. - Ran `go test ./internal/clients/clientimpl/...`. ## Checklist - [x] I have signed the [Contributor License Agreement](https://cla.developers.google.com/). - [x] I have run the linter using `./scripts/run_lints.sh`. - [x] I have run the unit tests using `go test ./internal/clients/clientimpl/...`. - [x] I have made my commits and PR title follow the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification. --------- Co-authored-by: Rohan Patnaik <rohan-patnaik@users.noreply.github.com> Co-authored-by: Rex P <rexpan@google.com>
1 parent 7daf183 commit 2ac093a

11 files changed

Lines changed: 1777 additions & 11834 deletions

cmd/osv-scanner/scan/image/testdata/cassettes/TestCommand_Docker.yaml

Lines changed: 3 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ interactions:
55
proto: HTTP/1.1
66
proto_major: 1
77
proto_minor: 1
8-
content_length: 2036
8+
content_length: 1367
99
host: api.osv.dev
1010
body: |
1111
{
@@ -17,13 +17,6 @@ interactions:
1717
},
1818
"version": "3.4.3-r1"
1919
},
20-
{
21-
"package": {
22-
"ecosystem": "Alpine:v3.18",
23-
"name": "alpine-baselayout"
24-
},
25-
"version": "3.4.3-r1"
26-
},
2720
{
2821
"package": {
2922
"ecosystem": "Alpine:v3.18",
@@ -45,13 +38,6 @@ interactions:
4538
},
4639
"version": "1.36.1-r7"
4740
},
48-
{
49-
"package": {
50-
"ecosystem": "Alpine:v3.18",
51-
"name": "busybox"
52-
},
53-
"version": "1.36.1-r7"
54-
},
5541
{
5642
"package": {
5743
"ecosystem": "Alpine:v3.18",
@@ -73,20 +59,6 @@ interactions:
7359
},
7460
"version": "3.1.7-r0"
7561
},
76-
{
77-
"package": {
78-
"ecosystem": "Alpine:v3.18",
79-
"name": "openssl"
80-
},
81-
"version": "3.1.7-r0"
82-
},
83-
{
84-
"package": {
85-
"ecosystem": "Alpine:v3.18",
86-
"name": "musl"
87-
},
88-
"version": "1.2.4-r2"
89-
},
9062
{
9163
"package": {
9264
"ecosystem": "Alpine:v3.18",
@@ -101,13 +73,6 @@ interactions:
10173
},
10274
"version": "1.3.7-r1"
10375
},
104-
{
105-
"package": {
106-
"ecosystem": "Alpine:v3.18",
107-
"name": "busybox"
108-
},
109-
"version": "1.36.1-r7"
110-
},
11176
{
11277
"package": {
11378
"ecosystem": "Alpine:v3.18",
@@ -128,7 +93,7 @@ interactions:
12893
proto: HTTP/2.0
12994
proto_major: 2
13095
proto_minor: 0
131-
content_length: 524
96+
content_length: 276
13297
body: |
13398
{
13499
"results": [
@@ -138,8 +103,6 @@ interactions:
138103
{},
139104
{},
140105
{},
141-
{},
142-
{},
143106
{
144107
"vulns": [
145108
{
@@ -152,26 +115,6 @@ interactions:
152115
}
153116
]
154117
},
155-
{
156-
"vulns": [
157-
{
158-
"id": "ALPINE-CVE-2024-13176",
159-
"modified": "2026-02-08T14:17:02.498117Z"
160-
},
161-
{
162-
"id": "ALPINE-CVE-2024-9143",
163-
"modified": "2025-12-03T22:57:50.413061Z"
164-
}
165-
]
166-
},
167-
{
168-
"vulns": [
169-
{
170-
"id": "ALPINE-CVE-2025-26519",
171-
"modified": "2025-12-11T11:16:21.978419Z"
172-
}
173-
]
174-
},
175118
{
176119
"vulns": [
177120
{
@@ -181,13 +124,12 @@ interactions:
181124
]
182125
},
183126
{},
184-
{},
185127
{}
186128
]
187129
}
188130
headers:
189131
Content-Length:
190-
- "524"
132+
- "276"
191133
Content-Type:
192134
- application/json
193135
status: 200 OK

0 commit comments

Comments
 (0)