Commit 2ac093a
fix(osvmatcher): dedupe bulk queries (#2808)
## Overview
This fixes duplicate package queries being sent in the same OSV bulk
request.
Fixes #2654
## Details
When the scan input contains the same package more than once,
`OSVMatcher` currently sends that same package/version query multiple
times to OSV. That does not change the final result, but it does add
unnecessary API work and makes larger scans do more work than needed.
This PR deduplicates equivalent OSV queries before calling
`BatchQueryPaging`, then expands the hydrated results back to the
original package order. That keeps the scanner output behavior the same
while reducing duplicate requests.
I also updated the cached matcher query collection to use the same query
key, so it does not rely on pointer identity when collecting missing
package queries.
## Testing
- Added a regression test that passes duplicate package inputs and
verifies only unique queries are sent while all original result slots
are preserved.
- Ran `go test ./internal/clients/clientimpl/osvmatcher`.
- Ran `go test ./internal/clients/clientimpl/...`.
## Checklist
- [x] I have signed the [Contributor License
Agreement](https://cla.developers.google.com/).
- [x] I have run the linter using `./scripts/run_lints.sh`.
- [x] I have run the unit tests using `go test
./internal/clients/clientimpl/...`.
- [x] I have made my commits and PR title follow the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification.
---------
Co-authored-by: Rohan Patnaik <rohan-patnaik@users.noreply.github.com>
Co-authored-by: Rex P <rexpan@google.com>1 parent 7daf183 commit 2ac093a
11 files changed
Lines changed: 1777 additions & 11834 deletions
File tree
- cmd/osv-scanner/scan
- image/testdata/cassettes
- source/testdata/cassettes
- internal/clients/clientimpl/osvmatcher
Lines changed: 3 additions & 61 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | 20 | | |
28 | 21 | | |
29 | 22 | | |
| |||
45 | 38 | | |
46 | 39 | | |
47 | 40 | | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | 41 | | |
56 | 42 | | |
57 | 43 | | |
| |||
73 | 59 | | |
74 | 60 | | |
75 | 61 | | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | 62 | | |
91 | 63 | | |
92 | 64 | | |
| |||
101 | 73 | | |
102 | 74 | | |
103 | 75 | | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | 76 | | |
112 | 77 | | |
113 | 78 | | |
| |||
128 | 93 | | |
129 | 94 | | |
130 | 95 | | |
131 | | - | |
| 96 | + | |
132 | 97 | | |
133 | 98 | | |
134 | 99 | | |
| |||
138 | 103 | | |
139 | 104 | | |
140 | 105 | | |
141 | | - | |
142 | | - | |
143 | 106 | | |
144 | 107 | | |
145 | 108 | | |
| |||
152 | 115 | | |
153 | 116 | | |
154 | 117 | | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | 118 | | |
176 | 119 | | |
177 | 120 | | |
| |||
181 | 124 | | |
182 | 125 | | |
183 | 126 | | |
184 | | - | |
185 | 127 | | |
186 | 128 | | |
187 | 129 | | |
188 | 130 | | |
189 | 131 | | |
190 | | - | |
| 132 | + | |
191 | 133 | | |
192 | 134 | | |
193 | 135 | | |
| |||
0 commit comments