2
2
3
3
This document describes the manual test plan for the QL extension for Visual Studio Code.
4
4
5
- The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
5
+ The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
6
6
effort vs value basis).
7
7
8
- #### What this doesn't cover
8
+ ## What this doesn't cover
9
+
9
10
We don't need to test features (and permutations of features) that are covered by automated tests.
10
11
11
- ### Before releasing the VS Code extension
12
+ ## Before releasing the VS Code extension
13
+
12
14
- Go through the required test cases listed below
13
- - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
15
+ - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
14
16
choose to go through some of the Optional Test Cases.
15
17
- Run a query using the existing version of the extension (to generate an "old" query history item)
16
18
@@ -24,23 +26,25 @@ choose to go through some of the Optional Test Cases.
24
26
25
27
1 . Open the [ UnsafeJQueryPlugin query] ( https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql ) .
26
28
2 . Run a MRVA against the following repo list:
27
- ```
28
- {
29
- "name": "test-repo-list",
30
- "repositories": [
31
- "angular-cn/ng-nice",
32
- "apache/hadoop",
33
- "apache/hive"
34
- ]
35
- }
36
- ```
29
+
30
+ ``` json
31
+ {
32
+ "name" : " test-repo-list" ,
33
+ "repositories" : [
34
+ " angular-cn/ng-nice" ,
35
+ " apache/hadoop" ,
36
+ " apache/hive"
37
+ ]
38
+ }
39
+ ```
40
+
37
41
3 . Check that a notification message pops up and the results view is opened.
38
42
4 . Check the query history. It should:
39
43
- Show that an item has been added to the query history
40
44
- The item should be marked as "in progress".
41
45
5 . Once the query starts:
42
- - Check the results view
43
- - Check the code paths view, including the code paths drop down menu.
46
+ - Check the results view
47
+ - Check the code paths view, including the code paths drop down menu.
44
48
- Check that the repository filter box works
45
49
- Click links to files/locations on GitHub
46
50
- Check that the query history item is updated to show the number of results
@@ -74,7 +78,7 @@ choose to go through some of the Optional Test Cases.
74
78
1 . Click a history item (for MRVA):
75
79
- Check that exporting results works
76
80
- Check that sorting results works
77
- - Check that copying repo lists works
81
+ - Check that copying repo lists works
78
82
2 . Open the query results directory:
79
83
- Check that the correct directory is opened and there are results in it
80
84
3 . View logs
@@ -84,12 +88,12 @@ choose to go through some of the Optional Test Cases.
84
88
85
89
Run one of the above MRVAs, but cancel it from within VS Code:
86
90
- Check that the query is canceled and the query history item is updated.
87
- - Check that the workflow run is also canceled.
91
+ - Check that the workflow run is also canceled.
88
92
- Check that any available results are visible in VS Code.
89
93
90
- ### Test Case 6: MRVA - Change to a different colour theme
94
+ ### Test Case 6: MRVA - Change to a different colour theme
91
95
92
- Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
96
+ Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
93
97
Are there any components that are not showing up?
94
98
95
99
## Optional Test Cases
@@ -99,24 +103,27 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
99
103
### Selecting repositories to run on
100
104
101
105
#### Test case 1: Running a query on a single repository
102
- 1 . When the repository exists and is public
103
- 1 . Has a CodeQL database for the correct language
104
- 2 . Has a CodeQL database for another language
106
+
107
+ 1 . When the repository exists and is public
108
+ 1 . Has a CodeQL database for the correct language
109
+ 2 . Has a CodeQL database for another language
105
110
3 . Does not have any CodeQL databases
106
111
2 . When the repository exists and is private
107
112
1 . Is accessible and has a CodeQL database
108
113
2 . Is not accessible
109
114
3 . When the repository does not exist
110
115
111
116
#### Test case 2: Running a query on a custom repository list
117
+
112
118
1 . The repository list is non-empty
113
- 1 . All repositories in the list have a CodeQL database
119
+ 1 . All repositories in the list have a CodeQL database
114
120
2 . Some but not all repositories in the list have a CodeQL database
115
121
3 . No repositories in the list have a CodeQL database
116
122
2 . The repository list is empty
117
123
118
124
#### Test case 3: Running a query on all repositories in an organization
119
- 1 . The org exists
125
+
126
+ 1 . The org exists
120
127
1 . The org contains repositories that have CodeQL databases
121
128
2 . The org contains repositories of the right language but without CodeQL databases
122
129
3 . The org contains repositories not of the right language
@@ -126,20 +133,25 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
126
133
### Using different types of controller repos
127
134
128
135
#### Test case 1: Running a query when the controller repository is public
136
+
129
137
1 . Can run queries on public repositories
130
138
2 . Can not run queries on private repositories
131
139
132
140
#### Test case 2: Running a query when the controller repository is private
141
+
133
142
1 . Can run queries on public repositories
134
143
2 . Can run queries on private repositories
135
144
136
145
#### Test case 3: Running a query when the controller repo exists but you do not have write access
146
+
137
147
1 . Cannot run queries
138
148
139
149
#### Test case 4: Running a query when the controller repo doesn’t exist
150
+
140
151
1 . Cannot run queries
141
152
142
153
#### Test case 5: Running a query when the "config field" for the controller repo is not set
154
+
143
155
1 . Cannot run queries
144
156
145
157
### Query History
@@ -150,6 +162,7 @@ The first test case specifies actions that you can do when the query is first ru
150
162
with this since it has quite a limited number of actions you can do.
151
163
152
164
#### Test case 1: When variant analysis state is "pending"
165
+
153
166
1 . Starts monitoring variant analysis
154
167
2 . Cannot open query history item
155
168
3 . Can delete a query history item
@@ -160,8 +173,8 @@ with this since it has quite a limited number of actions you can do.
160
173
2 . By query date
161
174
3 . By result count
162
175
5 . Cannot open query directory
163
- 6 . Can open query that produced these results
164
- 1 . When the file still exists and has not moved
176
+ 6 . Can open query that produced these results
177
+ 1 . When the file still exists and has not moved
165
178
2 . When the file does not exist
166
179
7 . Cannot view logs
167
180
8 . Cannot copy repository list
@@ -171,6 +184,7 @@ with this since it has quite a limited number of actions you can do.
171
184
12 . Cannot cancel analysis
172
185
173
186
#### Test case 2: When the variant analysis state is not "pending"
187
+
174
188
1 . Query history is loaded when VSCode starts
175
189
2 . Handles when action workflow was canceled while VSCode was closed
176
190
3 . Can open query history item
@@ -204,12 +218,14 @@ with this since it has quite a limited number of actions you can do.
204
218
4 . A popup allows you to open the directory
205
219
206
220
#### Test case 3: When variant analysis state is "in_progress"
221
+
207
222
1 . Starts monitoring variant analysis
208
- 1 . Ready results are downloaded
209
- 2 . Can cancel analysis
223
+ 1 . Ready results are downloaded
224
+ 2 . Can cancel analysis
210
225
1 . Causes the actions run to be canceled
211
226
212
227
#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
228
+
213
229
1 . Stops monitoring variant analysis
214
230
1 . All results are downloaded if state is succeeded
215
231
2 . Otherwise, ready results are downloaded, if any are available
@@ -220,16 +236,18 @@ with this since it has quite a limited number of actions you can do.
220
236
This requires running a MRVA query and seeing the results view.
221
237
222
238
#### Test case 1: When variant analysis state is "pending"
239
+
223
240
1 . Can open a results view
224
241
2 . Results view opens automatically
225
242
- When starting variant analysis run
226
243
- When VSCode opens (if view was open when VSCode was closed)
227
244
3 . Results view is empty
228
245
229
246
#### Test case 2: When variant analysis state is not "pending"
247
+
230
248
1 . Can open a results view
231
249
2 . Results view opens automatically
232
- 1 . When starting variant analysis run
250
+ 1 . When starting variant analysis run
233
251
2 . When VSCode opens (if view was open when VSCode was closed)
234
252
3 . Can copy repository list
235
253
1 . Text is copied to clipboard
@@ -240,43 +258,45 @@ This requires running a MRVA query and seeing the results view.
240
258
6 . Can open query file
241
259
1 . When the file still exists and has not moved
242
260
2 . When the file does not exist
243
- 7 . Can open query text
244
- 8 . Can sort repos
245
- 1 . By name
246
- 2 . By results
247
- 3 . By stars
261
+ 7 . Can open query text
262
+ 8 . Can sort repos
263
+ 1 . By name
264
+ 2 . By results
265
+ 3 . By stars
248
266
4 . By last updated
249
267
9 . Can filter repos
250
- 10 . Shows correct statistics
251
- 1 . Total number of results
252
- 2 . Total number of repositories
268
+ 10 . Shows correct statistics
269
+ 1 . Total number of results
270
+ 2 . Total number of repositories
253
271
3 . Duration
254
- 11 . Can see live results
272
+ 11 . Can see live results
255
273
1 . Results appear in extension as soon as each query is completed
256
274
12 . Can view interpreted results (i.e. for a "problem" query)
257
- 1 . Can view non-path results
275
+ 1 . Can view non-path results
258
276
2 . Can view code paths for "path-problem" queries
259
277
13 . Can view raw results (i.e. for a non "problem" query)
260
278
1 . Renders a table
261
- 14 . Can see skipped repositories
262
- 1 . Can see repos with no db in a tab
263
- 1 . Shown warning that explains the tab
279
+ 14 . Can see skipped repositories
280
+ 1 . Can see repos with no db in a tab
281
+ 1 . Shown warning that explains the tab
264
282
2 . Can see repos with no access in a tab
265
- 1 . Shown warning that explains the tab
283
+ 1 . Shown warning that explains the tab
266
284
3 . Only shows tab when there are skipped repos
267
- 15 . Result downloads
268
- 1 . All results are downloaded automatically
285
+ 15 . Result downloads
286
+ 1 . All results are downloaded automatically
269
287
2 . Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
270
- 3 . Only 3 items are downloaded at a time
271
- 4 . Results for completed queries are still downloaded when
272
- 1 . Some but not all queries failed
288
+ 3 . Only 3 items are downloaded at a time
289
+ 4 . Results for completed queries are still downloaded when
290
+ 1 . Some but not all queries failed
273
291
2 . The variant analysis was canceled after some queries completed
274
292
275
293
#### Test case 3: When variant analysis state is in "succeeded" state
294
+
276
295
1 . Can view logs
277
- 2 . All results are downloaded
296
+ 2 . All results are downloaded
278
297
279
298
#### Test case 4: When variant analysis is in "failed" or "canceled" state
299
+
280
300
1 . Can view logs
281
301
1 . Results for finished queries are still downloaded.
282
302
@@ -305,14 +325,17 @@ This requires running a MRVA query and seeing the results view.
305
325
1 . Collapse/expand tree nodes
306
326
307
327
Error cases that trigger an error notification:
308
- 1 . Try to add a list with a name that already exists
328
+
329
+ 1 . Try to add a list with a name that already exists
309
330
1 . Try to add a top-level database that already exists
310
331
1 . Try to add a database in a list that already exists in the list
311
332
312
333
Error cases that show an error in the panel (and only the edit button should be visible):
334
+
313
335
1 . Edit the db config file directly and save invalid JSON
314
336
1 . Edit the db config file directly and save valid JSON but invalid config (e.g. add an unknown property)
315
- 1 . Edit the db config file directly and save two lists with the same name
337
+ 1 . Edit the db config file directly and save two lists with the same name
316
338
317
339
Cases where there the welcome view is shown:
318
- 1 . No controller repo is set in the user's settings JSON.
340
+
341
+ 1 . No controller repo is set in the user's settings JSON.
0 commit comments