forked from CollaboraOnline/online
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
318 lines (220 loc) · 10.7 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
Cypress based test framework for Collabora Online
===================================================
Installation
------------
You need to have run configure with the --enable-cypress option.
In a normal desktop environment you only need to install npm
packages for running cypress tests. This is done by the build
system, so running 'make check' will do the basic installation.
https://docs.cypress.io/guides/getting-started/installing-cypress.html#npm-install
For CI you might need to install some additional dependencies:
https://docs.cypress.io/guides/guides/continuous-integration.html#Dependencies
Running tests
-------------
All tests are part of the make check build expect notebookbar. So you can
just execute it from the root folder or under the
cypress_test folder to run cypress tests only.
make check
IMPORTANT: Before stepping under cypress_test folder
and running any command listed here, make sure you've
done a top-level make, so everything is up-to-date.
Running commands from under cypress_test folder won't
trigger a rebuild.
To run cypress test cases selectively, you need to
go in to the cypress_test folder first and run one of
the following commands.
To run all desktop tests (this includes notebookbar UI related tests):
make check-desktop
To run all mobile tests:
make check-mobile
To run one specific test suite of desktop tests,
use spec argument with a relative path to
cypress_test/integration_tests/desktop/:
make check-desktop spec=writer/form_field_spec.js
To run one specific test suite of mobile tests,
use spec argument with a relative path to
cypress_test/integration_tests/mobile/:
make check-mobile spec=writer/toolbar_spec.js
Running one specific test
-------------------------
To run one test case of a test suite you can use Mocha's
'only' feature. Just replace the it(...) function with
it.only(...) in the spec file for the selected test case
and run the test suite using the spec parameter.
For example, to run the test with title 'Apply font name.'
inside apply_font_spec.js file, you need to add it.only():
- it('Apply font name.', function() {
+ it.only('Apply font name.', function() {
Then run the test suite with:
make check-mobile spec=writer/apply_font_spec.js
Or open the test suite in the interactive test runner:
make run-mobile spec=writer/apply_font_spec.js
Opening interactive test runner
-------------------------------
Cypress has an interactive test runner application which
runs the test in the browser. So you can see the result of
the different steps your test makes in the browser. It's useful
during writing new tests or checking why an existing
test fails.
https://docs.cypress.io/guides/core-concepts/test-runner.html
To open desktop tests in the test runner:
make run-desktop
To open mobile tests in the test runner:
make run-mobile
To open one specific test suite of desktop tests,
use spec argument with a relative path to
cypress_test/integration_tests/desktop/:
make run-desktop spec=writer/form_field_spec.js
To open one specific test suite of mobile tests,
use spec argument with a relative path to
cypress_test/integration_tests/mobile/:
make run-mobile spec=writer/toolbar_spec.js
During the build we run the tests with Chrome browser, so make sure
you select Chrome browser on the GUI while checking tests.
We are using different configuration and environment variables for
mobile and desktop tests, that's why there are two separate commands
for them and there is no option to open all the tests in the
test runner.
Running tests to update the help dialog screenshots
--------------------
There are two spec files which you can to update the screenshots in help dialog
1. cypress_test/integration_tests/desktop/writer/help_dialog_update_spec.js
2. cypress_test/integration_tests/desktop/calc/help_dialog_update_spec.js
You can run the specfile individually using:
1. make -C cypress_test UPDATE_SCREENSHOT=true check-desktop spec=writer/help_dialog_update_spec.js
2. make -C cypress_test UPDATE_SCREENSHOT=true check-desktop spec=calc/help_dialog_update_spec.js
If UPDATE_SCREENSHOT is true then only cypress will run the spec file and replace the original screenshot with new screenshot, the default value of UPDATE_SCREENSHOT is false
Interference testing
--------------------
The idea is to run an existing cypress test while the same document
is opened by a second user. The second user does some activity
(e.g. moving cursor, clicking, etc) and we check whether it makes
any interference with the first user's view. The first user is doing
the actual test's steps, so we can see if interference breaks the assertions.
In Writer, the interfering user moves the cursor left and right. In Calc,
the interfering user moves the cellcursor left and right. In Impress, it
just does mouse-clicking next to the slide. All these activities trigger
view switches in the core, which can hit issues with multiple users.
To run all mobile tests with interference testing:
make check-interfer-mobile
To run one specific mobile test suit:
make check-interfer-mobile spec=writer/toolbar_spec.js
We can also run the interactive test runner with interference
testing. In this case, the test user is visible in the browser
and the interfering user runs in the background. Yon can see
the activity of the interfering user as another view appears
in the document moving its cursor left and right.
To open one specific mobile test suit in interactive test runner:
make run-interfer-mobile spec=writer/toolbar_spec.js
To run all desktop tests with interference testing:
make check-interfer-desktop
To run one specific desktop test suit:
make check-interfer-desktop spec=writer/form_field_spec.js
To open one specific desktop test suit in interactive test runner:
make run-interfer-desktop spec=writer/form_field_spec.js
The implementation of interference testing can be found in
cypress_test/integration_tests/common/interference_user_spec.js
test file.
Running tests in different browsers
-----------------------------------
By default, the tests are run with chrome / chromium. If you need to
run the tests with a different browser, you can use the CYPRESS_BROWSER
variable:
CYPRESS_BROWSER="firefox" make check
This variable can be set to any value which is accepted by cypress
--browser command line argument:
https://docs.cypress.io/guides/guides/command-line.html#cypress-run-browser-lt-browser-name-or-path-gt
Running tests with nextcloud integration
----------------------------------------
You can run any test runner command in an NC environment with setting
CYPRESS_INTEGRATION environment variable. For example:
CYPRESS_INTEGRATION="nextcloud" make check
Prerequisites:
* Need a local NC 20 installation connected with Collabora Online
* NC should be available at http://localhost/nextcloud/ (no ssl)
* Need an NC user with 'cypress_test' as user name and password.
Limitations:
* NC integration uses iframes which makes harder to test with cypress.
* cy.document() and cy.window() not properly works by now.
Running tests with php-proxy
----------------------------
You can run any test runner command with php-proxy with setting
CYPRESS_INTEGRATION environment variable. For example:
CYPRESS_INTEGRATION="php-proxy" make check
Prerequisites:
* Download php-proxy script:
https://github.com/CollaboraOnline/richdocumentscode/blob/master/proxy.php
* Put it under /srv/www/htdocs/richproxy/ folder.
Other useful flags
------------------
1. ENABLE_VIDEO_REC
You can enable video recording during running a test suite in headless mode.
So you can check visually what happens in the browser during the test.
Unfortunately this does not work for multi-user tests.
ENABLE_VIDEO_REC="1" make check-mobile spec=writer/apply_font_spec.js
2. ENABLE_CONSOLE_LOG
With this flag we can enable logging for console.error() messages and so they
will be displayed in the command line. It's useful when we can reproduce a test
failure only in headless mode and we need to debug the client-side JS code.
ENABLE_CONSOLE_LOG="1" make check-mobile spec=writer/apply_font_spec.js
3. CYPRESS_A11Y (true|false, default: true)
You can enable/disable accessibility support during running a test suite.
Moreover in describe you can use the taga11yenabled tag for tests that should
be run only when accessibility is enabled, and the taga11ydisabled tag
for tests that should be run only when accessibility is disabled.
Known issues
------------
1. Different builddir and sourcedir with symlinks.
Cypress has an issue with symlinks:
https://github.com/cypress-io/cypress/issues/3482
We use the related feature only when the build directory
is different from the source directory. So to avoid this
issue with the supportFile, you should build in the source
directory or you should avoid symlinks in the path of the
build directory.
Code coverage
-------------
We use nyc to instrument the code and then cypress code coverage
plugin is used to generate coverage numbers. This workflow
is called by the following command:
make run-cov
The output is put under cypress_test/coverage folder.
Open the cypress_test/coverage/lcov-report/index.html
file to get the summary report.
The make command above touches the source files under
browser/dist/src folder, so after testing the coverage
doing a clean build is a good idea (e.g. make clean).
See also this link:
https://docs.cypress.io/guides/tooling/code-coverage.html
Used Packages
-------------
- cypress:
Cypress integration test framework.
- cypress-terminal-report:
This package makes cypress to dump test logs to the command line
when a test fails. You can write things to this log by calling
cy.log() in the test code.
- cypress-log-to-output:
This one can be used to dump console.error() messages to the command
line. To enable this functionality you need to set ENABLE_CONSOLE_LOG
environment variable. e.g. `ENABLE_CONSOLE_LOG="1" make check-mobile`.
- cypress-select-tests:
We can filter out tests or test suites before execution using this
package. Now, it is used to filter out tests based on the core's version.
See plugins/blacklists.js.
- cypress-wait-until:
Introduces cy.waitUntil() command which can be used as a while loop.
We can't write loops in a cypress test otherwise. It's useful when
the cypress tools can't be used to wait on something.
- eslint:
A JS linter tool for identifying patterns in JavaScript code. We use this
to make sure code conventions are met. Run by make check and make run-*.
- eslint-plugin-cypress-rules:
This is our own eslint plugin to catch cypress specific patterns in the
test code.
- get-port-cli:
Used by the build system to find an available port to use as coolwsd's
communication port.
- wait-on:
Used by the build system to wait on coolwsd server to start before
tests are started.