Skip to content

Commit 7c4363e

Browse files
[scout] update 'npx playwright' with 'node scripts/playwright' (elastic#269902)
## Summary Closes elastic/appex-qa-team#866 Moving away from using `npx playwright` in favour of `node scripts/playwright` because `npx` might auto-install things, circumventing dependency version restrictions. Most of changes are in readme files and mostly informative for the teams. From now on please use `node scripts/playwright` to interact with Playwright CLI. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
1 parent 62bb930 commit 7c4363e

39 files changed

Lines changed: 327 additions & 82 deletions

File tree

.agents/skills/scout-api-testing/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ Tests then import `apiTest` from the local fixtures: `import { apiTest } from '.
9494
- Use either `--config` or `--testFiles` (they are mutually exclusive).
9595
- Run by config: `node scripts/scout.js run-tests --arch stateful --domain classic --config <module-root>/test/scout*/api/playwright.config.ts` (or `.../api/parallel.playwright.config.ts` for parallel API runs)
9696
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --arch stateful --domain classic --testFiles <module-root>/test/scout*/api/tests/my.spec.ts`
97-
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag>`.
97+
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `node scripts/playwright test --config <...> --project local --grep <tag>`.
9898
- `run-tests` auto-detects custom config sets from `.../test/scout_<name>/...` paths.
9999
- `start-server` has no Playwright config to inspect, so pass `--serverConfigSet <name>` when your tests require a custom config set.
100100
- Debug: `SCOUT_LOG_LEVEL=debug`

.agents/skills/scout-migrate-from-ftr/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ Once execution is complete, run the new Scout tests and fix failures until they
6464
For faster feedback during the loop, start the test servers once and reuse them across iterations:
6565

6666
1. Start servers (one-time): `node scripts/scout.js start-server --stateful --serverConfigSet <configSet>` (or `--serverless <project>`).
67-
2. Run the specs per iteration: `npx playwright test --config <playwright.config.ts> <test-file>`.
67+
2. Run the specs per iteration: `node scripts/playwright test --config <playwright.config.ts> <test-file>`.
6868

6969
Falling back to `node scripts/scout.js run-tests` works but restarts the servers each time, which is much slower for iterative debugging.
7070

.agents/skills/scout-ui-testing/SKILL.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,10 +113,10 @@ test('creates and verifies a dashboard', async ({ pageObjects, page }) => {
113113
- Use either `--config` or `--testFiles` (they are mutually exclusive).
114114
- Run by config: `node scripts/scout.js run-tests --arch stateful --domain classic --config <module-root>/test/scout*/ui/playwright.config.ts` (or `.../ui/parallel.playwright.config.ts` for parallel UI)
115115
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --arch stateful --domain classic --testFiles <module-root>/test/scout*/ui/tests/my.spec.ts`
116-
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag> --headed`.
116+
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `node scripts/playwright test --config <...> --project local --grep <tag> --headed`.
117117
- `run-tests` auto-detects custom config sets from `.../test/scout_<name>/...` paths.
118118
- `start-server` has no Playwright config to inspect, so pass `--serverConfigSet <name>` when your tests require a custom config set.
119-
- Debug: `SCOUT_LOG_LEVEL=debug`, or `npx playwright test --config <...> --project local --ui`
119+
- Debug: `SCOUT_LOG_LEVEL=debug`, or `node scripts/playwright test --config <...> --project local --ui`
120120

121121
## CI enablement
122122

.eslintrc.js

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1022,6 +1022,7 @@ module.exports = {
10221022
rules: {
10231023
'@kbn/eslint/no_unsafe_dynamic_http_path': 'warn',
10241024
'@kbn/eslint/no_wrapped_error_in_logger': 'error',
1025+
'@kbn/eslint/no_npx_playwright': 'error',
10251026
'no-restricted-imports': ['error', ...RESTRICTED_IMPORTS],
10261027
'@kbn/eslint/no_deprecated_imports': [
10271028
'warn',
@@ -3025,6 +3026,19 @@ module.exports = {
30253026
],
30263027
},
30273028
},
3029+
{
3030+
// These files are allowed to reference 'npx playwright' — either because they define
3031+
// the rule itself, test it with invalid-code fixtures, or mention it in an error message
3032+
// to explain what went wrong.
3033+
files: [
3034+
'src/platform/packages/private/kbn-scout-reporting/src/helpers/cli_processing.ts',
3035+
'packages/kbn-eslint-plugin-eslint/rules/no_npx_playwright.js',
3036+
'packages/kbn-eslint-plugin-eslint/rules/no_npx_playwright.test.js',
3037+
],
3038+
rules: {
3039+
'@kbn/eslint/no_npx_playwright': 'off',
3040+
},
3041+
},
30283042
],
30293043
};
30303044

docs/extend/scout/debugging.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ This page lists the fastest ways to debug Scout tests locally and in CI.
1616
After a run, Playwright generates an HTML report. The console output includes the report path. To open the latest report:
1717

1818
```bash
19-
npx playwright show-report <plugin-path>/test/scout/ui/output/reports
19+
node scripts/playwright show-report <plugin-path>/test/scout/ui/output/reports
2020
```
2121

2222
::::::{note}
@@ -28,7 +28,7 @@ npx playwright show-report <plugin-path>/test/scout/ui/output/reports
2828
[UI Mode](https://playwright.dev/docs/test-ui-mode) lets you run and debug tests interactively.
2929

3030
```bash
31-
npx playwright test \
31+
node scripts/playwright test \
3232
--config <plugin-path>/test/scout/ui/playwright.config.ts \
3333
--project local \
3434
--ui \
@@ -68,7 +68,7 @@ If you need more control (for example, targeting a specific spec file), you can
6868
Example (repeat a single spec 30 times):
6969

7070
```bash
71-
npx playwright test dashboard_search_by_value.spec.ts \
71+
node scripts/playwright test dashboard_search_by_value.spec.ts \
7272
--project mki \
7373
--grep @cloud-serverless-search \
7474
--config src/platform/plugins/shared/dashboard/test/scout/ui/parallel.playwright.config.ts \

docs/extend/scout/run-tests.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ node scripts/scout.js start-server \
3131
And then run tests how often you'd like against the same test servers:
3232

3333
```bash
34-
npx playwright test --config <plugin-path>/test/scout/ui/playwright.config.ts \
34+
node scripts/playwright test --config <plugin-path>/test/scout/ui/playwright.config.ts \
3535
--project local \
3636
--grep @<location>-<arch>-<domain>
3737
```

packages/kbn-eslint-plugin-eslint/index.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,5 +41,6 @@ module.exports = {
4141
require_include_in_check_a11y: require('./rules/require_include_in_check_a11y'),
4242
no_wrapped_error_in_logger: require('./rules/no_wrapped_error_in_logger'),
4343
no_sync_import_from_plugin: require('./rules/no_sync_import_from_plugin'),
44+
no_npx_playwright: require('./rules/no_npx_playwright'),
4445
},
4546
};
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the "Elastic License
4+
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
5+
* Public License v 1"; you may not use this file except in compliance with, at
6+
* your election, the "Elastic License 2.0", the "GNU Affero General Public
7+
* License v3.0 only", or the "Server Side Public License, v 1".
8+
*/
9+
10+
/**
11+
* Bans the string `npx playwright` from appearing in string literals and template literals
12+
* inside shell-execution call expressions (exec, execSync, spawn, spawnSync, execa, execaSync).
13+
*
14+
* Use `node scripts/playwright` instead, which is a thin, safe wrapper that goes through
15+
* Kibana's setup and uses the pinned @playwright/test version without auto-installing anything.
16+
*/
17+
18+
const SHELL_EXEC_FUNCTIONS = new Set([
19+
'exec',
20+
'execSync',
21+
'spawn',
22+
'spawnSync',
23+
'execa',
24+
'execaSync',
25+
'execaCommand',
26+
'execaCommandSync',
27+
]);
28+
29+
const ERROR_MSG =
30+
"Do not use 'npx playwright' — use 'node scripts/playwright' instead. " +
31+
'npx may auto-install a different Playwright version, bypassing the pinned @playwright/test dependency.';
32+
33+
/**
34+
* Returns true if the given string value contains 'npx playwright'.
35+
* @param {string} value
36+
*/
37+
function containsNpxPlaywright(value) {
38+
return typeof value === 'string' && value.includes('npx playwright');
39+
}
40+
41+
/** @type {import("eslint").Rule.RuleModule} */
42+
module.exports = {
43+
meta: {
44+
type: 'problem',
45+
docs: {
46+
description: "Disallow 'npx playwright' in shell execution calls",
47+
category: 'Best Practices',
48+
recommended: true,
49+
},
50+
fixable: null,
51+
schema: [],
52+
messages: {
53+
noNpxPlaywright: ERROR_MSG,
54+
},
55+
},
56+
57+
create(context) {
58+
/**
59+
* Report if a node is a string literal or template literal containing 'npx playwright'.
60+
*/
61+
function checkStringNode(node) {
62+
if (node.type === 'Literal' && containsNpxPlaywright(node.value)) {
63+
context.report({ node, messageId: 'noNpxPlaywright' });
64+
} else if (node.type === 'TemplateLiteral') {
65+
for (const quasi of node.quasis) {
66+
if (containsNpxPlaywright(quasi.value.raw)) {
67+
context.report({ node, messageId: 'noNpxPlaywright' });
68+
break;
69+
}
70+
}
71+
}
72+
}
73+
74+
/**
75+
* Resolve the function name from a callee node, handling both direct calls
76+
* (exec(...)) and member-expression calls (child_process.exec(...)).
77+
*/
78+
function getCalleeName(callee) {
79+
if (callee.type === 'Identifier') {
80+
return callee.name;
81+
}
82+
if (callee.type === 'MemberExpression' && callee.property.type === 'Identifier') {
83+
return callee.property.name;
84+
}
85+
return null;
86+
}
87+
88+
return {
89+
CallExpression(node) {
90+
const calleeName = getCalleeName(node.callee);
91+
if (!calleeName || !SHELL_EXEC_FUNCTIONS.has(calleeName)) {
92+
return;
93+
}
94+
for (const arg of node.arguments) {
95+
checkStringNode(arg);
96+
}
97+
},
98+
};
99+
},
100+
};
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the "Elastic License
4+
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
5+
* Public License v 1"; you may not use this file except in compliance with, at
6+
* your election, the "Elastic License 2.0", the "GNU Affero General Public
7+
* License v3.0 only", or the "Server Side Public License, v 1".
8+
*/
9+
10+
const { RuleTester } = require('eslint');
11+
const rule = require('./no_npx_playwright');
12+
const dedent = require('dedent');
13+
14+
const ERROR_MSG =
15+
"Do not use 'npx playwright' — use 'node scripts/playwright' instead. " +
16+
'npx may auto-install a different Playwright version, bypassing the pinned @playwright/test dependency.';
17+
18+
const ruleTester = new RuleTester({
19+
parser: require.resolve('@typescript-eslint/parser'),
20+
parserOptions: {
21+
sourceType: 'module',
22+
ecmaVersion: 2020,
23+
ecmaFeatures: {
24+
jsx: true,
25+
},
26+
},
27+
});
28+
29+
ruleTester.run('@kbn/eslint/no_npx_playwright', rule, {
30+
valid: [
31+
// Safe: uses node scripts/playwright
32+
{
33+
code: dedent`
34+
execSync('node scripts/playwright test --config config.ts');
35+
`,
36+
},
37+
// Safe: uses node_modules/.bin/playwright (programmatic, no auto-install risk)
38+
{
39+
code: dedent`
40+
spawn('./node_modules/.bin/playwright', ['test']);
41+
`,
42+
},
43+
// Safe: unrelated exec call
44+
{
45+
code: dedent`
46+
exec('node scripts/jest src/foo.test.ts');
47+
`,
48+
},
49+
// Safe: npx with something other than playwright
50+
{
51+
code: dedent`
52+
execSync('npx some-other-tool');
53+
`,
54+
},
55+
// Safe: string not inside a shell call
56+
{
57+
code: dedent`
58+
const cmd = 'npx playwright test';
59+
`,
60+
},
61+
],
62+
63+
invalid: [
64+
// exec with npx playwright
65+
{
66+
code: dedent`
67+
exec('npx playwright test --config config.ts');
68+
`,
69+
errors: [{ message: ERROR_MSG }],
70+
},
71+
// execSync with npx playwright
72+
{
73+
code: dedent`
74+
execSync('npx playwright test');
75+
`,
76+
errors: [{ message: ERROR_MSG }],
77+
},
78+
// spawn with npx playwright as first arg
79+
{
80+
code: dedent`
81+
spawn('npx playwright show-report ./output/reports');
82+
`,
83+
errors: [{ message: ERROR_MSG }],
84+
},
85+
// spawnSync with npx playwright
86+
{
87+
code: dedent`
88+
spawnSync('npx playwright show-trace ./trace.zip');
89+
`,
90+
errors: [{ message: ERROR_MSG }],
91+
},
92+
// execa with npx playwright
93+
{
94+
code: dedent`
95+
execa('npx playwright test', ['--config', 'config.ts']);
96+
`,
97+
errors: [{ message: ERROR_MSG }],
98+
},
99+
// member expression: child_process.exec
100+
{
101+
code: dedent`
102+
child_process.exec('npx playwright test');
103+
`,
104+
errors: [{ message: ERROR_MSG }],
105+
},
106+
// template literal with npx playwright
107+
{
108+
code: dedent`
109+
execSync(\`npx playwright test --config \${configPath}\`);
110+
`,
111+
errors: [{ message: ERROR_MSG }],
112+
},
113+
],
114+
});

packages/kbn-failed-test-reporter-cli/failed_tests_reporter/report_failure.test.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,7 @@ describe('updateFailureIssue()', () => {
253253
location: '/path/to/test.ts',
254254
duration: 5000,
255255
owners: 'team:test',
256-
commandLine: 'npx playwright test --config=config.ts',
256+
commandLine: 'node scripts/playwright test --config=config.ts',
257257
}
258258
);
259259

@@ -434,7 +434,7 @@ describe('createFailureIssue() - Scout failures', () => {
434434
location: '/path/to/test.ts',
435435
duration: 5000,
436436
owners: 'team:test',
437-
commandLine: 'npx playwright test --config config.ts',
437+
commandLine: 'node scripts/playwright test --config config.ts',
438438
},
439439
api,
440440
'main',
@@ -499,7 +499,7 @@ describe('createFailureIssue() - Scout failures', () => {
499499
location: '/path/to/test.ts',
500500
duration: 3000,
501501
owners: 'team:test',
502-
commandLine: 'npx playwright test',
502+
commandLine: 'node scripts/playwright test',
503503
kibanaModule: {
504504
id: 'test-module',
505505
type: 'plugin',

0 commit comments

Comments
 (0)