Introduce video recording of tests

[sovoid]

🚀 Feature Proposal

Implement an out of the box solution for recording videos of test runs with Jest and Puppeteer.

Motivation

Well, there are three main reasons why you should be recording videos of all your tests:

• Eliminates the "but it works for me" – forget about those unreproducible bugs that developers reject as a voodoo. If you have a bug on a video for everyone to see, then it did happen and is something that needs immediate attention.
• Because textual bug reports can be lacking – Textual descriptions of bugs often tend to leave important information out. Video recording the bugs as they happen allow developers to notice further details that may help track down the bug and get it fixed.
• Exploratory testing – relying solely on memory and note-taking in exploratory testing doesn’t really cut the mustard. A video of all your testing actions is much better than that. Use it for future reference to see what was tested, or to roll back when encountering a bug to understand what brought it to the surface.

Example

The video recorder primarily runs as a custom reporter.

• We use ffmpeg to record videos of our tests on CI.
• The entire test runs in the context of an Xvfb server and currently works only for Linux.
• In order to record videos, we set up an environment variable of the name DISPLAY which basically triggers the test to run in a browser on the CI within an Xvfb server.

videorecorder.js

/* eslint-disable no-console */
const { spawn } = require( 'child_process' );
const { toFilepath } = require( './Util' );

class Recorder {
	constructor() {
		this.ffmpeg = false;
	}

	logBuffer( buffer, prefix ) {
		const lines = buffer.toString().trim().split( '\n' );
		lines.forEach( function ( line ) {
			console.log( prefix + line );
		} );
	}

	start( test, outputDir, display ) {
		if ( display && display[ 0 ] === ':' ) {
			const videoPath = toFilepath( test, outputDir, 'mp4' );
			this.ffmpeg = spawn( 'ffmpeg', [
				'-f',
				'x11grab', //  grab the X11 display
				'-video_size',
				'1280x1024', // video size
				'-i',
				display, // input file url
				'-loglevel',
				'error', // log only errors
				'-y', // overwrite output files without asking
				'-pix_fmt',
				'yuv420p', // QuickTime Player support, "Use -pix_fmt yuv420p for compatibility with outdated media players"
				videoPath // output file
			] );

			this.ffmpeg.stdout.on( 'data', ( data ) => {
				this.logBuffer( data, 'ffmpeg stdout: ' );
			} );

			this.ffmpeg.stderr.on( 'data', ( data ) => {
				this.logBuffer( data, 'ffmpeg stderr: ' );
			} );
		}
	}

	stop( test, outputDir ) {
		const videoPath = toFilepath( test, outputDir, 'mp4' );
		if ( this.ffmpeg ) {
			console.log( '\n\tVideo location:', videoPath, '\n' );
			this.ffmpeg.kill( 'SIGINT' );
			this.ffmpeg = false;
		}
	}
}

module.exports = new Recorder();

• We start the video recorder before each test and kill the ffmpeg instance after each test is over. The video is saved as a .mp4 file in the folder specified by outputDir. This is how we call the video recorder in a global beforeEach and kill it in a global afterEach. The code resides in a file jest.setup.js which is passed to setupFilesAfterEnv in the jest.config.js file

jest.setup.js

const { screenshot, videorecorder } = require( '../helpers' );

jasmine.getEnv().addReporter( {
	specStarted: result => {
		jasmine.currentTest = result;
	},
	specDone: result => {
		jasmine.currentTest = result;
	}
} );

beforeEach( async () => {
	global.page = await global.browser.newPage();
	await global.page.goto( global.baseUrl );
	videorecorder.start( jasmine.currentTest.fullName, global.logpath, global.display );
	await screenshot( global.page, jasmine.currentTest.fullName, global.logpath );
} );

afterEach( async () => {
	videorecorder.stop( jasmine.currentTest.fullName, global.logpath );
	await screenshot( global.page, jasmine.currentTest.fullName, global.logpath, false );
	await global.page.close();
} );

jest.config.js

const path = require( 'path' );

module.exports = {
	globals: {
		baseUrl: ( process.env.MW_SERVER || 'http://localhost:8080' ) + (
			process.env.MW_SCRIPT_PATH || '/'
		),
		display: process.env.DISPLAY,
		logpath: process.env.LOG_DIR || path.join( __dirname, 'log' )
	},
	preset: 'jest-puppeteer-preset',
	roots: [ 'specs' ],
	verbose: true,
	globalSetup: './env/global/setup.js',
	globalTeardown: './env/global/teardown.js',
	setupFilesAfterEnv: [ './env/jest.setup.js' ],
	testEnvironment: './env/testEnvironment.js',
	testTimeout: 60000
};

We have bootstrapped the code for a video recorder as a custom reporter and can be found here:

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/602302

The recorder video for test can be found here:

https://integration.wikimedia.org/ci/job/mediawiki-quibble-selenium-vendor-docker/18336/artifact/log/Page-should-be-deletable.mp4

Pitch

Why does this feature belong in the Jest Puppeteer ecosystem?

Currently I am working as a student developer at Wikimedia Foundation for Google Summer of Codes 2020. My major task is to evaluate browser automation replacements for WebdriverIO and we have Puppeteer as one of our contenders, the other being Cypress.

What we found to be highly useful for debugging our E2E test was having video recording of each test run. Most of our tests run on our Jenkins CI environment which runs a slightly different configuration of our core software than the one we use for development and often things that run locally would fail on CI primarily because of differing Node.js versions, differing Chrome versions, etc. Screenshots and video recording have helped us come a long way and I think this would help the community at large as well and it would be awesome if we could include it as an out of the box solution.

[rodoabad]

Any updates on this? @und3fined-v01d what's in Utils?

[ma-nk]

Thanks for sharing and detailed description

+1, this feature is needed, it also works "out of the box" for cypress, which is becoming an increasingly attractive alternative for jest over the years