Test strategy

Purpose

To define the process for testing features in the BlackBerry Native Plug-in for Microsoft Visual Studio

Audience

Developers who use the open source project and may want to contribute to the project

Test deliverables

• Test cases
• Test strategy document (this document)
• Test execution results
• Recommendation for release

Product requirements management process

See Tracking issues

Project deliverables

Standalone Windows installer for Windows as an executable file that conducts installation of all required dependencies and allows the user to set all required settings.

Test entry criteria

• BlackBerry official release build
• Development team testing of build completed
• Development team testing results posted
• Release notes issued (listing new functionality, fixed issues, known issues, known limitations)

Test exit criteria

Prior to release to external test users, all planned testing will be completed and the defect status must comply with the following:

• External beta releases:
• No Priority 1 issues
• Limited number of Priority 2 issues allowed per component according to triage decisions
• Gold releases:
• No Priority 1 or Priority 2 issues

Communicating results process

An email is sent to internal BlackBerry stakeholders with a link to a summary of the testing results.

Assumptions

The plug-in is supported and tested only on Microsoft Visual Studio 2013 running Windows 7 and 8.1.

Dependencies

This project relies on Microsoft Visual Studio and its underlying structure (like GDB) for its operation. Some issues are beyond the scope of this project to fix and may require a patch or update from Microsoft at some point in the future.

Risks

No known risks.

Test cases

Test cases are used by the test team to validate the features of the plug-in. Test cases are created based on the acceptance criteria in each user story. Each test case should validate one verification point. Each test case is reviewed and approved by the development team to ensure that it is valid and verifies the appropriate point.

Test cases are currently managed internally.

Test coverage

There are three levels of test coverage. The requirements for each level are as follows:

• Sanity (15% coverage): Install the plug-in, create a project, launch on a device and simulator, attach the debugger, hit a breakpoint, and step over
• Limited regression (50% coverage): Sanity testing plus a set of basic functional test cases from all areas which are determined on a case by case basis
• Full regression (80% coverage): Limited regression testing plus a set of extended functional test cases which are determined on a case by case basis

Scope of testing

The testing environment for Microsoft Visual Studio 2010 consists of:

Table 1: OS and device configurations

OS	Device
	PlayBook Devices	PlayBook Simulator	Z10 Devices	Z10 Simulator	Q10 Devices	Q10 Simulator	Q5 Devices
Win 8-64	N/A	N/A
Win 7-64
Win 7-32

“N/A” indicates that the OS is not applicable to the device configuration.

Note: There are several different versions of the BlackBerry PlayBook tablet, BlackBerry Z10 smartphone, BlackBerry Q10 smartphone, and BlackBerry Q5 smartphone devices. Devices must be tested for different features, markets, and carriers.

When formal testing on a BlackBerry certified build is done, the focus is on:

• Issue verification
• New Feature testing (and adding any required test cases)
• Regression testing

Regression testing

Regression testing consists of a minimum of sanity testing on each of the OS and device configurations listed in Table 1: OS and device configurations and at least one full regression cycle on one of the OS and device configurations.

Release

If there are no P1 or P2 issues open on the build under testing, then the testing for each OS and device configuration passes and the test team recommends the release of the build. When all the required BlackBerry teams have contributed their recommendations, a senior BlackBerry executive signs off, and the build is posted externally.