How Much Testing is enough

How Much Testing is enough

This is an important question to address as part of the test objective and strategy. Determining how much testing is sufficient is a judgment call, depending on the particular situation being faced. Factors that help determine how much system testing is enough include:

•Desired functional coverage – breadth of functional testing.

•Degree of quality, reliability or cleanliness required in the delivered product – depth of functional testing.

•Range of types of test that needed to be included, (e.g. usability, performance, security and control, compatibility/configuration – scope & variety of testing).

•Anticipated level of quality when it is delivered for testing.

•Risk and consequences of the defects that may be hidden.

•Ability to meet established audit standards, test completion criteria and quality goals for the system

•Constraints on the test effort, such as: the time and resources available for testing, and the feasibility, difficulties and costs of testing

One method to determine how many tests are needed is incremental justification. The idea is to define and accumulate test requirements in a series of cycles. The test are prioritized by the assessed level of criticality or importance, for example:

•Critical – MUST test

•Important – Should test

•Desirable – Nice if we get to it

The first cycle of test requirement addresses the critical tests. The second iteration addresses the important tests, and so on. During each iteration cycle, the incremental or additional time and cost of the newly added tests must be justified. Eventually, when the next iteration of added test cases can no longer be justified, the set of test to be performed is complete.