|
This section includes guidelines on applying coverage, based on our experience.
Coverage should not be used if the resources required for it can be better spent elsewhere. If the budget is very tight and there is not enough time to even finish the test plan, designing new tests is not useful because not even all of the old tests will be executed.
Coverage should be used only if there is a full commitment to use the data collected. Measuring coverage in order to report coverage percentiles is practically worthless. Coverage points out parts of the application that have not been tested and guides test generation to these parts. Moreover, it is very important to try to reach full coverage or at least set high coverage goals, because many bugs hide in hard-to-reach places. It is usually worthwhile to create an automatic test generator instead of generating all the tests manually. The combination of an automatic test generator and coverage is very potent as the tests can be biased in the direction in which coverage is missing.
Coverage is a very useful criteria for test selection of regression suites. Whenever a small set of tests is needed, the test suite should be selected so that it will cover as many requirements or coverage tasks as possible.
When coverage and reviews are used for the same project, reviews can put less emphasis on things that will be found by coverage. For example, a review for dead code is unnecessary if statement coverage is used. A review for boundary error in loops is redundant if the appropriate mutation coverage model [8] is used. Manually checking that some values of a variable can be attained is unnecessary if the appropriate functional coverage model is used.
Keep the coverage reports as succinct as possible, especially if one coverage expert is working on several projects. Feedback in the form "function SORT was never called" is more useful than "line 2 in function SORT was never executed (as well as lines 5,8,11...)." This is even more important for functional coverage. A functional coverage model is usually an N dimensional space over some variables. The information that a single point has not been covered is not as useful as the information that a sub-space (as large as possible) was not covered.
The following guidelines apply only to functional coverage:
- First, identify the most complex, error prone part of the application. Coverage models should be written for these parts at the high-level design stage. The benefits include a fresh look at the design, a good start for the automation of the test plan, and the creation of a model that will fit the implementation as it evolves (assuming that it still implements the same design). These benefits begin even before testing, as opposed to program based coverage, in which coverage begins to have an impact only after the tests have been executed.
- Coverage models should be created hierarchically. One should start with simple small models and combine them to create larger models. Figure 3 illustrates a hierarchy of functional coverage models. Using hierarchical models enables us to debug the coverage process for simple cases and find problems in the coverage models early.
- Restrictions should be created when the model is created. It is our experience that programmers have a very hard time specifying the correct restrictions, but they do learn a lot about the design. Before coverage is measured, sample traces should be examined to check that the restrictions are correct.
| 
