|
Coverage is usually used to find new testing requirements that have been overlooked in the test plan. Many times the test requirements are written during the design process and do not take into account details of the implementation. For example, the implementation of a sorting function might use two different algorithms, depending on the size of the array sorted, a detail which does not appear in the specifications. In this case, statement coverage might show that the inputs did not include a short array that uses one of the algorithms, and that a new test is needed. Working with coverage as a guide to improve the quality of testing has been shown to be a cost effective use of resources [12].
Another common application of coverage is the generation of regression suites [10]. Generation of regression suites involves two contradictory requirements: the suite must be small enough so that it is economical to execute it after every design change, yet it must be comprehensive enough to find new bugs at each stage. Coverage enables us to identify a relatively small set of tests which is comprehensive, in the sense that it covers the required metrics [5].
Coverage also provides other benefits to the testing process that are often overlooked. One such benefit is the use of coverage or, more specifically, functional coverage, to help define testing requirements and specifications. Another benefit of functional coverage is that the process of coverage model definition helps create a better understanding of the program being tested.
While the use of coverage as an aid to the testing process has many benefits, centering the testing process around coverage has its own risks. A common misconception about coverage is that the testing methodology should involve deciding on an appropriate coverage metric and then generating tests that achieve high coverage. The weakness of this policy is shown in the following example:
Assume that one uses a process that paints a wall with an inch of protective paint. A quality assurance process (coverage) chooses a hundred points at random at which to drill and measure the paint thickness. If at all of the points there is one inch of paint, we have a reasonably good assurance that the wall is well painted. However, what do we do if the layer of paint is not thick enough at 50 points? If we tell the painter the location of these 50 points and he fixes them, the fact that the 100 points are now covered no longer guarantees that the wall is covered, because our painting process (test generation) and our coverage process are not independent. One way to overcome this difficulty is to inform the painter that some region of the wall is not well painted. This is equivalent in testing to generating a test requirement (as opposed to a specific test) from an uncovered point.
Another drawback of coverage is that many coverage models are ill suited to deal with many common problems. For example, control flow models, such as statement and branch coverage, are ill suited to deal with missing code. If, for example, a case statement should have six cases but, in practice, has only four, statement or branch coverage will not help to locate the missing code. One way to overcome this difficulty is to use several coverage models derived from different domains, so that one model will cover the weaknesses of another model.
An additional risk in using coverage is setting low coverage goals. It has been shown that using coverage to assess quality with a lower coverage target (50 – 90 percent) is not useful [12]. The reason is that the probability of bugs in hard-to-cover areas tends to be larger than the probability of bugs in well-covered areas. Therefore, it is better to use simpler coverage models with high coverage goals than more complex models with lower coverage goals.
Another risk is to create the simplest set of tests that covers the model. This is not advisable for a number of reasons; the main one is demonstrated by the above analogy to wall painting. More on misusing coverage can be found in the paper "How to Misuse Code Coverage" at http://www.rstcorp.com/marick/.
| 
