Cross-product Functional Coverage

Code coverage, the more common meaning of the term "coverage" measures tests against the source code of the program. For example, one can measure which statements of the source code are executed by a test. Code coverage provides a metric to manage testing, it finds omissions in the test plan, and it provides an objective basis for selecting regression test suites. A number of standards, as well as policies, require testing to achieve some level of code coverage. For example, one requirements of the DOA standard [15] is 100 percent statement coverage. Code coverage is an appealing metric because it is objective, unbiased, and quantitative. It is understandable by people who do not understand the subtlety and tradeoffs of practical software testing, (for example the judges and juries of software liability cases). It becomes increasingly difficult to obtain better code coverage above 80%, and better coverage yields diminishing returns. For example, it is expensive to cover code that handles difficult to create and very rare error conditions. Code coverage provides no guarantees, because it does not measure several important dimensions of software quality. For example, statement coverage does not measure path coverage, and so common user scenarios (paths of execution) may go untested with perfect (and very expensive) statement coverage. The highest value is returned from investment in software testing when complementary techniques are combined in economical proportions. Following this rule-of-thumb, code and functional coverage are best used in combination (in practical quantities).

There are many coverage tools supporting all major programming languages. Each tool implements a number of coverage models for a particular combination of operating system, compiler, and programming language. Most of them work by instrumenting the source code and adding counters which can later be used by the tool's user interface to show the status and progress of the coverage in some detail. To apply such a tool, the user (in most cases) must recompile the software with the tool and execute the tests. After the tests are executed, an interface highlights the parts of the program that were not covered. Almost all coverage tools implement the statement and branch coverage models. Many tools implement multi-condition coverage that measures which parts of a condition had impact (e.g., A or B and C). For example, say the condition of an "if clause" is (A or B), multiple condition coverage will log whether the cases where A is true and where A is false are both covered. A few tools implement the more complex models, such as define-use, mutation, and path coverage variants [6].

Code coverage tools take less effort to deploy because the coverage models are predefined. (A unique functional coverage model must be devised for each program.) The feedback from the tool is straightforward and objective. The main disadvantage of code coverage tools is that the tools do not "understand" the application domain (such understanding is embedded in a functional coverage model). Therefore, it is very hard to tune code coverage tools to areas that the user considers important. Functional coverage models are inherently tuned to what the test engineer (as proxy for end-users) considers important.

Reasons for Creating Functional Coverage Models

1. Find areas in which the coverage is missing or low, and to improve testing in these areas. This is the most common reason for using both functional and code coverage. This use tends to be iterative: Tests are created to "plug" the big holes; coverage is measured; new tests are created to "plug" the new big holes.
   
   
2. Check if testing is comprehensive. This measure is used to compare the coverage level at different testing stages to the expected coverage level for those stages.
   
   
3. Find tests that create illegal tasks and correct the problem. In functional coverage, unlike code coverage, many of the tasks are illegal tasks or tasks requested by the programmer/tester that can not be created. In practice, many things that should not happen do happen. In such cases, the problem may be in the restrictions/definition of legal tasks, the application, or the coverage process. The problem must be located and corrected.
   
   
4. Improve the understanding of the application. Creating restrictions requires a fresh point of view on the application, which tends to help in the design and test plan creation phase.
   
   
5. Find patterns of behavior in the applications. Correlations between attributes and other details can be used to fine tune the application.
   
   
6. Measure statistics on how the application is used.
   
   
7. Create criteria for proceeding to the next level of testing. For example, in some models, functional coverage is not finished until 100 percent coverage is achieved.