TCBeans Software Test Tool Kit

White Paper

Kenneth Nagin (nagin@haifa.vnet.ibm.com)

Alan Hartman (hartman@il.ibm.com)

November 1998

What is the Tool Kit:

The TCBeans Software Test Tool Kit is a framework designed to assist testers in developing, executing and organizing function tests directed against Application Program Interfaces (APIs) and software protocols written in Java, C or C++ . Test time is saved because the tool kit supports reuse of many common testing functions. Test quality and complexity is enhanced because the tool kit supports a systematic approach to test suite generation. The systematic approach, in turn, yields higher function and code coverage, uncovering more defects early on in the software development cycle. Testing is more interesting since non-creative tasks associated with testing are automated, shielding against the high attrition rate associated with testing personnel. Distributed application testing is simplified by enabling a single test to propagate seamlessly across many host computers.

Why invest in another test tool:

Tools for testing a particular application are often developed on an ad hoc basic. Each project develops its own tools and framework to solve its particular testing needs. Usually a tool begins as a simple interpreter that can execute a limited set of APIs. The test case commands are submitted interactively or by way of a batch text file. With time, the test tool becomes more complicated as it tries to solve more and more problems, e.g. concurrent and parallel testing. However the primary focus of the project is not the tester so resources are scarce and often inadequate to satisfy all the testing requirements. This ad hoc approach to test tool development is expensive. It is preferable to develop a framework that could be used by many test organizations to amortize the cost and create a more powerful, general-purpose tool set.

Test tools do exist in the market. However these tools are either extremely expensive or not intended for testing APIs and protocols. Automatic Test Suite generators like Teradyne's TestMaster start at $50000 + $20000 for instruction. Less expensive record and replay test tools like Mercury Interactive's WinRunner are designed to assist in testing Graphic User Interfaces, not APIs or protocols.

There are two reasons to support an API, even when an application's current market requirement does not include a programming interface. The first is directly related to the testing of the application, and the second relates to its future market requirements. Most present day applications contain a lot of function, supported by a large code base. The functions interact with themselves and the outside world. The manual approach to testing simply can not exercise the complex behavior inherent in the function interactions or cover many code paths beyond those of the main function paths. Furthermore, a good API extends the utility and life span of an application, since it allows the application to be used as a component in other applications. In both cases, a programming solution is required, meaning that the application must support an API.

How does it work:

We present a novel approach to software testing by treating a test case as a software component that can be reused, customized and assembled off-line during code development, or dynamically at run time. Specifically, we treat a test case as a JavaBeans component which can be customized using a Bean Development Kit (BDK) or during actual testing by loading the test cases within the tool kit's run time testing framework which supports the functions expressed in a BDK. Reusability is gained primarily through the mechanism of class inheritance. Basic functionality shared by most test cases is supported in a super class JavaBeans test case which testers extend and specialize for their own special test environment. Testing of distributed applications is supported since versions of the test cases can be customized on one remote host and shipped to other host machines for execution.

We automate software testing by exploiting the characteristics of JavaBeans to bridge the gap between an abstract test suite and concrete test execution. Typically an abstract test suite is a text file describing a set of test cases. The test cases describe the procedure to exercise a software application being tested. The test case procedure may simply be a list of execution statements or in some cases the execution statements are accompanied by clauses to determine the correctness of the execution. The abstract test suite may be generated manually, or automatically using an automatic test suite generator, e.g. GOTCHA . In either case, the abstract test suite must be translated into object code in order to actually execute the test. Typically the text is parsed into source code and recompiled for execution. Alternatively, specialized parsers are designed for the specific software application being tested. The parser examines the test suite and executes the test against the software application. However, we exploit the characteristics of JavaBeans to create a general-purpose parser that directly executes the test case procedures and determines the correctness of the execution.

What is GOTCHA?

GOTCHA is the tool kit's engine for generating an abstract test suite for a finite state machine (FSM) driven by a coverage model. The finite state machine is described in a high level language for modeling concrete systems. Such systems may be hardware architectures or components, software systems, communication protocols, or other complex systems and processes. A test case is a sequence of stimuli for the model. The test suites generated by GOTCHA guarantee 100% coverage of the coverage model specified by the user. The GOTCHA prototype was developed for hardware architecture models but we have exploited it for modeling and testing software systems. The coverage models currently supported by GOTCHA are projected FSM state and transition coverage.

The GOTCHA tool uses a variant of the hardware verification methodology introduced by R. C. Go, C. Han Yang, M. A. Horowitz and D. L. Dill . This methodology contains four basic phases: (1) describing the processor implementation control as a Finite State Machine, (2) deriving an abstract test suite guaranteeing coverage of the FSM model using methods from formal verification, (3) automatic translation of the covering tours to test vectors, (4) translating the test vector to executable tests.

The use of GOTCHA in the context of automated software testing is slightly different than its hardware counterpart. In phase 1, the behavior of a software application is described by the software functional specification and other user requirement documention. In the hardware context, the processor implementation is described in a hardware definition language like VHDL or Verilog. The software tester translates the specification into a FSM model by coding in the GOTCHA definition language, a Pascal like derivation of the Murf Description Language. In the hardware context, the VHDL is translated automatically into the Murf Description Language with an automated extraction of the control signals. The software model builder constrains the size of the test suite by specifying which state variables must be covered. The size of the FSM being modelled is controlled by the level of abstraction used in the manual modelling process. In phases 2 and 3, the GOTCHA tool generates the abstract test suite insuring that all desired states and/or state transitions are covered. The test generation algorithms used by GOTCHA are different from those used by Go et. al. and provide for a larger range of user defined constraints on the test suite. The final phase which is only briefly discussed in the original hardware verification methodology is the subject of the TCBeans GOTCHA Automatic Software Test Factory (see below).

GOTCHA modeling can begin as soon as a functional specification is prepared. Our experience shows that the analysis required to generate a model uncovers defects in the specification that escape the usual review process. Specifically, specification defects related to ambiguity, inconsistency and incompleteness are discovered due to the systematic analysis required when building the GOTCHA model. We also note that several GOTCHA models may be built at varying levels of abstraction. A more detailed model implies a larger FSM to be explored, stretching the limits of the underlying formal verification technology. In the hardware context, only a single model is built automatically extracting the control FSM from the hardware description. This can easily lead to state explosion problems.

The GOTCHA test suite may contain a multitude of tests that cover the application's complex behavior. The tester creates a JavaBean test object which maps the GOTCHA test suite to the application being tested. The tool kit supports an automatic test function called TCBeans GOTCHA Automatic Software Test Factory (see below) which executes the GOTCHA abstract test suite against the JavaBean test object. Once the application is ready for test, the whole test suite can be run automatically or the tester can select particular tests to run.

A frequently used metric for the quality of a test suite is to gauge how much of the code and/or paths through the code are covered by the test suite. Hardware studies have shown that FSM state and transition coverage achieves high levels of branch and statement coverage of hardware definition languages. We still need to verify whether similar results can be achieved in software testing. However, we have already seen that the test suites generated exercise paths through the application under test that would typically be ignored by a less systematic approach.

Why Java?

Test cases, test objects, and the tool kit are coded in the Java programming language. Ideally the same test code can run on all target machines. The Java programming language is extremely portable since it runs on all systems that support a Java Virtual Machine and most major computer vendors support Java. When the application being tested is written in C or C++ code, Java provides a standardized procedure for accessing the C or C++ code called native implementation. This means that the native portion of the JavaBeans test objects may have to be ported between different platforms.

What is a TCBeans Test Case?

A TCBeans Test Case is a JavaBeans component that extends the Runnable class and implements Serializable. We exploit the JavaBean properties to simplify reuse, customization, run time interaction, test execution, test administration, and test assembly:

• Reuse: Test case reuse is gained primarily through the mechanism of class inheritance. Basic functionality shared by most test cases is supported in a super class TCBeans test case which testers extend and specialize for their own special test environment. The super class, TCBeans test case, supports the following functionality, which may be reused by its subclasses. (note: The test case's associated GUI (see below) supports run time interactive with the supported functions)
• It can start itself running as a thread.
• It can stop itself from running upon command or based on some wait time property.
• It can iterate the number of runs per start.
• It can notify to the outside when it completes and the results of its execution. The tool kit supports central logging to write the results.
• Three methods are supported that formalize the test process and promote reusability. They are intended to be overridden by a subject subclass:
• run(): The run() method is invoked as a thread. The user's subclass inserts test logic here, e.g. it invokes methods belonging to the object under test or changes the state of the object under test in some way.
• postCondition(): The postCondition() method is invoked when the run() method completes. The user's subclass inserts test verification logic here, e.g. it determines whether the state of the object under test is as expected. The postCondition() determines the success or failure of the test performed by the run() method.
• invariant(): The invariant() method is invoked prior to invoking the run() method and after the run() method completes. The user's subclass inserts test verification logic here that should always be true for the object under test regardless of any particular run() method testing logic. Invariance promotes reuse within a particular test suite.
• It can load and instantiate versions of itself, other test cases or resources that the test cases requires. The test cases may be in their raw class form, serialized form, or wrapped in visual components created within other BDK environments. JavaBeans test cases created within other BDK environments are usually housed within some other visual JavaBean component, e.g. Frame, and are not directly accessible so a Wrapper function is required to point to the subject JavaBean. The java.beans package is exploited to achieve interactive loading and instantiation of a JavaBeans Test Case.
• It can save and restore versions of itself, other test cases or resources to and from the file system. This is referred to as Java Serialization.
• Test case customization: TCBeans test case properties, i.e. variables, may be modified prior to testing or during testing. Prior to testing, the properties may be customized interactively using a commercial Java BDK or simply by extending the test case class and altering the properties. During actual testing, the properties may be modified programmatically since properties are expressed as public members (public setter and getter methods). Or they may be modified interactively with a run time testing framework, which supports the functions expressed in a BDK. Since a test case is serializable a particular version can be saved and reloaded from a host file system.
• Run time interaction: TCBeans test cases are non-visual JavaBeans components however testers need to interact with the test cases during testing. Thus, a test case has a Graphic User Interface (GUI) with which it is associated. The GUI must support customization of a test case's properties and the ability to activate its methods. Reuse is supported by the following innovations:
• JavaBeans Test Case properties and test case resources may be customized using the BDK's property sheet supported by the tool kit. This means that a test case writer need not create special GUI components in order to customize itself and associated resources. Rather test cases express their public variables using JavaBeans property design patterns and they automatically receive interactive customization support from the tool kit.

• Methods belonging to a TCBeans test case and associated resources may be deduced and invoked using functions supported by the tool kit. This means that a tester need not create special GUI components in order to invoke its methods, rather it declares the methods as public and the tool kit supports a mechanism with which the tester interacts with the methods and their parameters during testing. The Java reflection feature enables the framework to support the method interaction.

• Test Execution: TCBeans test case execution is simplified since JavaBeans are simply Java classes, and these classes can by dynamically loaded from the file system using a Java class loader, a common Java function.
• Test Administration: Test administration is simplified since TCBeans test cases are individual Java files. The test cases can be managed organized and archived like any standard file. Also JavaDoc provides a simplified and standardized documentation facility that can be used to easily document the test cases which compromise a test suite.
• TCBeans Test Case Assembly: Test cases are often assembled into groups. The test cases may directly associated with each, e.g. a step in a series of processes. Or they may be independent members of a test suite. In either case it required to assemble the test cases together. Two subclasses of the JavaBeans Test Case are introduced to facilitate test case assembly:
• Sequential: The Sequential class supports running a vector of JavaBeans Test Cases sequentially, i.e. one completes before the next starts.
• Concurrent: The Concurrent class supports running a vector of JavaBeans Test Cases concurrently, i.e. they all start together.

What is the TCBeans GOTCHA Automatic Software Test Factory?

The TCBeans Automatic Software Test Factory is a component in the tool kit which extends the functionality of the TCBeans test case. It bridges the gap between the GOTCHA abstract test suite and the JavaBeans test objects created by the tester. GOTCHA generates an abstract test suite that is a text file describing a set of test cases. The test cases describe the assertions (input/stimuli), called rules, that exercise the application being tested followed by post conditions which express the state of the abstract software model after a series of assertions.

TCBeans test objects are Java classes that publicize their variables as properties. These properties are mapped to the abstract test suite post condition state variables. The TCBeans test object methods are mapped to the abstract test suite assertions. The methods exercise the application being tested and the properties express the application's state.

The factory parses the abstract test suite by translating the assertion statements and post conditions to JavaBeans test objects' methods and property settings, i.e. assertion statement -> method and post condition -> property setting. The objects' methods are invoked in the order prescribed by the abstract test suite. The objects' property settings are compared against the abstract test suite's post conditions. If the post conditions and property settings are equivalent the test succeeded otherwise it fails. It is worth noting that there is no explicit compile and link phase between parsing the abstract test suite and executing the test. This means that there are no additional class files to administer.

What is TCBeans Distributed Test Support?

Distributed applications, like E Business services that exploit the Internet applications, require distributed test support. In order to support distributed testing, the tool kit introduces the novel idea of combining JavaBeans with the Java Remote Method Invocation (RMI) feature to enable customized versions of TCBeans test cases to be propagated seamlessly across many remote host machines. The super class, TCBeans test case, supports a property, called remoteTesterGroup, which represents a number of remote hosts. Tool kits, running on remote hosts, register themselves to a given remoteTesterGroup using the RMI binding protocol. The tester, located at a remote control site, customizes a test case and assigns it to a remoteTesterGroup. When the test case is started it executes on all the remote hosts belonging to the remoteTesterGroup. The tool kit marshals and unmarshals the serialized version of the test case and starts the test case running on all the remote hosts. Test case results are reported back to remote control site. The remoteTesterGroup support extends to groups of test cases and thus allows entire test suites to be remotely distributed and synchronized.

Examples:

The examples below illustrate how the tool kit is exploited to build a test suite. TC is the super class of all test cases in the tool kit. It provides common function shared by most test cases, e.g. interactive starting and stopping of itself, centralized reporting to a log, etc. Another class frequently used is TCSequential. TCSequential extends TC and supports a property called TCNames which may be customized, online or offline, to reference the class names of test cases in a test suite. TCSequential runs the test cases listed in TCNames sequentially and accumulates their results.

AnApplication Test Suite

1) Manual Test Suite Development with TCBeans Tool Set:

In the figure AnApplication Test Suite we illustrate how a tester reuses the TCBeans Software Test Tool Kit to manually create a test suite. We describe the test driver for an example application, AnApplication. The AnApplication test suite contains a set of methods that invoke AnApplication functions. AnApplicationBaseTester extends TC. AnApplicationBaseTester contains functions and properties common to all of the AnApplication test suite. Most importantly it overrides the TC's invarient() method with application verification logic common to all of the test suite. The rest of the test suite extends AnApplicationBaseTester. Each test overrides TC's run() and postCondition() to reflect its particular test logic. The test suite is assembled into the AnApplicationSequential which extends TCSequential by customizing TCNames with names of the AnApplication Test Suite.

TCGotchaFactory Usage

2) Atomatic Software Test with TCBeans GOTCHA Factory:

In the figure TCGotchaFactory Usage we illustrate how a tester reuses the TCBeans Software Test Tool Kit to create an automatic software test driver for a GOTCHA test suite. We describe the actual test driver for the IBM's JavaPhone. The JavaPhone test driver consists of only two classes, JavaPhoneTest and JavaPhoneTestGotcha. The rest of the classes are provided by the tool kit, i.e. reused. The important point here is that the tester writes only code that is specific to the application being tested. The reporting mechanism, the graphic user interface, etc. are all provided by the tool kit.

The class JavaPhoneTest contains a set of methods that invoke JavaPhone functions and a set of properties that reflect the JavaPhone current state. JavaPhoneTestGotcha extends TCGotchaFactory. It is only a few lines of code which instruct TCGotchaFactory to focus on JavaPhoneTest as its test object. TCGotchaFactory understands the special format of the GOTCHA abstract test suite and executes it against its test object, e.g. JavaPhoneTest. TCGotchaFactory extends TCBatch and references TCSequential and TCStateTransition. TCBatch parses any text file and executes it methods against a specified test case. TCSequential runs a set of test cases in sequence and accumulates their results. TCStateTransition executes methods, sets properties and compares objects. TC is the super class of all the above TC* classes.

Conclusion:

The TCBeans Software Test Tool Kit exploits JavaBeans, RMI and GOTCHA to assist testers in developing, executing and organizing function tests directed against Application Program Interfaces (APIs) and software protocols written in Java, C or C++ . The tool kit's component architecture expressed as JavaBeans components save test development time by supporting reuse of many common testing functions. GOTCHA FSM modeling building and simulation yield high quality test suites, that cover the complex behavior of the application being tested. In addition, GOTCHA automates non-creative testing tasks, and thus makes testing more interesting. The tool kit's combined support of JavaBeans and Java RMI simplifies distributed application testing by enabling a single test to propagate seamlessly across many host computers.