An Object-Oriented Testing Framework

This is some preliminary material on an object-oriented testing framework.

Overview and purpose

The testing framework is designed to support testing of object-oriented class hierarchies. The languages supported by the test framework are

Testing is supported at two levels: For all of these, the test framework uses the class hierarchy to maximize the coverage of tests and to reuse test suites in subclasses. This approach is also the key to testing abstract classes. The principle on which hierarchical testing is called the "substitution principle": an instance of a subclass can be used anywhere an instance of one of its superclasses can be used. This is generally considered to be one of the principles rules of object-oriented programming, so using this test framework can even encourage better object-oriented design! See Hierarchical testing for a more detailed example.

Although it would make sense to mimic the class hierarchies of the object/s under test (OUT) in the test suite, I will assume here that the test framework and the tests suites are implemented in plain Tcl. This is purely for the purpose of gaining a wider audience. The test framework is structured using Tcl namespaces: each test "object" in the test suite is implemented as a namespace. "Inheritance" is implemented by importing namespaces. There are two key root objects: one for object testing, and one for behavioral testing. Each of these in turn is subdivided into namespaces for Java and Itcl testing.

Requirements

[I would like to take this piece of software through the new Tycho code rating system, including going so far as to have design and code reviews. To aid the design review, here is my statement of requirements.]

The purpose of the test framework is to provide a reasonably simple but powerful framework for testing object-oriented class hierarchies. It will directly support testing of individual objects and of small sets or clusters of objects.

The test framework will deal properly with the probems of testing class hierarchies. In particular, it will support testing of abstract classes and it will enable tests to be reused in subclasses, conforming to the principles of inheritance.

The test framework will encourage and reward good object-oriented design. It will be flexible enough to work around designs that are not considered "good" in recognition of pragmatic concerns.

The test framework will support Itcl and Java, with the most important of these being Java. To encourage its use in Java installations, it will be written in plain Tcl.

The test framework will generate a report that includes at least the following information for each class or cluster tested:

The test framework must also be able to produce a history report, showing key historical data and test statistics.

The object test

Object testing is designed to test a single class. A test object is created by the ::test::class and ::test::abstractclass commands, which take the name of the class under test as the first argument, an optional language specifier, and a series of test declarations.

::test::abstractclass classname ?-language language script
Create a new namespace for testing the class named classname. This command is the same as ::test::class, except that the class will not actually be instantiated and tested. since it is abstract. The instantiation and testing will be performed only on concrete subclasses.

::test::class classname ?-language language script
Create a new namespace for testing the class named classname. The -language option specifies the language in which the class is written, and default to Java. The script is a script that declares the object-level tests for this class.
The script of this class contains call to a number of commands implemented by the test framework. These commands execute tests on the object-under-test and tabulate the results for later reporting. The script can also contain arbitrary Tcl commands, for defining utility procedures and the like. Note that the tests are not executed when the script is sources, but merely defined for later execution with the test execution commands. The following commands can be used:
abstract command arguments...
Declare that the following command is abstract, where command can be one of behavior, constructor, or method. This declaration prevents the test framework from running this particular test command on objects of this class. To activate the test, the subclass that can support the test must use the concrete declaration.

behavior description tests
Initialize the framework for testing a behavior of a single object, and create a single object named $this. The description is a short descriptive string that will appear in the test reports. The tests argument is a script that is executed for this behavior.

concrete command arguments...
Declare that the following command is concrete, where command can be one of behavior, constructor, or method. This declaration allows the test framework to run this particular test command on objects of this class and all subclasses.

constructor description ?script? ?results?
Initialize the framework for testing a constructor of the object. The scripts argument is a script that is executed to create a new object -- this script must set the $this variable to the newly created object. results a script that is executed to check that the object meets a minimum confidence level, and generally contains a series of query commands.

Each time constructor is executed, script becomes the default construction script to use for subsequent tests. If script is not supplied, then description is looked up in the existing constructors, and its script is used to construct objects for following tests. There is always a default constructor called "default" that construct an object with no arguments.

inherit classname ?classname... ?
Declare the class from which this class inherits. This statement allows the test framework to properly test the class hierarchy using the substitution principle.

method methodname tests
Initialize the framework for testing the method given by methodname. There can be more than one call to testmethod within each test object. The tests argument is a script that is executed for this method.
The test execution and result comparison within method, constructor and behavior is split into several parts rather than into a single call. This provides better control over the test process and better test reporting. Within each of these, the following commands can be executed:
exception message
Compare the given message with an exception produced by the most recent test, and output or log the result. If a test produces an exception and the exception is not matched with this command, then the exception will be reported.

query ?description? result script
Execute script and make its result the result value of the most recently executed test. The description, if supplied, is used to identify the query in the test report (it is normally only supplied if there is more than one query on following a single test.

test description ?result? script
If inside a call to method, create a new object named $this. If inside a call to behavior, the object already exists. Then execute script. The description, if supplied, is a short descriptive string that will appear in the test reports. The result of script is the default result for this test. result, if supplied, is the result against which to compare the return value of this call.

Here is a complete example, showing tests for a directed graph class. The result against which calls on the object under test are compared are highlighted in bold type:

::test::class ::tycho::Digraph -language Itcl {
    inherit ::tycho::AbstractGraph

    constructor "Construct an empty graph" {
	set this [::tycho::Digraph #auto]
    } {
	query {} {$this vertices}
	query {} {$this edges}
    }

    constructor "Construct a small graph" {
	set this [::tycho::Digraph #auto]
	$this type configure vertex -label ""
	$this type configure edge -weight 0
	$this parse {
	    vertex a -label "First vertex"
	    vertex b -label "Second vertex"
	    vertex c
	    vertex aa
	    edge a b
	    edge b c
	    edge a c -weight 3
	}
    } {
	query {a aa b c} {lsort [$this vertices]}
	query {a b a c b c} {$this edges}
    }

    method edges {
	test "Get all edges" {a b a c b c} {
	    $this edges
	}
	test "Get edges by pattern" {a b a c} {
	    $this edges a
	}
    }

    method delete {
	test "Delete single vertex" {
	    $this delete a
	}
	query "Check vertices" {aa b c} {
	    lsort [$this vertices]
	}
	query "Make sure edges not touched" {a b a c b c} {
	    $this edges
	}
	test "Delete non-existent vertex" {
	    $this delete z
	}
	exception {Vertex "z" does not exist}
    }

    behavior "Delete vertex and undo" {
	test "Delete the vertex" {
	    $this delete a
	}
	query "Check vertices" {aa b c} {
	    lsort [$this vertices]
	}
	test "Undo the deletion" {
	    $this undo
	}
	query "Check vertices again" {a aa b c} {
	    lsort [$this vertices]
	}
    }
}

How hierarchical testing works

Hierarchical testing is best explained with a simple example. Suppose we had a CodeComment class, that contained an array of strings. One of its functions is to comment out the comment text that it contains. This operation is generic, in that each line needs to be commented, but it also needs to be specialized by the particular language that we are using.

class CodeComment {
    public variable lines[];
    public abstract String getPrefix();
    public String[] getComment() {
	return lines;
    }
    public void comment() {
	int i;
	for (i=0; i<lines.size; i++) {
	    lines[i] = (String) (getPrefix() + " " + numbers[i]);
	}
    }
}
Now, we cannot instantiate this class, because it is abstract, but we can write the test code for it (let's just ignore the problem of getting text in there in the first place):

Note: this is severely broken: we can't compute the result until we have an object, but the test creates the object and runs the test immediately! Hm.... maybe constructor should construct an object, and test won't create a new one if there exists one that hasn't already been used.

::test::abstractclass CodeComment {
    method comment {
	# Now execute the test
        test "Comment out a line" {
	    $this comment
        }
        # Now compute the result we should have got
        set result {}
        foreach line [$this getComment] {
            lappend result "[$this getPrefix] $line"
        }
	# Now compare with the actual result	
	query $result {
            $this getComment
        }
    }
}
Because this class is abstract, the test framework will not attempt to instantiate an instance of it and run the test on it. However, CodeComment wil have a number of concrete subclasses, which can be instantiated. For example:
class JavaComment extends CodeComment {
    public String getPrefix() {
        return "//";
    }
}
The test suite for this class only looks like this:
::test::class JavaComment {
    inherit CodeComment
    method getPrefix {
        test getPrefix "//" {
            $this getPrefix
        }
    }
}
Now, when the test framework runs the test for the class JavaComment, it not only runs the (very simple) test for the getPrefix method, but also runs the test for the comment method that it inherits from the CodeComment class. Thus, test code written for a class is reused in testing subclasses. This ensures that all classes meet the contract agreed to by their superclasses. It also has the advantage of placing the test code for an abstract class in the corresponding test object -- trivial concrete subclasses can then be written solely to exercise the tests on the functionality provided by the abstract class. This is also the mechanism by which tests for Java interfaces (which are purely abstract) can be written.

The collaboration test

The object-level test is only the first step in testing for a reliable and robust class hierarchy. The next step is to test objects in combination. The technique recommended here is create a separate test object for each identifiable set of collaborating objects. A set of objects that implements a design pattern, for example, is an ideal (and highly visible) candidate for a collaboation test suite. Another way of identifying a suitable set of objects is by functionality: the set of objects needs to perform a certain function of the system need to be tested to see if they do perform it. (Strictly speaking, these are two different kinds of testing, but we just lump them in together.)

Because collaboration relies on the individual objects functioning properly, collaboration tests should only be written when the object-level tests are essentially complete. Because collaboration between object is often defined at an abstract level, the collaboration tests also support the notion of hierarchical testing: A test can be written that can only actually be executed when concrete subclasses become available.

Collaboration tests use a different set of commands to the object tests. The key commands are:

::test::collaboration name ?-language language script
Create a new namespace for testing the collaboration named name. The -language option specifies the language in which the classes are written, and defaults to Java. The script is a script that declares the collbaration tests.
The collaboration script contains a series of declarations about the collaborating objects, and the behaviors which represent particular collaboration scenarios. A well-designed collaboration test is best built based on UML sequence diagrams or interaction diagrams.

The collaboration script can contain the following commands:

behavior description tests
Initialize the framework for testing a behavior -- that is, a sequence of interactions between the collaborators. The current constructor script is executed to create the collaborating objects. The description is a short descriptive string that will appear in the test reports. The tests argument is a script that is executed for this behavior.

collaborator classname ?option value ...?
Declare that the given class is a collaborator. A series of option-value arguments follows the class name, and can be any of:
-abstract boolean
Declare whether the given collaborator is abstract. If it is, this test suite will only be executed when concrete subclasses of the collaborator become available.

constructor description ?script? ?results?
Initialize the framework for testing a constructor of the object. The script argument is a script that is executed to create a set of collaborating objects. results a script that is executed to check that the created objects meet a minimum confidence level, and generally contains a series of query commands.

Each time constructor is executed, script becomes the default construction script to use for subsequent tests. If script is not supplied, then description is looked up in the existing constructors, and its script is used to construct objects for following tests. There is no default constructor for collaborations.

inherit name
A collaboration can extend another collaboration. This happens when a subclass of one or more of the collaborators add more complex behavior to the collaboration. If a collaboration inherits from a another, then at least one of the collaborator clauses must define a subclass of an inherited collaborator. All of the inherited test will be run as well as any defined within this collaboration.

Running tests

The test functions described in the previous sections only serve to define tests -- they do not execute any tests. The test framework defines the following commands for executing and reporting on tests:

::test::verbose flag
If flag is true, output is generated on the console while the tests are executing. Otherwise the tests are executed silently.
::test::reset
Reset the test suite. All test results accumulated so far are cleared from memory.
::test::report ?option value... ?
Generate a string that contains a report of the test result to date. By default, a useful summary of all tests run since the test framework was loaded or rest is produced. The optional arguments can be used to control the output:
-classes { class }
List one or more classes to include in the report. The special keywords all and last stand for all classes tested and the most recent, respectively. The default is all.

-format format
Select an output format. format can be one of short, default, or long. The generated report includes more or less information accordingly.

Infrequently-asked questions

Blah blah blah?
Blah. Blah blah. Blah.

Unresolved issues

Should the test framework check against source code? For example, whether a method is abstract, and whether all methods have been tests provided.
In Java, there is no need to have access to the source code, since the reflection API can be used to gain access to all information about the object under test. Given that, the test framework should definitely check the test suite against the information in the object under test where appropriate. In Itcl, less information is available, although this may change with the next release. In either case, it would be preferable to avoid trying to parse source code in the test framework.

Tycho Home Page


Copyright © 1996-1997, The Regents of the University of California. All rights reserved.
Last updated: %G%, comments to: johnr@eecs.berkeley.edu