Testing Tycho


This page is primarily for Tycho Developers. Some of the commands mentioned below are not included in the Tycho distribution.

Contents:

  • Test Suite
  • Testing Java
  • Testing Documentation
  • Testing the tycho Script
  • Test Suite

    We have included regression tests for some of the code. Usually, wherever there is a Itcl file, the tests are in the test directory.

    Running the tests

    There are several ways to run the tests:
    1. The BuilderDialog has a Test menu which has choices for the tests arranged by directory.
    2. Open a test file such as $TYCHO/kernel/basic/test/testStack.itcl with Tycho and then evaluate the contents of the file with the Evaluate choice from the File menu (C-x C-r).
    3. The $TYCHO/mk/tycommon.mk file automagically generates two Itcl files:
      all.itcl
      This file merely sources all of the *.itcl files in the current directory.
      alltests.itcl
      This file sources all of the files that are listed as test files in the makefile.
      After running make sources, you can open up either of these files and then evaluate it, which will run all the tests in that directory.
    4. $TYCHO/kernel/gui/test/alltychotests.tcl
      alltychotests.tcl will run all the alltests.itcl files. $TYCHO/makefile has two rules:
      make alltests
      Run all the tests in a vanilla tycho
      make alltests.ptiny
      Run all the tests in a tycho -ptiny
      If you are on a machine that does not have make, you can try
      source [file join $TYCHO kernel gui test alltychotests.tcl]
      
    5. The tests can also be run by changing to a test directory and typing make from a shell. To try this out, click on the Tcl code below:
      ::tycho::execModal {make} "[file join $TYCHO kernel gui test]"
      
    6. The Tycho test suite can be run from the Tycho top-level ($TYCHO) directory with the command: make tests

    Writing your own tests

    The test suite infrastructure is based on the Itcl test suite code, which is in turn based on the Tcl test suite code. The Itcl Editor has a Test Template under the Special menu. This menu choice loads the file $TYCHO/edit/textedit/templates/testTemplate into the Itcl editor.

    The file $TYCHO/kernel/gui/test/testDefs.tcl defines the Tcl proc test.

    test takes four arguments:

    1. The name of the test, for example: foo-1.0
      The name of the test should strictly follow the format below. The Tcl tests that come with the Tcl distribution follow a similar format, so unless there is a strong need to not follow the format, please stick with what works.
    2. The first part of name of the test should reflect the command that is being tested.
    3. The test number should be separated by a dash '-'
    4. Each test number consists of a major value and a minor value, separated by a dot. Usually the major value changes as different parts of the command are being tested. The minor value changes for different tests for the particular part of the command under test.
    5. Test numbers usually start with 1, though if you are doing setup, you can start with 0.
    6. If you go back later and want to stick a test in between foo-1 and foo-2, you can always call your new test foo-1.1
    7. The test description, usually a single sentence.
    8. The contents of the test, usually Tcl code that does the action to be tested. The last line of the contents should return a value.
    9. The results to be compared against.
    Below is a sample piece of code that sources the testDefs.tcl file and then runs one test. The code below has the incorrect value return results to be compared against, so the test suite properly indicates that the test failed.
    if {[string compare test [info procs test]] == 1} then { 
        source [file join $TYCHO kernel gui test testDefs.tcl]
    } {}
    test testExample-1.1 {This is the first test example, it does very little} {
    	catch {this is an error} errMsg1
    	set a "this is the value of a"
    	list $errMsg1 $a
    } {{invalid command name "this"} {this is NOT the value of a}}
    

    Parts of a test file

    Test files should be located in the test directory.

    The EditItcl class has a test file template menu choice under the Special menu. The template is located at $TYCHO/edit/textedit/templates/testTemplate.itcl

    It is better to have many small test files as opposed to a few large test files so that other developers can quickly find the tests for the class they are working with. Usually tests for the class Foo are found in the file test/testFoo.itcl

    See $TYCHO/kernel/gui/test/testGraph.itcl for an example test file.

    Each test file should have the following parts:

    1. The name of the test file should begin with the word test. For example, if we were testing the Tcl foo command, then the file would be named testFoo.itcl. The extension of the file is either .tcl or .itcl:
    2. .tcl if the file can be sourced in a tclsh that does not contain Itcl. $TYCHO/kernel/gui/test/testPath.tcl is an example.
    3. .itcl if the file contains Itcl constructs or tests Itcl classes.
    4. Most of the tests are .itcl files.
    5. The Copyright
    6. The code that loads the test system package
      package require tycho.util.tytest
      
    7. A line that the user can uncomment if they want the test system to produce verbose messages:
      #set VERBOSE 1
      
    8. The individual tests, which should loosely follow the Tycho file format standard:
      ############################################################################
      #### Foo
      test Foo-1.1 {Test out Foo} {
      
      } {}
      

    Test Styles

    There are two types of tests:
    1. Tests that handle all necessary setup in each individual test.
    2. Tests that rely on the earlier tests to do setup.
    In general, each test file should be able to be run over and over again in a binary without exiting the binary (it should be idempotent).

    It is up to the author of the tests as to whether each individual test does all the set up necessary. If each test is atomic, then it makes it easy to highlight the text of an individual test and run it. If lots of tests are sharing common setup, then using a separate procedure to do setup might help. On the negative side, atomic tests usually are longer and have more complicated return results.

    Testing Modal Dialog Widgets

    Testing modal dialog widgets and the methods that use them is tricky because we want the test suite to be able to just 'do the right thing' and invoke the actions of an arbitrary method.

    The problem is that when a model dialog box comes up, execution halts, so during testing the user would need to know what button to hit. To work around this, the Dialog::wait procedure (which is user to wait for the result of all modal dialog boxes in Tycho) checks for the TY_TESTING global variable, to see if it has been called from within a test script. If so, then Dialog::wait will not actually make the dialog model but will do one of two things:

    For example, the $TYCHO/kernel/gui/test/testWidgets.itcl script includes a test for YesNoQuery:

    if {[string compare test [info procs test]] == 1} then { 
        source $TYCHO/kernel/gui/test/testDefs.tcl
    } {}
    
    test YesNoQuery-2.1 {Test the modal ::tycho::askuser proc} {
        set TY_TEST_MODAL {
            {
    	    %0 invoke yes
    	}
        }
        ::tycho::askuser "Are you awake?"
    } 1
    

    Note that TY_TEST_MODAL must be a list of scripts, even if the list contains only one script. Here is another sample test that simulates the user interaction to a series of two dialog boxes. The code being test is an example from the Dialog Classes tutorial:

    if {[string compare test [info procs test]] == 1} then { 
        source $TYCHO/kernel/gui/test/testDefs.tcl
    } {}
    
    test sampleTest-1 {Test a series of modal dialogs} {
        # Set the series of response to the dialogs. The
        # first one clicks on the Yes button, the second clicks
        # on the OK button. Note that comments
        set TY_TEST_MODAL {
            {
                # This script simulates pressing the Yes button
                # of the YesNoQuery box
                %0 invoke yes
    	}
            {
                # This script simulates pressing the OK button
                # of the Message box invoked by :tycho::inform
                %0 invoke ok
    	}
        }
        # Now execute the code that creates the modal dialogs.
        # In this case, the result is null.
        ::tycho::YesNoQuery .z -text {Are you awake?}
        if [::tycho::Dialog::wait .z] {
            ::tycho::inform {You said yes!}
        } {
            ::tycho::inform {Then how did you click on the No button?}
        }
    } {}
    

    Using event generate

    The Tk event generate command is very handy for testing widgets by generating an event in a window as if it had come from the window system.

    The tests below are doing an <Enter> and <Leave> of a button and checking the text placed into a status bar:

    update
    test ButtonBox-3.3 {Enter button and check status bar} {
        event generate [.bb component buttonfoo] <Enter>
        .sb get
    } {The Foo button}
    
    update
    test ButtonBox-3.4 {Leave the button} {
        event generate [.bb component buttonfoo] <Leave>
        .sb get
    } {}
    

    Writing Testable Code

    Below are some hints about writing code that can be tested.
  • If a method creates a window, and the window is present after the method ends, then the method should probably return the name of the window. This makes it easier for callers and testing scripts to perform operations on the window.
  • Separating interface and implementation can help. If a largish operation brings up lots of windows, it can help if there is a method that performs the interface operations which then calls a series of smaller methods. It is easier to test a series of small scripts than to test a large monolithic method.

  • Testing Java

    Tcl Blend is Sun's Tcl/Java interface. We use Tcl Blend to test Java by writing Tcl code that exercises the Java classes.

    Running the tests

    If you run make in a test directory that contains tests written in Tcl for testing Java classes, then the 'right thing' should just happen. The Java testbed does have a few differences:
    The make test_jsimple rule
    This rule will run only the Java tests in a tclsh binary that can load the Tcl Blend extension
    The alljtests.tcl file
    This file will run all the Java tests that require Tcl Blend. The reason we have yet another file is that Tcl Blend will not necessarily work on all platforms, and we want to be able to run the non-Tcl Blend tests on those platforms.

    Writing Tests for Java

    Below we discuss some of the details of writing tests in Tcl that test Java classes.

    Simple Example

    Tcl Blend allows us to instantiate objects in a class and call public methods. We use Tcl Blend and the standard Tcl test bed to create tests. In the example below, we call java::new to create an instance of the Java NamedObj class. We can then call public methods of NamedObj by referring to the Java object handle $n:

    test NamedObj-2.1 {Create a NamedObj, set the name, change it} {
        set n [java::new pt.kernel.NamedObj]
        set result1 [$n getName]
        $n setName "A Named Obj"
        set result2 [$n getName]
        list $result1 $result2
    } {{} {A Named Obj}}
    

    Java Tcl Test Files

    It is best if each Java class has a separate Tcl file that contains tests. The base of the name of the Tcl test file should be the same of the Java class being tested. The Tcl test file should be located in the test subdirectory of the directory where the Java class is defined.

    For example, if we are testing NamedObj.java, then the Tcl test file should be at test/NamedObj.tcl.

    getJavaInfo Testing Proc

    Tcl Blend provides a Tcl procedure called java::info which returns information about an instance of a Java class. $TYCHO/kernel/gui/test/testDefs.tcl defines the getJavaInfo Tcl proc, which takes a Java object handle and returns a string that describes the object's fields, methods constructors, and superclass.

    Each Java class should have a Tcl test that calls getJavaInfo so that if the Java class definition changes, then the test will fail. This will hopefully prompt the developer to write a test for the changed feature.

    If a test fails because of changes in the values returned by getJavaInfo, then the way to fix the test is to run the test and grab the strings actually returned and substitute it back into the test file. Of course, you should write tests that cover any changes reported by getJavaInfo.

    JavaScope

    JavaScope is Sun's Java code coverage tool.

    Here's how to review the test suite code coverage:

    1. Run:
      cd $TYCHO/java/pt/kernel
      make jsall
      
      The jsall makefile rule does the following:
    2. JavaScope uses a program called jsinstr to instrument the Java files. jsinstr copies the original files to the jsoriginal directory and then adds Java function calls to copies of the files. These function calls increment counters in a database at runtime.
    3. To compile the instrumented classes the JavaScope.zip file is added to the CLASSPATH and then the classes are recompiled.
    4. The test suite is run.
    5. To views the code coverage run either jsreport or javascope
      jsreport -dbsummary
      
    6. To restore the files back to the original state, run make jsrestore.
      make jsrestore
      

    JavaScope Details

  • jsinstr saves the original Java files to the jsoriginal directory. The jsall makefile rule checks to see if this directory is present, and if it is not, runs jsinstr. The jsrestore rule runs jsrestore and then attempts to remove the jsoriginal directory so that the next run of the jsall rule will recreate it. If you are running jsinstr by hand on files that are not in the makefile, then you may find it necessary clean out the jsoriginal directory with:
    jsrestore *.java
    rm jsoriginal/README
    rmdir jsoriginal
    
  • Javascript writes the code coverage information to ~/jsdatabase. You might find it helpful to remove this directory periodically.
  • Flushing the JavaScope database

    The JavaScope database must be flushed by hand at the end of a run, or the code coverage data of the run will not be dumped out to disk.

    There are two ways to do this.

    1. The IFLUSHCLASS option will flush out the coverage information for the current class at the end of each method.
    2. The IFLUSH option will flush out all the coverage information at the end of each method.
    Obviously, using IFLUSH will take more time than using IFLUSHCLASS.

    The jsintr options are set by consulting the following resources in order.

    1. system properties - environment variables
    2. $HOME/javascope.properties
    3. ./javascope.properties in the current directory.
    4. Options can be embedded in files as comments:
      /*jsoptions: ...*/
      
      Embedded comment options can be set more than once in a file. Embedded comment options are in effect until is the option is changed by another embedded comment option.
    5. Command line arguments to jsinstr.
    See the jsinstr documentation for details. I tried to use Tcl Blend to call a dummy Java method that flushes the database using an embedded comment, and then call that method before exiting.

    Here's how I attempted to flush the database using a dummy Java method.

    I created the JavaScopeFlush class:

    public class JavaScopeFlush {
        /*jsoptions:IFLUSH=true*/
        public static void flush() {
        }
        /*jsoptions:IFLUSH=false*/
    }
    

    Ran jsinstr on JavaScopeFlush.java

    I then tried to call this method with:

    java::call JavaScopeFlush flush
    
    Unfortunately, it looks like embedded comments are broken.

    For more information, see the JavaScope FAQ.


    Testing Documentation

    The Tycho documentation is written in HTML. There are several tools that can be used.

    weblint

    Weblint tells the user about html errors. Weblint can be obtained from ftp://ftp.cre.canon.co.uk/pub/weblint/weblint.tar.gz To run weblint:
    cd $TYCHO
    make weblint
    

    htmlchek

    Htmlchek is another tool that tells the user about html errors. htmlchek also checks for bad links. The htmlchek output is a little hard to read, so we tend to use weblint for checking individual files. htmlchek can be obtained from ftp://ftp.cs.buffalo.edu/pub/htmlchek/

    The best way to run htmlchek is to create a sample distribution, create the files in the codeDoc directory and then run htmlchek

    1. Create the test distribution:
      cd /users/ptdesign/adm/gen-latest; make htmlchek
      
    2. Reset TYCHO to point to the test distribution:
      setenv TYCHO /users/ptdesign/adm/dists/tycho-latest
      cd $TYCHO
      
    3. Run make install. This will make the Itcl HTML docs twice, which will populate the doc/codeDoc directories. You need to make the Itcl HTML docs twice so that the cross references are correct.
    4. Run make htmlchek
    The output ends up in five files
  • htmlchekout.ERR - HTML usage errors
  • htmlchekout.NAME - Locations in the specified files that ware not referenced by any of those files
  • htmlchekout.HREF - References from the specified files that are not found in the files. This file is by far the most important file to look at.
  • htmlchekout.SRC - References to online images.
  • htmlchekout.MAP - Cross dependency information.
  • All of the references in htmlchekout.HREF that point to .html files should be checked. References to non-HTML files appear in htmlchekout.HREF because the non-HTML files were not included in the list of files that htmlchek ran on. One quick way to search all the the *.html files is

    cd $TYCHO
    grep mystring `find . -name "*.html" -print`
    

    Spellchecking

    Checking the spelling in all the HTML files can be done with:
    cd $TYCHO
    spell `find . -name "*.html" -print`
    
    You can spell check an individual HTML file by using the HTML editor built-in spell checker under the Edit menu.

    Checking Embedded Tcl

    Tycho extends html with <TCL> . . . <TCL> tags. In the Tycho HTML viewer, when the contents of these tags are moused on, the embedded Tcl code is executed.

    To test all of the embedded Tcl, the script $TYCHO/adm/bin/chktclpre will generated a tclpre.html file that contains all of the embedded Tcl. This script is not shipped with Tycho.

    After running chktclpre, start up Tycho, view tclpre.html and mouse on each block of embedded Tcl code. Note that some blocks depend on the widgets created in earlier blocks. If you run into problems, you can follow the hyperlink in tclpre.html to the source HTML file.

    Checking for bogus copyright SCCS keywords

    Run ~ptdesign/adm/copyright/chkq $TYCHO to print out the names of files that might have bogus %Q% values.

    Then run ~ptdesign/adm/bin/sccsadmin on each file that has no value for %Q% substituted in.

    Check the distribution for bogus files

    Run the following makefile rules and commands:
    make realclean
    This will remove the tclIndex files and the files in doc/codeDoc. The reason to remove the codeDoc files is so that we don't ship HTML files for any classes that have been removed.
    make install
    This will recreate the tclIndex files and the doc/codeDoc files.
    make checkjunk
    Look for files in the distribution that should not be there.
    adm/bin/chkgifs
    This file looks for gif files that are not used by HTML files in the distribution.

    Testing the tycho script

    The $TYCHO/bin/tycho script can be tested with the $TYCHO/kernel/gui/test/chktycho script.

    This script calls tycho with all the various combinations of command line arguments. chktycho adds a line to ~/.Tycho/tychorc.tcl so that tycho is exited after a few seconds. This allows the script to test many invocations of tycho without user intervention. However to test the -debug option, the user must type cont in the gdbwindow.

    Tycho Home Page


    Copyright © 1996-1998, The Regents of the University of California. All rights reserved.
    Last updated: 06/08/98, comments to: tycho@eecs.berkeley.edu