Release Management

This page describes how we release software.

The following pages might be of use:

  • Development Tree
  • Testing
  • Building a release

    Splitting the tree

    1. Look for auto directories that don't have corresponding junit/ directories.
      	  cd $PTII
      	  ls -d `find . -name auto | awk -F / '{for(i=1;i<(NF);i++) { printf("%s/", $i);}printf("junit\n")}'`
      								    
    2. Check that each demo has a test:
      	  cd $PTII/ptolemy/configs/doc
      	  make check_tests
      	
    3. Add any new models to $PTII/ptolemy/configs/doc/completeDemos.htm
    4. If necessary, add new demos to toc.htm files
      Run
      	  cd $PTII
      	  $PTII/adm/bin/demoLinkCheck
      	

      Note that not all the applets listed are missing. Some of them use the serial port, JAI, JMF etc. A sample command to create the demoLink lines:

      	  grep ptides $PTII/ptolemy/configs/doc/models.txt | sed 's@$CLASSPATH/@@' | awk '{nf = split($0, f, "/"); print "demoLink(\"\", \"" $1 "\", \"" f[nf] "\");"}' | sed 's@.xml")@")@' | sed 's@.xml",@",@'
      	
    5. Ptolemy handled about:copyright URLS specially. One facility on the copyright page is the ability to expand all the model, .html files and .pdf files on the various demonstration pages. This allows us to verify that all the resources are present in the tree.
      Following the links on about:copyright is an essential part of testing each release.
      For details, see actor.gui.GenerateCopyrights and actor.gui.HTMLViewer
    6. Ptolemy II is organized in to jar files, where each directory contains a jar file of the .class files and another resources that are necessary for that package. These resources include files like images and pdfs. Parent directories contain the jar files in children directories.
      When building a release, it is usually best to build the Web Start release first because it is easier to change one jar file and retry something in Web Start than it is to rebuild an entire Windows installer.
      Use about:copyright to test the Web Start release.
    7. Check out a clean tree and build:
      	  # Check out a clean tree
      	  svn co https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/trunk ptII
      	  cd ptII
      	  export PTII=`pwd`
      	  ./configure
      	  ant
      	
    8. Clean the tree by running make checkjunk and adding files as appropriate to the makefiles:
      	  cd $PTII
      	  make clean_shipping
      	  make checkjunk
      	
      Look for *** lines and add files to makefiles or add them to the clean_shipping rule in $PTII/makefile and the most recent src.ex file in $PTII/adm/gen-x.y/src.ex.
    9. Run $PTII/util/testsuite/chkjava on the java files and fix problems
      Shortcut: Run it only the most recent files added
      Create a list of java files by either running
      
      	  cd $PTII
      	  make cleanDerivedJavaFiles
      	  make clean_shipping
      	
      Or removing the files by hand:
      	  cd $PTII/ptolemy/actor/ptalon
      	  rm `make echo_OPTIONAL_JSRCS`
      	  cd $PTII/ptolemy/data/expr
      	  rm `make echo_OPTIONAL_JSRCS`
      	  cd $PTII/ptolemy/data/unit
      	  rm `make echo_OPTIONAL_JSRCS`
      	  cd $PTII/ptolemy/copernicus/kernel/fragment
      	  rm `make echo_OPTIONAL_JSRCS`
      
      	  # Create a file that contains the names of the java files to include
      	  cd $PTII
      	  rm -rf vendors ptolemy/apps
      	  rm -rf ptolemy/domains/ct/demo/Corba/util/*.java
      
      	
    10. Generate a list of files
      	  adm/bin/ptIItxtfiles > /tmp/f
      	    egrep '*.java$' /tmp/f | grep -v /vendors/ >& /tmp/j
      	
    11. Check for missing makefiles:
      	  cd $PTII/adm/test
      	  make missingMakefiles
      	
      Below are the missing makefiles that are ok to be missing:
      	  ./config/makefile
      	  ./doc/coding/templates/makefile
      	  ./ptolemy/actor/lib/fmi/fmipp/swig/makefile
      	  ./ptolemy/backtrack/automatic/ptolemy/actor/lib/makefile
      	  ./ptolemy/backtrack/automatic/ptolemy/domains/sdf/lib/makefile
      	  ./ptolemy/backtrack/automatic/ptolemy/math/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/actions/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/actions/codestyle/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/compatibility/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/console/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/dialogs/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/editor/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/preferences/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/util/makefile
      	  ./ptolemy/backtrack/eclipse/plugin/widgets/makefile
      	  ./ptolemy/backtrack/ui/makefile
      	  ./ptolemy/backtrack/util/java/util/makefile
      	  ./ptolemy/plot/servlet/makefile
      	
    12. Check for directories that have java files but are not listed in doc/makefile:
      	  cd $PTII/adm/test
      	  make missingDocPackages
      	
      Below are the packages that need not be in doc/makefile.
      	  config
      	  contrib.actor.lib.example
      	  diva.util.java2d.svg
      	  doc.coding.templates
      	  doc.tutorial
      	  doc.tutorial.domains
      	  doc.tutorial.graph
      	  doc.tutorial.graph.junit
      	  doc.tutorial.gui
      	  ptolemy.actor.corba
      	  ptolemy.actor.corba.CoordinatorUtil
      	  ptolemy.actor.corba.CorbaIOUtil
      	  ptolemy.actor.corba.util
      	  ptolemy.actor.lib.javasound.test.pitchshift
      	  ptolemy.actor.lib.mail
      	  ptolemy.actor.lib.reactable
      	  ptolemy.actor.lib.tutorial
      	  ptolemy.backtrack.automatic.ptolemy.math
      	  ptolemy.backtrack.eclipse.plugin.actions.codestyle
      	  ptolemy.backtrack.eclipse.plugin.compatibility
      	  ptolemy.backtrack.eclipse.plugin.console
      	  ptolemy.backtrack.eclipse.plugin.dialogs
      	  ptolemy.backtrack.eclipse.plugin.widgets
      	  ptolemy.backtrack.test.array1
      	  ptolemy.backtrack.test.random1
      	  ptolemy.backtrack.test.test1
      	  ptolemy.backtrack.test.test2
      	  ptolemy.backtrack.util.java.util
      	  ptolemy.caltrop.ddi.util
      	  ptolemy.chic
      	  ptolemy.component
      	  ptolemy.component.domains.ptinyos
      	  ptolemy.copernicus.interpreted
      	  ptolemy.domains.gr.lib.quicktime
      	  ptolemy.plot.servlet
      	
    13. Add trailing newlines to files with $PTII/adm/bin/addtrailingnl
      	  cd $PTII
      	  # Run addtrailingnl without actually doing anything:
      	  cat /tmp/j | xargs $PTII/adm/bin/addtrailingnl -n
      	  # Run addtrailingnl with adding the new lines:
      	  cat /tmp/j | xargs $PTII/adm/bin/addtrailingnl
      	  # Run with -n and then without:
      	  cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/adm/bin/addtrailingnl -n
      	  cat /tmp/f | egrep makefile | xargs $PTII/adm/bin/addtrailingnl -n
      	  cat /tmp/f | egrep README | xargs $PTII/adm/bin/addtrailingnl -n
      	  cat /tmp/f | egrep '*\.htm' | xargs $PTII/adm/bin/addtrailingnl -n
      	  cat /tmp/f | egrep '*\.tcl' | xargs $PTII/adm/bin/addtrailingnl -n
      	  cat /tmp/f | egrep '*\.xml' | xargs $PTII/adm/bin/addtrailingnl -n
      	
    14. Convert tabs with $PTII/adm/bin/rmtabs
      	  cd $PTII
      	  # Run rmtabs without actually doing anything:
      	  cat /tmp/j | xargs $PTII/util/testsuite/rmtabs -n 
      	  # Run rmtabs with adding the new lines:
      	  cat /tmp/j | xargs $PTII/util/testsuite/rmtabs 
      	  # Run with -n and then without:
      	  cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/util/testsuite/rmtabs -n
      	
    15. Remove trailing spaces with: $PTII/util/testsuite/rmtrailingspace
      	  cd $PTII
      	  # Run rmtrailingspace without actually doing anything:
      	  cat /tmp/j | xargs $PTII/util/testsuite/rmtrailingspace -n 
      	  # Run rmtrailingspace with adding the new lines:
      	  cat /tmp/j | xargs $PTII/util/testsuite/rmtrailingspace 
      	  # Run with -n and then without:
      	  cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/util/testsuite/rmtrailingspace -n
      	
    16. Fix common problems in java files:
      	  cat /tmp/j | xargs $PTII/adm/bin/fix-files -n
      	
    17. On a Unix box, check for files with control-Ms:
      	  cat /tmp/f | xargs $PTII/util/testsuite/controlm > /tmp/cms
      	

      To fix the files, run:

      	  grep "has controlms" /tmp/cms | awk '{print $1}' > /tmp/cmfs
      	  svn propset svn:keywords "Author Date Id Revision" `cat /tmp/cmfs`
      	  svn propset svn:eol-style native `cat /tmp/cmfs`
      	  svn commit -m "Added keywords and eol-style properties" `cat /tmp/cmfs`
      	
      You may need to use dos2unix to fix the files.

      Check keywords in all files:

      	    cd $PTII
      	    cat /tmp/f | xargs $PTII/adm/bin/svnpropcheck > /tmp/props
      	    grep keywords: /tmp/props | grep / | awk '{print $1}' | egrep -v '.xml$' > /tmp/keys
      	    grep eolstyle: /tmp/props | grep / | awk '{print $1}' | egrep -v '.xml$'> /tmp/eols
      	    cat /tmp/keys | xargs svn propset svn:keywords "Author Date Id Revision"
      	    cat /tmp/eols | xargs svn propset svn:eol-style native 
      	    svn commit -m "Added keywords and eol-style properties" `cat /tmp/eols /tmp/keys`
      	  
    18. Run $PTII/adm/copyright/fixcopyrights (not shipped with the release) and update the copyrights on the .java, .tcl and makefiles
    19. Run $PTII/adm/copyright/chkcopyright on all the nightly buildfiles.
      Do:
      	  cat /tmp/f | grep -v .class | grep -v codeDoc | grep -v xml | grep -v \
      	  alljtests.tcl | grep -v .htm | grep -v qcf | grep -v .dtd | grep -v \
      	  gif | grep -v .svn | grep -v jimblackler | grep -v openmodelica/lib/omc/corba \
      	  grep -v metroIIcomm.java | grep -v swig | grep -v shallow/test/*2java | xargs sh $PTII/adm/copyright/chkcopyright >&  /tmp/copycheck 
      	  grep "does not contain a Copyright" /tmp/copycheck | awk '$1 ~ /.java$/ {print $1}' | grep -v /actor/corba >/tmp/jc
      	

      Review /tmp/jc and remove files written by others that should not get the UCB copyright. Add these files to the commands above.
      Then, edit $PTII/adm/copyright/copyright.txt and $PTII/adm/copyright/insertcopyright and run
      	  adm/copyright/insertcopyright -p `cat /tmp/jc`
      	
      and then
      	  adm/copyright/insertcopyright `cat /tmp/jc`
      	
    20. Check out all the files again so we can build a release:
      	  cd $PTII
      	  svn update
      	  ant
      	  ant jars
      	
    21. Run the nightly build, build a release:
      For each release, we copy the most recent $PTII/adm/gen-X.Y, update the version number and run make. See the top of the file for the appropriate make command.
    22. Check the copyright on new files First, untar the previous source release in ptII/adm/dists.
      	  cd $PTII/adm/dists
      	  wget http://ptolemy.eecs.berkeley.edu/ptolemyII/ptIIlatest/ptII8.0.1.src.tar.gz
      	  tar -zxf ptII8.0.1.src.tar.gz
      	
      Then check for files that don't have this year's copyright
      	  cd $PTII/adm/dists
      	  diff -r ptII8.0.1 ptII10.0.1 > /tmp/pt.diffs
      	  cd $PTII
      	  grep Copyright `egrep "^Only in ptII10.0.1" /tmp/pt.diffs | \
      	  grep .java | sed 's@ptII10.0.1/@@' | awk '{print substr($3,1,length($3)-1) "/" $4}'` | grep -v `date +%Y` > /tmp/c2
      	
      Look over the output for files that should have their copyright updated See $PTII/adm/copyright/fixcopyright, but basically:
      	  cat /tmp/c2 | grep Regents | awk -F : '{print $1}'
      	
    23. Run $PTII/adm/bin/fixsince (not shipped with the release) and update the @since tags. To use fixsince, first edit $PTII/adm/bin/fixsince and update the variables at the top. Then
      	  cd $PTII
      	  $PTII/adm/bin/fixsince
      	
      Then check the output. Then edit the file and uncomment the cp line at the bottom:
      	  #cp $tmpfile $file
      	
      Then edit fixsince and comment out the cp line and then commit the changes.
    24. Run cd $PTII/doc; make chkpackagehtml (not shipped with the release) and create any missing package.html or README.txt files.
      	  cd $PTII/doc
      	  make chkpackagehtml > /tmp/s
      	  cd $PTII
      	  $PTII/adm/bin/mkpackagehtml `cat /tmp/s`
      	  svn add `egrep '(README.txt|package.html)' /tmp/s | awk '{print $2}'`
      	  svn commit -m "Added package.html and README.txt" `egrep '(README.txt|package.html)' /tmp/s | awk '{print $2}'`
      	
    25. Concatenate all the javadoc files in to one file and run $PTII/util/testsuite/ptspell.
      	  cd $PTII/doc
      	  make docs
      	  cd codeDoc; find . -name "*.html" > /tmp/h
      	  cat /tmp/h | xargs cat > /tmp/docs
      	  cat /tmp/docs | aspell list -H --run-together-limit=10 --personal=$PTII/util/testsuite/ptlocaldict | sort -u | 	sed -e 's/^\([A-Z]*\)\([A-Z]\)/\1 \2/g' | 	sed -e 's/\([a-z]\)\([A-Z]\)/\1 \2/g' | aspell list -H | sort -u
      	
    26. Run findbugs and look at the results:
      	  cd $PTII/doc
      	  make findbugs.htm
      	
    27. Consider using PMD or
    28. Use Eclipse to fix the imports of the entire tree.
    29. Use Eclipse to add @Override to appropriate methods.
    30. Consider fixing other Eclipse warnings.
    31. Use Eclipse Cleanup to clean the code
      1. Set up cleanup by going to the preferences window in Eclipse:
        (Windows: Window | Preferences)
        (Mac OS X: Eclipse | Preferences)
      2. Java | Code Style | Clean Up
      3. New, then type in a Profile name of "Ptolemy II" then hit OK
      4. In the "Profile 'Ptolemy II'" window:
        • Code Organizing: select "Format source code",
          select "Remove trailing whitespace",
          select "Correct indentation",
          select "Organize Imports".
        • Code Style: select "Use blocks in if/while/for/do statements"
        • Missing code: unselect "Add missing Annotations"
        • Unnecessary Code: select "Remove unused private members"
          select "Remove unused local variables"
      5. Save the cleanup settings by closing the Cleanup preferences window
    32. Consider using GCJ: The GNU Compiler for Java to report unused variables and other problems. Note that GCJ3.4 will fail to compile awt and swing classes, but it is still worth getting the other error messages.
      cd $PTII make clean make -k JAVAC=gcj JFLAGS="-O2 -Wunused -Wextraneous-semicolon -Wno-deprecated -C" all >& gcj.out & grep error: gcj.out | awk -F: '{for(i=3;i<=NF;i++) {printf("%s ",$i)} printf("\n");}' | sort | uniq | grep -v "not found in the decl"
    33. Consider building with the IBM JDK
    34. See Installers for how to build Windows installers. Building the Windows installer is fairly complex, below are some things to be aware of
    35. The Windows installer includes source files. These sources come from $PTII/adm/gen-N.M/ptIIN.M.src.jar. Obviously, this jar file should be created on a Windows machine so the end of line characters are Windows end of line characters.
    36. Look for files with bad permissions
      	  cat /tmp/f | egrep "(.java$|.htm$|.html$|makefile$|.tcl$|.xml$)" | xargs ls -l | awk '{ if ($1 ~ /x/) print $NF}' | grep -v chkjava > /tmp/execs
      	    cat /tmp/execs | xargs svn proplist
      	    cat /tmp/execs | xargs svn propdel svn:executable
      	    svn commit -m "Made regular .java files and other files not executable" `cat /tmp/execs`
      
    37. Tag the tree as the release tree and create a branch

      To make a branch for 10.0.1, use the following command:

            svn copy https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/trunk https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/branches/rel-10.0.1 -m "Creating Ptolemy II 10.0.1 branch."
          
      Then check out the release branch with
            svn co https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/branches/rel-10.0.1 ptII10.0.1
          
      Note that you should avoid dots in the tag name
    38. Update the version numbers, see $PTII/makefile and $PTII/ptolemy/plot/makefile.
    39. Remove files that are not to be shipped from the release:
      	cd ptII10.0.1
      	export PTII `pwd`
      	./configure
      	make svn_delete_clean_shipping
            
    40. The nightly build uses a makefile in ~ptII/adm (not shipped) to build the release. Copy the most recent makefile and set up a new tree for the release tree.
    41. We build a separate src.tar.gz file under Unix so that the line endings are correct.
              svn co https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/branches/rel-10.0.1 ptII10.0.1
      	cd ptII10.0.1
      	export PTII=`pwd`
      	export JAVA_HOME=/usr/lib/jvm/java-1.6.0
      	export PATH=${JAVA_HOME}/bin:${PATH}
      	./configure
      	ant
      	ant jars
      	cd $PTII/adm/gen-10.0
      	make release_sisyphus
            
      The resulting file with the .class files gets copied to the website and untar'd. Note that this does not require Java 3D on the build machine. Java 3D is needed for the applets, we copy the jars over later. While in $PTII/adm/gen-10.0, run:
      	make release_sisyphus_update_ftp
            
      1. On moog, fix the links to ptplot downloads:
        	    cd /home/www/ptweb/ptolemyII/ptII10.0/ptII10.0.1/ptolemy/plot
        	    ln -s ../../../../../java/ptplot5.10/ptolemy/plot/ptplot5.7* .
        	    cd doc
        	    rm -rf download
        	    ln -s ../../../../../../java/ptplot5.10.beta/ptolemy/plot/doc/download .
        	  
      2. Copy gr.jar from a Windows box to ptolemy/domains/gr
      3. Copy codeDoc.jar from a Windows box and unjar it
    42. We build the webstart jar files under Windows
            svn co https://repo.eecs.berkeley.edu/svn/projects/eal/ptII/branches/rel-10.0.1 ptII10.0.1
      	cd ptII10.0.1
      	export PTII=c:/tmp/cxh/ptII10.0.1
            
      Set up missing packages such as joystick, JAI, JMF, javacomm, quicktime Use an older javac so that the .class files are compatible. For example, use java 1.6.x so that java 1.7.x and 1.8.x can use the .class files. However, Javadoc from 1.6.0_26 hangs while running, so use a more recent javadoc.
       	  export PATH=/cygdrive/c/Program\ Files/Java/jdk1.6.0_26/bin:${PATH}
       	  export JAVADOC=c:/Program\ Files/Java/jdk1.8.0_11/bin/javadoc
      	  java -version
      	  ./configure 
      	  ant
      	  (cd $PTII/doc; make codeDoc/tree.html)
      	  ant jars
      	
    43. Build the webstart files. This is complex . . .
      The high level view is that we export the models as HTML by running a test using ant,then we export the models as WebStart JNLP files and then copy the appropriate files over to the web server.
    44. Create the webstart and installer files.
      	    cd $PTII
      	    make jar_dist
      	  
      Look for missing jar files in the output.
    45. Create the src.jar file:
      	    cd $PTII/adm/gen-10.0
      	    make USER=cxh PTIIHOME=/cygdrive/c/cxh/ptII10.0.1 COMPRESS=gzip TAR=tar clean all src.jar 
      	  
      Below is the command to compare the old tree and the new tree:
      	    diff -r -I '$Id:' -x alljtests.tcl -x allsimplejtests.tcl -x '*.jar' -x codeDoc -x '*.out' -x .svn -x '*.class' ~/ptII ~/src/ptII10.0.1 >& /tmp/pt.diffs
      	  
    46. Under Windows, build the installer.
      	    cd $PTII
      	    make exes
      	    cd $PTII/adm/gen-x.y
      	    make setup
      	  
    47. Test the installer

    Last Updated: $Date: 2014-12-16 13:52:11 -0800 (Tue, 16 Dec 2014) $