When are unit test framemworks useful?

• When there is a framework available for your language
• When your program is split into "units"

These points did not obtain in this case. I was writing a simple Perl program, about 130 lines long. I did not bother to break it out into objects.

On the other hand, the tested program was going to be used in an automated system. I wanted to test all circumstances that might crash it and catch them. This included problems with command-line input, input files, and output files. I wanted to test from the command line, and shell scripts make that easy.
That's why I chose to "roll my own".

There are other advantages to testing this way:

1. It's black box testing. The test does not have to know anything about the program.
2. Simple to do. No need to structure the program to match a particular programming paradigm for the test!

3. It's unusual enough that you get to write about it, and maybe even present the results at an OCLUG meeting!.

Making programs testable

For this situation, making the program testable means making the input and output configurable, so that we can read test data and save test results where we want.

If you are using unit test frameworks, you should write your program so that all the functionality is in objects and classes. Making your code testable will normally make it easier to maintain and write.

Identifying tests to write

What tests can you run on a small program?

Test the interface:

• This program takes two parameters - the source file and the output file
• Normal case
  • Should return 0 ("true" for linux programs, usually)
  • Should create the correct file
  • Should put the correct info in it (compare with expected output file)
• Less than two parameters
  • Should tell you there are not enough parameters
• Too many parameters
  • Should tell you there are too many parameters

When I started writing these tests, I added some extra checks in the program that I hadn't thought of before. When I started running them, I caught a case where the check didn't stop the program when it failed.

Test problems with the input and output files:

• Problem with the input file
  • Non-existent file
  • File exists, but you can't read it
• Problem with the output file
  • Can't write to it for some reason
• Bad data
  • What this means depends on your program
  • if the program is complicated, this may be a lot of tests

Creating data for tests

You can start with a "real" data file. For speed, you may want to shorten it - removing irrelevant chunks if possible. For this program, only a few lines of the real data file are used or checked, so I removed about 90% of the file.

I also created two data files for different data error cases, and a second "pass" case where the data was not in the expected order. This required a change to the program.

Writing the tests

You have choices as to when you should start writing tests. Some say write your tests before writing the related code. Here is an example of that:

I expect to change the XML output of this program, to make it more general. I will change the test output file first, so I can see when I am done.

On another level, don't wait until you have decided on all your test cases before writing any tests. That leads to procrastination.

Test sample

Here is a sample test program, written in shell script. I have kept only two tests here for a sample.

#!/bin/sh
echo -n "Date "
date
echo -n "PWD  "
pwd
echo

STARTDIR=`pwd`
TESTCOUNT=0
TESTPASS=0

function goodtest {
    echo -- Valid input file - should be a pass test
    let TESTCOUNT=$TESTCOUNT+1

    if ! ../../coverage-summary.pl good.html test-good.xml
	then
	echo "* coverage-summary.pl unexpectedly returned false"
	echo "* Test Failed"
	return
    fi
    
    if diff --brief expected-output.xml test-good.xml
	then
	let TESTPASS=$TESTPASS+1
    else
	echo "* Test Failed"
    fi
}

function toofew {
    echo -- Too few input parameters
    let TESTCOUNT=$TESTCOUNT+1

    if ../../coverage-summary.pl > toofew.out 2>&1
	then
	echo "* coverage-summary.pl unexpectedly returned true"
	echo "* Test Failed"
	return
    fi

    if diff --brief toofew.expected toofew.out
	then
	let TESTPASS=$TESTPASS+1
    else
	echo "* Test Failed"
    fi
}

# here is where we actually run the tests
echo "This script is intended to test coverage-summary.pl"
echo

cd $STARTDIR
goodtest
toofew
# toomany
# nofile
# badwrite
# nodatainfile
# difforder

echo
echo -- Test summary
echo Total tests:  $TESTCOUNT
echo Tests passed: $TESTPASS

This test program is intended to have its output piped or captured externally, so it doesn't write the results to a file, just to standard output. This is just one sample of how shell scripts are great for writing tests.

At the start of the program, the script writes info about the test run:

• the current directory
• the date and time

That's just what I thought was useful for these tests, the way I expected to use them. Add more or less if it's appropriate for you. I added the date stamp because the results were going to be used by Cruise Control. If my test program didn't run, and my input file was left over from a previous run, I wanted to have a chance to find that out.

Each test starts by

• printing the name of the test
• incrementing a numeric variable - total number of tests run
  • "let" is a simple way to make math happen in sh
  • if your tests take more than a second or two each, you may want to run only the one you are working on - this variable will remind you
  • yes, several lines of code are repeated in every test

Testing the results

The first "if" actually runs the program and checks the return value. Then:

• you must decide if you want the test to continue if the return value is wrong
• I chose to make sure that the number of tests that pass was NOT incremented by returning

In interface tests, the output of the program is captured and then the second "if" tests it against the expected output. In data tests, the second "if" tests the output XML file against the expected output file for that test.

How do you create an expected output file?

• manually - type it up
• run the test once and copy the actual output to the expected output file
  • remember to read that output file before you copy it (I have personally been caught by this)

Tabulating results

After the test functions, there are two sections:

• Run the tests
  • Note the commented out calls to the functions I removed for brevity
• Report on the results

The report merely prints the two variables:

• total tests run
• how many passed

This is incredibly important.
It allows you to see, at a glance:

• if the numbers don't match, then something failed
• if the numbers are too small, you didn't run all the tests
• it must be simple!

Writing tests from scratch

If you don't have an example handy of how to write tests, just write some.

• Write one test.
• Then write another one.
• Then make a system for yourself to tell you:
  • how many you ran,
  • how many passed,
  • and which ones failed.

Looking forward: Writing your own Really Simple Testing Tools (RSTT)

This test script was written in about half a day, including resulting fixes to the tested program. There are ways to improve it.

For example, you will note that there are duplicated lines in all tests. You can create a setup function and call it. You can create a function to check the right stuff, and pass it a string that is the call to the program.

These steps would simplify your program, and make certain types of mistakes less likely. The functions could be re-used in the next set of tests you write.