This document describes the "why's" and "how's" of unit testing for the AppKit team of the OpenBeOS project. Although it is intended for the AppKit team, there is no reason other teams couldn't use this information to develop a similar unit testing strategy.
The document has the following sections:
Unit testing is the process of showing that a part of a software system works as far as the requirements created for that part of the system. Unit testing is best if it has the following characteristics:
Unit testing is not the only type of testing but is definitely a very important part of any testing strategy. Following unit testing, software should go through "integration testing" to show that the components work as expected when put together.
A basic concept of software engineering is that the cost of fixing a bug goes up by a factor of 2-10x (depending on the source of the information) the later in the development process it is found. Unit testing is critical to finding implementation bugs within a particular component as quickly as possible.
Unit testing will also help to find requirements problems also. If you write the requirements (or use cases) for your component from the BeBook, hopefully the BeBook and your use cases will match the actual Be implementation. A good way to confirm that the BeBook documentation matches Be's implementation is to write your unit tests and run them against the original Be code.
Unit tests will also continue to be maintained and run in the future also. As the mailing lists obviously show, many people are looking forward to OpenBeOS post-R1 when new features will be introduced above and beyond BeOS R5. These unit tests will be critical to ensuring that any new feature or even just a bug fix doesn't break existing functionality.
Speaking of bug fixes, consider adding unit tests for any bugs you identify that slipped through your original unit test suite. This will ensure that this bug or a similar one is not re-introduced in the future.
Finally, unit testing is not the be all, end all of testing. As mentioned above, integration testing must be done to show that software components work together. If all unit tests cover all requirements and have run successfully against all components, then a failure has to be due to a bug in the interaction of two or more known working software components.
As the AppKit process document describes the recommended order for implementing a component is:
Please see the AppKit process document for more details about the entire sequence. The unit test are to be written once the use cases are written and before any implementation work is done. The use cases must be done because they determine what the tests will be. You need to write as many tests are required so that all use cases for that component are tested. The use cases should be detailed enough that you can write your unit tests from them.
The unit tests are to be done before implementation for a very good reason. You should be able to run these unit tests against the Be implementation and confirm that they all pass. If they do not pass, then either there is a bug in the unit test itself or you have found a difference between your use cases and the actual implementation. Even if your use cases match the BeBook, if that is not how the actual Be implementation works, we must match the current implementation and not the BeBook. You should go back and modify the use case. Change the use case so that it matches Be's implementation and consider adding a note indicating this doesn't match the BeBook.
Imagine if you completed the implementation and then wrote and ran the unit tests. If you run the tests against your implementation and Be's implementation, you will notice the test passes for your code but fails on Be's. At this point, you will have to change the implementation, change the unit test and change the use case which is more work that if you write the unit tests before the implementation. Worse, if you only ran the unit tests against your implementation and not Be's, you may not notice the problem at all.
The unit tests you write should cover all the functionality of your software module. That means your unit tests should include:
The AppKit team has chosen to use CppUnit version 1.5 as the basis of all of our unit tests. This framework provides very useful features and ensures that all unit tests for AppKit code are consistent and can be executed from a single environment.
There are two key components to the framework. First, there is a library called libCppUnit.so which provides all the C++ classes for defining your own testcases. Secondly, there is an executable called "TestRunner" which is capable of executing a set of testcases.
For more information on CppUnit, please refer to this website
The following are the modifications that have been introduced into the CppUnit v1.5 framework:
This is the list of the important modifications done to CppUnit v1.5 at the time this document is being written. For the latest information about modifications to CppUnit, check the code which can be found in the OpenBeOS CVS repository.
This framework will have to evolve as our needs grow. The main issues I think we need to solve are:
If you find you need some other features, feel free to add them to CppUnit.
As of writing this document, you can build the framework and all the current AppKit tests by performing the following steps:
Note that the build system for OpenBeOS is moving to jam so these steps may become obsolete. When you the make has finished, you should find the following files:
app_kit/test/CppUnit/TestRunner - this is the executable to use to execute
tests.app_kit/test/CppUnit/lib/libCppUnit.so - this is CppUnit library which your tests
must link against.app_kit/test/CppUnit/lib/libopenbeos.so - this is library which contains OpenBeOS
implementation of some Be classes (usually found in libbe.so, called libopenbeos.so to avoid a name
clash at runtime).app_kit/test/add-ons/BAutolockTests - this is the addon which contains the tests
which are run against the Be and OpenBeOS implementation of BAutolock.app_kit/test/add-ons/BLockerTests - this is the addon which contains the tests
which are run against the Be and OpenBeOS implementation of BLocker.app_kit/test/add-ons/BMessageQueueTests - this is the addon which contains the tests
which are run against the Be and OpenBeOS implementation of BMessageQueue.These are the key files which ensure that the tests can be run.
You have a few different options for how you run a test or a series of tests. Before you start however, you must build the code as describe in this section. Once it is built, you can run tests any of these ways:
The first step to writing your tests is to develop a plan for how you will test the functionality. For ideas of the kinds of tests you may want to consider, you should reference this section.
Once you know the kinds of tests you want, you need to:
For every test you want, define a class which derives from the "TestCase" class in the CppUnit framework.
Within each test class you define, create a "void setUp(void)" and "void tearDown(void)" member function if required. If before executing your test, you need to perform some actions, put those actions in the "setUp()" member. If you need to cleanup after your test, put those actions in the "tearDown()" member.
Within each test class you define, create a member function which takes "void" and returns "void". Within this member function, write the code to execute the test. Whenever you want to ensure that some condition is true during your test, add a line within the member function that looks like "assert(condition)". For example, if the variable "result" must have the value B_OK at a particular point in your test, you should add a line which reads "assert(result = B_OK)".
Create a constructor for all of your test classes that takes a "std::string name" argument and pass that onto the TestCase parent class. Add whatever actions you need to take in the constructor.
Create a destructor for all of your test classes and take whatever actions are appropriate.
Within each test class you define, create a member with the signature "static Test *suite(void)". For a simple test where only one test needs to be run for this class, the contents of this member should look like:
return(new TestCaller<ClassName>("", &ClassName::MemberName));
Replace "ClassName" with the name of your test class and "MemberName" with the name of the member function you defined your test in. If you need to define more than one test to run from this class, refer to instructions below on how to use the TestSuite class of CppUnit. If you are creating a threaded test, refer to this section.
Create one ".cpp" file for defining the "addonTestFunc()" function. This function must exist in global scope within your test addon. The contents of this ".cpp" file will look something like:
#include "TestAddon.h"
Test *addonTestFunc(void)
{
TestSuite *testSuite = new TestSuite("<TestSuiteName>");
testSuite->addTest(<ClassName1>::suite());
testSuite->addTest(<ClassName2>::suite());
/* etc */
return(testSuite);
}
In the above example, replace <TestSuiteName> with an appropriate name for the group of tests and <ClassName1> and <ClassName2> with the names of the test classes you have defined.
Create a build system around a BeIDE project, Makefile or preferrably a jam file which builds all the necessary code you have written into an addon.
Put this addon into the app_kit/test/add-ons directory and follow the above instructions for how to run your tests.
There are example tests which you can find in the following directories:
app_kit/test/lib/application/BMessageQueueapp_kit/test/lib/support/BAutolockapp_kit/test/lib/support/BLockerThere are some things done in these tests which make things a bit more complex, but you may want to do similar things:
Even with the complexity, I think this code provides a pretty good example of how to write your tests.
If you have a test which you want to define that requires more than one thread of execution (most likely a concurrency test of you code), you need to use the ThreadedTestCaller class. The steps which differ from the above description on how to write a test case are:
In your test class, define a member function for each thread you will be starting. All of these member functions must take "void" and return "void". If all the threads in your test perform the exact same actions, it is OK to just define one member function. Usually in the tests I have written, I have called these member functions "TestThread1()", "TestThread2()", etc.
If your "static Test *suite()" function for your test class, you must return a ThreadedTestCaller. Imagine that the test class name is "MyTestClass" and you want two threads which run member functions "TestThread1()" and "TestThread2()". That code would look like:
Test *MyTestClass::suite(void)
{
MyTestClass *theTest = new MyTestClass("");
ThreadedTestCaller<MyTestClass> *threadedTest = new TreadedTestCaller<MyTestClass>("", theTest);
threadedTest->addThread(":Thread1", &MyTestClass::TestThread1);
threadedTest->addThread(":Thread2", &MyTestClass::TestThread2);
return(threadedTest);
}
If you need to, you can put a number of ThreadedTestCaller instances into a TestSuite and return them in the suite() member function. Examples of this can be found in the BLocker and BMessageQueue test examples.
Otherwise the steps are the same as for other tests. The code gets much more complex if you define your test classes as templates as the examples do.