ffe8204c6b
Changes in this release: * Clauses 3 and 4 of the BSD license used by the project were dropped. All the code is now under a 2-clause BSD license compatible with the GNU General Public License (GPL). * Added a C-only binding so that binary test programs do not need to be tied to C++ at all. This binding is now known as the atf-c library. * Renamed the C++ binding to atf-c++ for consistency with the new atf-c. * Renamed the POSIX shell binding to atf-sh for consistency with the new atf-c and atf-c++. * Added a -w flag to test programs through which it is possible to specify the work directory to be used. This was possible in prior releases by defining the workdir configuration variable (-v workdir=...), but was a conceptually incorrect mechanism. * Test programs now preserve the execution order of test cases when they are given in the command line. Even those mentioned more than once are executed multiple times to comply with the user's requests.
165 lines
6.8 KiB
Plaintext
165 lines
6.8 KiB
Plaintext
Automated Testing Framework (atf)
|
|
The NetBSD Foundation, Inc.
|
|
---------------------------------------------------------------------------
|
|
|
|
|
|
Introduction
|
|
============
|
|
|
|
This document contains a *non-exhaustive* list of things that should be
|
|
addressed in ATF before it can be considered stable. Some of the items
|
|
are planned for specific future releases, but take this with a big grain
|
|
of salt: the focus of a specific release may change completely depending
|
|
on how things develop.
|
|
|
|
|
|
Plans for 0.6
|
|
=============
|
|
|
|
* Add a binary atf-check tool that replaces atf-sh's atf_check function.
|
|
I suspect this will speed things up considerably, and will also expose
|
|
this extremely useful functionality to C/C++ test programs.
|
|
|
|
* Add support for ATF_REQUIRE, making ATF_CHECK non-fatal. See Boost.Test
|
|
for details on this idea. Most useful for unit tests.
|
|
|
|
* Add a module to atf-c to manage dynamic memory. Should provide a "mem
|
|
chunk" object that can only be managed through functions (i.e. not
|
|
directly) so that access to buffers can be safely controlled. Dealing
|
|
with strdup and similar functions, for example, might be complex.
|
|
|
|
See these old revisions for a start, but these did not work correctly
|
|
because the use of (void **) casts brought aliasing problems:
|
|
|
|
78eeacf3b9284493e5e96b7866fc1d163a13bc02
|
|
8e200683a2b70eadca728c2e0347d4790777b3fc
|
|
872393ed0177cbcc30ffacede228eb3986c42ab7
|
|
|
|
* Optimize timeout handling in atf-sh: there are too many processes
|
|
involved to run each test case, and this hurts a lot in OS X, for
|
|
example. One possibility is to integrate the timeout handling in
|
|
atf-exec and then kill the auxiliary code in atf.footer.subr.
|
|
|
|
|
|
Plans for 0.7
|
|
=============
|
|
|
|
* Add a tool to collect multiple outputs of atf-run (from different
|
|
machines) and generate a single (XML-only?) log with everything.
|
|
Must allow easy conversion to HTML for online publishing.
|
|
|
|
* Allow grouping of test programs in tiers in an Atffile. This is to
|
|
permit the user specify "dependencies" between test programs: e.g. do
|
|
not run a specific test program if its dependencies have failed, because
|
|
it will certainly fail also.
|
|
|
|
* Provide a kernel-level unit testing API (for NetBSD only, at the moment).
|
|
This should come in the form of an atf.ko module that provides functions
|
|
to define and register test cases, functions for results reporting and
|
|
an interface (a trivial file system?) that transports the
|
|
application/X-atf-tcs output to user-space, provides information to
|
|
user-space about the available test cases (a list) and allows user-space
|
|
to launch the execution of test cases.
|
|
|
|
* Add expected failures and unexpected passes as test case results? The
|
|
former are used to indicate known failures at specific points and thus
|
|
not raise false positives when working on some unrelated feature. The
|
|
latter are used to detect changes that fixed expected failures. See
|
|
Monotone's test suite for an example.
|
|
|
|
|
|
Plans for pre-1.0
|
|
=================
|
|
|
|
* Fix all occurrences of XXX, TODO and FIXME.
|
|
|
|
* Think of a way to properly add tests for (almost?) all error paths.
|
|
Most of them are probably broken already.
|
|
|
|
* Improve error reporting: aside from clarifying error messages, this also
|
|
implies adding more error cases to give them more semantic meaning at the
|
|
source code level..
|
|
|
|
* Add an atf_check-like function to the C++ binding to allow easy testing
|
|
of executables.
|
|
|
|
* Make the shell library work with 'set -e'?
|
|
|
|
* Shell test programs dynamically locate where the shell library is by
|
|
calling atf-config (done by atf.init.subr). Contrarywise, binary test
|
|
programs are directly linked against the final location of libatf.
|
|
It may be nice if the latter loaded the library dynamically based on
|
|
what atf-config said, so the user could switch atf installations by
|
|
simply changing its PATH (and effectively making atf relocatable on the
|
|
file system). Why could this be nice? To painlessly run an older atf
|
|
test suite against a more recent version of the code base to ensure
|
|
there are no regressions even with older tests. Just a crazy idea, as
|
|
maybe what the shell test programs currently do is stupid.
|
|
|
|
* Allow users to customize the build of atf by defining additional meta-data
|
|
for test cases. At the moment this is possible because the meta-data is
|
|
not sanity-checked, but I think it should be. Following the previous
|
|
item, NetBSD could add a 'netbsd.pr' variable and then use this data when
|
|
generating reports to add direct links to the appropriate PRs.
|
|
|
|
* Make sure that the tests in tests/atf have, at the very least, the same
|
|
coverage as the ones in tests/bootstrap.
|
|
|
|
* Document the code.
|
|
|
|
* Possibly add a way to automatically gain or drop privileges when
|
|
require.user is set.
|
|
|
|
* Add a way to specify which bug/issue/whatever a given test case is
|
|
stress-testing. This information is useful when detecting regressions.
|
|
|
|
|
|
Plans for 1.0-RC1
|
|
=================
|
|
|
|
* Build libatf as a shared library and set -version-info accordingly.
|
|
|
|
* Set the DTDs' versions to 1.0.
|
|
|
|
|
|
Plans for post-1.0
|
|
==================
|
|
|
|
* Allow the parallel execution of tests. Achieving this with a test
|
|
program granularity is easy: only need to change atf-run. Lowering it
|
|
to a finer granularity (test cases) is harder and maybe not worth it.
|
|
|
|
|
|
Things that we will not do
|
|
==========================
|
|
|
|
This is a list of things that will *not* be addressed anytime soon in the
|
|
project. Of course most of them would be nice to have in the future, but
|
|
they will not block releases nor drive development. We can obviously
|
|
change our mind in the future and add move some of these to the above list.
|
|
|
|
* Native Win32 support: we are not talking about building atf with tools
|
|
such as Cygwin or MinGW, which should already be possible. Native Win32
|
|
support means modifying the code to use Win32 library calls and easily
|
|
build out of the box (i.e. the GNU Autotools are not useful in that
|
|
case).
|
|
|
|
* Extreme efficiency: even though we will focus on using the most suitable
|
|
data structures in each situation, we will not attempt to get extreme
|
|
efficiency by adding hacks that make the code uglier. Testing is a task
|
|
that requires a lot of resources on its own, and some tests will be
|
|
specially intensive, so there is no point in micro-optimizing atf if the
|
|
global gains are negligible.
|
|
|
|
* Extreme portability: the initial goal of this project was to provide a
|
|
testing framework for the NetBSD Operating System. Given that this
|
|
system has a very nice build framework, we can be sure that atf will be
|
|
built and used with sane tools (basically a decent C++ compiler and a
|
|
POSIX-compliant shell interpreter). We will definitely not aim for
|
|
portability to broken systems by tweaking our code to not follow the
|
|
standards.
|
|
|
|
|
|
---------------------------------------------------------------------------
|
|
End of document
|