165 lines
6.8 KiB
Plaintext
165 lines
6.8 KiB
Plaintext
Automated Testing Framework (atf)
|
|
The NetBSD Foundation, Inc.
|
|
---------------------------------------------------------------------------
|
|
|
|
|
|
Introduction
|
|
============
|
|
|
|
This document contains a *non-exhaustive* list of things that should be
|
|
addressed in ATF before it can be considered stable. Some of the items
|
|
are planned for specific future releases, but take this with a big grain
|
|
of salt: the focus of a specific release may change completely depending
|
|
on how things develop.
|
|
|
|
|
|
Plans for 0.6
|
|
=============
|
|
|
|
* Add a binary atf-check tool that replaces atf-sh's atf_check function.
|
|
I suspect this will speed things up considerably, and will also expose
|
|
this extremely useful functionality to C/C++ test programs.
|
|
|
|
* Add support for ATF_REQUIRE, making ATF_CHECK non-fatal. See Boost.Test
|
|
for details on this idea. Most useful for unit tests.
|
|
|
|
* Add a module to atf-c to manage dynamic memory. Should provide a "mem
|
|
chunk" object that can only be managed through functions (i.e. not
|
|
directly) so that access to buffers can be safely controlled. Dealing
|
|
with strdup and similar functions, for example, might be complex.
|
|
|
|
See these old revisions for a start, but these did not work correctly
|
|
because the use of (void **) casts brought aliasing problems:
|
|
|
|
78eeacf3b9284493e5e96b7866fc1d163a13bc02
|
|
8e200683a2b70eadca728c2e0347d4790777b3fc
|
|
872393ed0177cbcc30ffacede228eb3986c42ab7
|
|
|
|
* Optimize timeout handling in atf-sh: there are too many processes
|
|
involved to run each test case, and this hurts a lot in OS X, for
|
|
example. One possibility is to integrate the timeout handling in
|
|
atf-exec and then kill the auxiliary code in atf.footer.subr.
|
|
|
|
|
|
Plans for 0.7
|
|
=============
|
|
|
|
* Add a tool to collect multiple outputs of atf-run (from different
|
|
machines) and generate a single (XML-only?) log with everything.
|
|
Must allow easy conversion to HTML for online publishing.
|
|
|
|
* Allow grouping of test programs in tiers in an Atffile. This is to
|
|
permit the user specify "dependencies" between test programs: e.g. do
|
|
not run a specific test program if its dependencies have failed, because
|
|
it will certainly fail also.
|
|
|
|
* Provide a kernel-level unit testing API (for NetBSD only, at the moment).
|
|
This should come in the form of an atf.ko module that provides functions
|
|
to define and register test cases, functions for results reporting and
|
|
an interface (a trivial file system?) that transports the
|
|
application/X-atf-tcs output to user-space, provides information to
|
|
user-space about the available test cases (a list) and allows user-space
|
|
to launch the execution of test cases.
|
|
|
|
* Add expected failures and unexpected passes as test case results? The
|
|
former are used to indicate known failures at specific points and thus
|
|
not raise false positives when working on some unrelated feature. The
|
|
latter are used to detect changes that fixed expected failures. See
|
|
Monotone's test suite for an example.
|
|
|
|
|
|
Plans for pre-1.0
|
|
=================
|
|
|
|
* Fix all occurrences of XXX, TODO and FIXME.
|
|
|
|
* Think of a way to properly add tests for (almost?) all error paths.
|
|
Most of them are probably broken already.
|
|
|
|
* Improve error reporting: aside from clarifying error messages, this also
|
|
implies adding more error cases to give them more semantic meaning at the
|
|
source code level..
|
|
|
|
* Add an atf_check-like function to the C++ binding to allow easy testing
|
|
of executables.
|
|
|
|
* Make the shell library work with 'set -e'?
|
|
|
|
* Shell test programs dynamically locate where the shell library is by
|
|
calling atf-config (done by atf.init.subr). Contrarywise, binary test
|
|
programs are directly linked against the final location of libatf.
|
|
It may be nice if the latter loaded the library dynamically based on
|
|
what atf-config said, so the user could switch atf installations by
|
|
simply changing its PATH (and effectively making atf relocatable on the
|
|
file system). Why could this be nice? To painlessly run an older atf
|
|
test suite against a more recent version of the code base to ensure
|
|
there are no regressions even with older tests. Just a crazy idea, as
|
|
maybe what the shell test programs currently do is stupid.
|
|
|
|
* Allow users to customize the build of atf by defining additional meta-data
|
|
for test cases. At the moment this is possible because the meta-data is
|
|
not sanity-checked, but I think it should be. Following the previous
|
|
item, NetBSD could add a 'netbsd.pr' variable and then use this data when
|
|
generating reports to add direct links to the appropriate PRs.
|
|
|
|
* Make sure that the tests in tests/atf have, at the very least, the same
|
|
coverage as the ones in tests/bootstrap.
|
|
|
|
* Document the code.
|
|
|
|
* Possibly add a way to automatically gain or drop privileges when
|
|
require.user is set.
|
|
|
|
* Add a way to specify which bug/issue/whatever a given test case is
|
|
stress-testing. This information is useful when detecting regressions.
|
|
|
|
|
|
Plans for 1.0-RC1
|
|
=================
|
|
|
|
* Build libatf as a shared library and set -version-info accordingly.
|
|
|
|
* Set the DTDs' versions to 1.0.
|
|
|
|
|
|
Plans for post-1.0
|
|
==================
|
|
|
|
* Allow the parallel execution of tests. Achieving this with a test
|
|
program granularity is easy: only need to change atf-run. Lowering it
|
|
to a finer granularity (test cases) is harder and maybe not worth it.
|
|
|
|
|
|
Things that we will not do
|
|
==========================
|
|
|
|
This is a list of things that will *not* be addressed anytime soon in the
|
|
project. Of course most of them would be nice to have in the future, but
|
|
they will not block releases nor drive development. We can obviously
|
|
change our mind in the future and add move some of these to the above list.
|
|
|
|
* Native Win32 support: we are not talking about building atf with tools
|
|
such as Cygwin or MinGW, which should already be possible. Native Win32
|
|
support means modifying the code to use Win32 library calls and easily
|
|
build out of the box (i.e. the GNU Autotools are not useful in that
|
|
case).
|
|
|
|
* Extreme efficiency: even though we will focus on using the most suitable
|
|
data structures in each situation, we will not attempt to get extreme
|
|
efficiency by adding hacks that make the code uglier. Testing is a task
|
|
that requires a lot of resources on its own, and some tests will be
|
|
specially intensive, so there is no point in micro-optimizing atf if the
|
|
global gains are negligible.
|
|
|
|
* Extreme portability: the initial goal of this project was to provide a
|
|
testing framework for the NetBSD Operating System. Given that this
|
|
system has a very nice build framework, we can be sure that atf will be
|
|
built and used with sane tools (basically a decent C++ compiler and a
|
|
POSIX-compliant shell interpreter). We will definitely not aim for
|
|
portability to broken systems by tweaking our code to not follow the
|
|
standards.
|
|
|
|
|
|
---------------------------------------------------------------------------
|
|
End of document
|