NetBSD/dist/atf/ROADMAP

165 lines
6.8 KiB
Plaintext

Automated Testing Framework (atf)
The NetBSD Foundation, Inc.
---------------------------------------------------------------------------
Introduction
============
This document contains a *non-exhaustive* list of things that should be
addressed in ATF before it can be considered stable. Some of the items
are planned for specific future releases, but take this with a big grain
of salt: the focus of a specific release may change completely depending
on how things develop.
Plans for 0.6
=============
* Add a binary atf-check tool that replaces atf-sh's atf_check function.
I suspect this will speed things up considerably, and will also expose
this extremely useful functionality to C/C++ test programs.
* Add support for ATF_REQUIRE, making ATF_CHECK non-fatal. See Boost.Test
for details on this idea. Most useful for unit tests.
* Add a module to atf-c to manage dynamic memory. Should provide a "mem
chunk" object that can only be managed through functions (i.e. not
directly) so that access to buffers can be safely controlled. Dealing
with strdup and similar functions, for example, might be complex.
See these old revisions for a start, but these did not work correctly
because the use of (void **) casts brought aliasing problems:
78eeacf3b9284493e5e96b7866fc1d163a13bc02
8e200683a2b70eadca728c2e0347d4790777b3fc
872393ed0177cbcc30ffacede228eb3986c42ab7
* Optimize timeout handling in atf-sh: there are too many processes
involved to run each test case, and this hurts a lot in OS X, for
example. One possibility is to integrate the timeout handling in
atf-exec and then kill the auxiliary code in atf.footer.subr.
Plans for 0.7
=============
* Add a tool to collect multiple outputs of atf-run (from different
machines) and generate a single (XML-only?) log with everything.
Must allow easy conversion to HTML for online publishing.
* Allow grouping of test programs in tiers in an Atffile. This is to
permit the user specify "dependencies" between test programs: e.g. do
not run a specific test program if its dependencies have failed, because
it will certainly fail also.
* Provide a kernel-level unit testing API (for NetBSD only, at the moment).
This should come in the form of an atf.ko module that provides functions
to define and register test cases, functions for results reporting and
an interface (a trivial file system?) that transports the
application/X-atf-tcs output to user-space, provides information to
user-space about the available test cases (a list) and allows user-space
to launch the execution of test cases.
* Add expected failures and unexpected passes as test case results? The
former are used to indicate known failures at specific points and thus
not raise false positives when working on some unrelated feature. The
latter are used to detect changes that fixed expected failures. See
Monotone's test suite for an example.
Plans for pre-1.0
=================
* Fix all occurrences of XXX, TODO and FIXME.
* Think of a way to properly add tests for (almost?) all error paths.
Most of them are probably broken already.
* Improve error reporting: aside from clarifying error messages, this also
implies adding more error cases to give them more semantic meaning at the
source code level..
* Add an atf_check-like function to the C++ binding to allow easy testing
of executables.
* Make the shell library work with 'set -e'?
* Shell test programs dynamically locate where the shell library is by
calling atf-config (done by atf.init.subr). Contrarywise, binary test
programs are directly linked against the final location of libatf.
It may be nice if the latter loaded the library dynamically based on
what atf-config said, so the user could switch atf installations by
simply changing its PATH (and effectively making atf relocatable on the
file system). Why could this be nice? To painlessly run an older atf
test suite against a more recent version of the code base to ensure
there are no regressions even with older tests. Just a crazy idea, as
maybe what the shell test programs currently do is stupid.
* Allow users to customize the build of atf by defining additional meta-data
for test cases. At the moment this is possible because the meta-data is
not sanity-checked, but I think it should be. Following the previous
item, NetBSD could add a 'netbsd.pr' variable and then use this data when
generating reports to add direct links to the appropriate PRs.
* Make sure that the tests in tests/atf have, at the very least, the same
coverage as the ones in tests/bootstrap.
* Document the code.
* Possibly add a way to automatically gain or drop privileges when
require.user is set.
* Add a way to specify which bug/issue/whatever a given test case is
stress-testing. This information is useful when detecting regressions.
Plans for 1.0-RC1
=================
* Build libatf as a shared library and set -version-info accordingly.
* Set the DTDs' versions to 1.0.
Plans for post-1.0
==================
* Allow the parallel execution of tests. Achieving this with a test
program granularity is easy: only need to change atf-run. Lowering it
to a finer granularity (test cases) is harder and maybe not worth it.
Things that we will not do
==========================
This is a list of things that will *not* be addressed anytime soon in the
project. Of course most of them would be nice to have in the future, but
they will not block releases nor drive development. We can obviously
change our mind in the future and add move some of these to the above list.
* Native Win32 support: we are not talking about building atf with tools
such as Cygwin or MinGW, which should already be possible. Native Win32
support means modifying the code to use Win32 library calls and easily
build out of the box (i.e. the GNU Autotools are not useful in that
case).
* Extreme efficiency: even though we will focus on using the most suitable
data structures in each situation, we will not attempt to get extreme
efficiency by adding hacks that make the code uglier. Testing is a task
that requires a lot of resources on its own, and some tests will be
specially intensive, so there is no point in micro-optimizing atf if the
global gains are negligible.
* Extreme portability: the initial goal of this project was to provide a
testing framework for the NetBSD Operating System. Given that this
system has a very nice build framework, we can be sure that atf will be
built and used with sane tools (basically a decent C++ compiler and a
POSIX-compliant shell interpreter). We will definitely not aim for
portability to broken systems by tweaking our code to not follow the
standards.
---------------------------------------------------------------------------
End of document