190 lines
6.6 KiB
Plaintext
190 lines
6.6 KiB
Plaintext
|
Automated Testing Framework (atf)
|
||
|
The NetBSD Foundation, Inc.
|
||
|
---------------------------------------------------------------------------
|
||
|
|
||
|
|
||
|
Version 0.6
|
||
|
===========
|
||
|
|
||
|
Release date: January 18th, 2009
|
||
|
Status: Experimental
|
||
|
|
||
|
* Make atf-exec be able to kill its child process after a certain period of
|
||
|
time; this is controlled through the new -t option.
|
||
|
|
||
|
* Change atf-sh to use atf-exec's -t option to control the test case's
|
||
|
timeouts, instead of doing it internally. Same behavior as before, but
|
||
|
noticeably faster.
|
||
|
|
||
|
* atf-exec's -g option and atf-killpg are gone due to the previous change.
|
||
|
|
||
|
* Added the atf-check(1) tool, a program that executes a given command and
|
||
|
checks its exit code against a known value and allows the management of
|
||
|
stdout and stderr in multiple ways. This replaces the previous atf_check
|
||
|
function in the atf-sh library and exposes this functionality to both
|
||
|
atf-c and atf-c++.
|
||
|
|
||
|
* Added the ATF_REQUIRE family of macros to the C interface. These help
|
||
|
in checking for fatal test conditions. The old ATF_CHECK macros now
|
||
|
perform non-fatal checks only. I.e. by using ATF_CHECK, the test case
|
||
|
can now continue its execution and the failures will not be reported
|
||
|
until the end of the whole run.
|
||
|
|
||
|
* Extended the amount of ATF_CHECK_* C macros with new ones to provide more
|
||
|
features to the developer. These also have their corresponding
|
||
|
counterparts in the ATF_REQUIRE_* family. The new macros (listing the
|
||
|
suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and
|
||
|
_STREQ_MSG.
|
||
|
|
||
|
|
||
|
Version 0.5
|
||
|
===========
|
||
|
|
||
|
Release date: May 1st, 2008
|
||
|
Status: Experimental
|
||
|
|
||
|
* Clauses 3 and 4 of the BSD license used by the project were dropped.
|
||
|
All the code is now under a 2-clause BSD license compatible with the
|
||
|
GNU General Public License (GPL).
|
||
|
|
||
|
* Added a C-only binding so that binary test programs do not need to be
|
||
|
tied to C++ at all. This binding is now known as the atf-c library.
|
||
|
|
||
|
* Renamed the C++ binding to atf-c++ for consistency with the new atf-c.
|
||
|
|
||
|
* Renamed the POSIX shell binding to atf-sh for consistency with the new
|
||
|
atf-c and atf-c++.
|
||
|
|
||
|
* Added a -w flag to test programs through which it is possible to specify
|
||
|
the work directory to be used. This was possible in prior releases by
|
||
|
defining the workdir configuration variable (-v workdir=...), but was a
|
||
|
conceptually incorrect mechanism.
|
||
|
|
||
|
* Test programs now preserve the execution order of test cases when they
|
||
|
are given in the command line. Even those mentioned more than once are
|
||
|
executed multiple times to comply with the user's requests.
|
||
|
|
||
|
|
||
|
Version 0.4
|
||
|
===========
|
||
|
|
||
|
Release date: February 4th, 2008
|
||
|
Status: Experimental
|
||
|
|
||
|
Changes:
|
||
|
|
||
|
* Added two new manual pages, atf-c++-api and atf-sh-api, describing the
|
||
|
C++ and POSIX shell interfaces used to write test programs.
|
||
|
|
||
|
* Added a pkg-config file, useful to get the flags to build against the
|
||
|
C++ library or to easily detect the presence of ATF.
|
||
|
|
||
|
* Added a way for test cases to require a specific architecture and/or
|
||
|
machine type through the new 'require.arch' and 'require.machine'
|
||
|
meta-data properties, respectively.
|
||
|
|
||
|
* Added the 'timeout' property to test cases, useful to set an upper-bound
|
||
|
limit for the test's run time and thus prevent global test program stalls
|
||
|
due to the test case's misbehavior.
|
||
|
|
||
|
* Added the atf-exec(1) internal utility, used to execute a command after
|
||
|
changing the process group it belongs to.
|
||
|
|
||
|
* Added the atf-killpg(1) internal utility, used to kill process groups.
|
||
|
|
||
|
* Multiple portability fixes. Of special interest, full support for SunOS
|
||
|
(Solaris Express Developer Edition 2007/09) using the Sun Studio 12 C++
|
||
|
compiler.
|
||
|
|
||
|
* Fixed a serious bug that prevented atf-run(1) from working at all under
|
||
|
Fedora 8 x86_64. Due to the nature of the bug, other platforms were
|
||
|
likely affected too.
|
||
|
|
||
|
|
||
|
Version 0.3
|
||
|
===========
|
||
|
|
||
|
Release date: November 11th, 2007
|
||
|
Status: Experimental
|
||
|
|
||
|
Changes:
|
||
|
|
||
|
* Added XML output support to atf-report. This is accompanied by a DTD for
|
||
|
the format's structure and sample XSLT/CSS files to post-process this
|
||
|
output and convert it to a plain HTML report.
|
||
|
|
||
|
* Changed atf-run to add system information to the report it generates.
|
||
|
This is currently used by atf-report's XML output only, and is later
|
||
|
printed in the HTML reports in a nice and useful summary table. The user
|
||
|
and system administrator are allowed to tune this feature by means of
|
||
|
hooks.
|
||
|
|
||
|
* Removed the test cases' 'isolated' property. This was intended to avoid
|
||
|
touching the file system at all when running the related test case, but
|
||
|
this has not been true for a long while: some control files are
|
||
|
unconditionally required for several purposes, and we cannot easily get
|
||
|
rid of them. This way we remove several critical and delicate pieces of
|
||
|
code.
|
||
|
|
||
|
* Improved atf-report's CSV output format to include information about
|
||
|
test programs too.
|
||
|
|
||
|
* Fixed the tests that used atf-compile to not require this tool as a
|
||
|
helper. Avoids systems without build-time utilities to skip many tests
|
||
|
that could otherwise be run. (E.g. NetBSD without the comp.tgz set
|
||
|
installed.)
|
||
|
|
||
|
* Many general cleanups: Fixed many pieces of code marked as ugly and/or
|
||
|
incomplete.
|
||
|
|
||
|
|
||
|
Version 0.2
|
||
|
===========
|
||
|
|
||
|
Release date: September 20th, 2007
|
||
|
Status: Experimental
|
||
|
|
||
|
Changes:
|
||
|
|
||
|
* Test cases now get a known umask on entry.
|
||
|
|
||
|
* atf-run now detects many unexpected failures caused by test programs
|
||
|
and reports them as bogus tests. atf-report is able to handle these
|
||
|
new errors and nicely reports them to the user.
|
||
|
|
||
|
* All the data formats read and written by the tools have been documented
|
||
|
and cleaned up. These include those grammars that define how the
|
||
|
different components communicate with each other as well as the format
|
||
|
of files written by the developers and users: the Atffiles and the
|
||
|
configuration files.
|
||
|
|
||
|
* Added the atf-version tool, a utility that displays information about the
|
||
|
currently installed version of ATF.
|
||
|
|
||
|
* Test cases can now define an optional cleanup routine to undo their
|
||
|
actions regardless of their exit status.
|
||
|
|
||
|
* atf-report now summarizes the list of failed (bogus) test programs when
|
||
|
using the ticker output format.
|
||
|
|
||
|
* Test programs now capture some termination signals and clean up any
|
||
|
temporary files before exiting the program.
|
||
|
|
||
|
* Multiple bug fixes and improvements all around.
|
||
|
|
||
|
|
||
|
Version 0.1
|
||
|
===========
|
||
|
|
||
|
Release date: August 20th, 2007
|
||
|
Status: Experimental
|
||
|
|
||
|
Changes:
|
||
|
|
||
|
* First public version. This was released coinciding with the end of the
|
||
|
Google Summer of Code 2007 program.
|
||
|
|
||
|
|
||
|
---------------------------------------------------------------------------
|
||
|
End of document
|