The main change here is that the atf-config, atf-report, atf-run and
atf-version tools no longer depend on libatf-c nor libatf-c++. Instead,
they depend on an internal libtools.a that contains code specifically
for these tools and nothing else, making them self-contained.
Changes in version 0.19
***********************
Experimental version released on February 7th, 2014.
This is the last release to bundle the code for the deprecated tools.
The next release will drop their code and will stop worrying about
backwards compatibility between the ATF libraries and what the old tools
may or may not support.
If you still require the old tools for some reason, grab a copy of the
'tools' directory now. The code in this directory is standalone and
does not depend on any internal details of atf-c++ any longer.
* Various fixes and improvements to support running as part of the FreeBSD
test suite.
* Project hosting moved from Google Code (as a subproject of Kyua) to
GitHub (as a first-class project). The main reason for the change is
the suppression of binary downloads in Google Code on Jan 15th, 2014.
See https://github.com/jmmv/atf/
* Removed builtin help from atf-sh(1) and atf-check(1) for simplicity
reasons. In other words, their -h option is gone.
* Moved the code of the deprecated tools into a 'tools' directory and
completely decoupled their code from the internals of atf-c++. The
reason for this is to painlessly allow a third-party to maintain a
copy of these tools after we delete them because upcoming changes to
atf-c++ would break the stale tools.
Changes in version 0.18
***********************
Experimental version released on November 16th, 2013.
* Issue 45: Added require.memory support in atf-run for FreeBSD.
* Fixed an issue with the handling of cin with libc++.
* Issue 64: Fixed various mandoc formatting warnings.
* NetBSD PR bin/48284: Made atf-check flush its progress message to
stdout so that an interrupted test case always shows the last message
being executed.
* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3).
- Delete some unnecessary files.
- Print out both the list of deleted files and added files in the import.
Useful when adjusting file lists and the reachover Makefiles.
shell so it can't cleanup stuff set in the environment of the first shell.
There are 2 problems:
- calling the test routine directly does not cleanup (not using
atf-run)
- when using atf-run, the cleanup routine needs state from the
running code which can only be stored in files. This will never
allow us to run tests in parallel since we need to keep a known
place to pass state.
character of a get() for ungetch() to work. Prefer putback() to make it
work with current implementations of cin in libc++. Tracked as
http://llvm.org/bugs/show_bug.cgi?id=15867 with test case.
bconfig.h. Also make the build of these files depends on bconfig.h itself.
This should fix once and for all the problems of the
atf/atf-{c,c++}/pkg_config_test:version tests breaking because the
pkgconfig files hold an old version number during update builds.
Keeping these files up to date with every new import is too easy to get
wrong. Would be nice if we'd extract the version number in some other
manner, like from lib/libatf-c/bconfig.h.
Found by martin@.
Experimental version released on February 14th, 2013.
* Added the atf_utils_cat_file, atf_utils_compare_file,
atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists,
atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string,
atf_utils_readline, atf_utils_redirect and atf_utils_wait utility
functions to atf-c-api. Documented the already-public
atf_utils_free_charpp function.
* Added the cat_file, compare_file, copy_file, create_file, file_exists,
fork, grep_collection, grep_file, grep_string, redirect and wait
functions to the atf::utils namespace of atf-c++-api. These are
wrappers around the same functions added to the atf-c-api library.
* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and
ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a
string against a regular expression.
* Miscellaneous fixes for manpage typos and compilation problems with
clang.
* Added caching of the results of those configure tests that rely on
executing a test program. This should help crossbuild systems by
providing a mechanism to pre-specify what the results should be.
* PR bin/45690: Make atf-report convert any non-printable characters to
a plain-text representation (matching their corresponding hexadecimal
entities) in XML output files. This is to prevent the output of test
cases from breaking xsltproc later.
Experimental version released on July 10th, 2012.
* Added a --enable-tools flag to configure to request the build of the
deprecated ATF tools, whose build is now disabled by default. In order
to continue running tests, you should migrate to Kyua instead of enabling
the build of the deprecated tools. The kyua-atf-compat package provides
transitional compatibility versions of atf-run and atf-report built on
top of Kyua.
* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can
detect defined but unused test cases.
* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and
tp-time XML tags leaking into the generated HTML file. Also improved
the CSS file slightly to correct alignment and color issues with the
timestamps column.
* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during
compilation with GNU G++.
* Flipped the default to building shared libraries for atf-c and atf-c++,
and started versioning them. As a side-effect, this removes the
--enable-unstable-shared flag from configure that appears to not work any
more (under NetBSD). Additionally, some distributions require the use of
shared libraries for proper dependency tracking (e.g. Fedora), so it is
better if we do the right versioning upstream.
* Project hosting moved from an adhoc solution (custom web site and
Monotone repository) to Google Code (standard wiki and Git). ATF now
lives in a subcomponent of the Kyua project.
Fix some XSLT/CSS bugs that leaked some internal tags into the HTML file.
This, together with some CSS inconsistency, supposedly prevented the
timestamp column from being right-aligned... although I have not been able
to reproduce it with neither Chrome nor Firefox. That said, the changes
in here should fix this.
While doing this, also change the background color of the timestamp column
for test programs to improve the divider bar between test programs.
Based on patches and ideas from pgoyette@. Should fix PR bin/45859.
changes to the atf codebase with upstream code. All local changes should
have been backported, with appropriate portability workarounds where
necessary.
This new release also includes other changes though, such as performance
improvements and bug fixes, and also a major new feature partially
implemented by pgoyette@: the time to execute each test cases and test
program is now recorded and included in the output reports.
The import into NetBSD has been tested natively on amd64 and macppc, and
the full test suite has also been run through anita on amd64 and i386.
No regressions observed... but you never know.
From the NEWS file, the changes in this version are as follows:
Experimental version released on January 16th, 2012.
* Respect stdin in atf-check. The previous release silenced stdin for any
processes spawned by atf, not only test programs, which caused breakage
in tests that pipe data through atf-check.
* Performance improvements to atf-sh.
* Enabled detection of unused parameters and variables in the code and
fixed all warnings.
* Changed the behavior of "developer mode". Compiler warnings are now
enabled unconditionally regardless of whether we are in developer mode or
not; developer mode is now only used to perform strict warning checks and
to enable assertions. Additionally, developer mode is now only
automatically enabled when building from the repository, not for formal
releases.
* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and
ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag
in configure scripts and detecting the presence of any of the ATF
bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it
has now been modified to also honor --with-atf if instantiated.
* Added timing support to atf-run / atf-report.
* Added support for a 'require.memory' property, to specify the minimum
amount of physical memory needed by the test case to yield valid results.
* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by
atf-report so that invalid data in the output of test cases does not
mangle our report.
we migrate to Kyua (atf v2), so it's better to use a generic name that does
not depend on the specific implementation. Also, this user has not gone
out yet into any stable release, so we can easily rename it.
Suggested by jruoho@.
-----
Only silence stdin for test programs
atf-check is expected to accept data in its stdin, and a previous change
broke this behavior.
-----
This should fix a few tests that broke during the 0.14 import. In
particularly, the tests in libc/stdlib and libc/ssp that redirect stuff
to atf_check.
Experimental version released on June 14th, 2011.
* Added a pkg-config file for atf-sh and an aclocal file to ease the
detection of atf-sh from autoconf scripts.
* Made the default test case body defined by atf_sh fail. This is to
ensure that test cases are properly defined in test programs and helps
in catching typos in the names of the body functions.
* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero.
This provides more consistent results with "normal" execution (in
particular, when tests are executed detached from a terminal).
* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but
that does not take into account that libc determines the current timezone
from a configuration file.
* All test programs will now print a warning when they are not run through
atf-run(1) stating that this is unsupported and may deliver incorrect
results.
* Added support for the 'require.files' test-case property. This allows
test cases to specify installed files that must be present for the test
case to run.
Unset TERM when running GDB
GDB inserts some funny control characters in its output when TERM is set to
e.g. xterm. Workaround this by simply unsetting TERM.
Reported by martin@ and diagnosed by pooka@/martin@.
Experimental version released on March 31st, 2011.
This is the first release after the creation of the Kyua project, a more
modular and reliable replacement for ATF. From now on, ATF will change to
accomodate the transition to this new codebase, but ATF will still continue
to see development in the short/medium term. Check out the project page at
http://code.google.com/p/kyua/ for more details.
The changes in this release are:
* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
new package that aims to replace atf-run and atf-report. The ATF tests
can be run with the new system by issuing a 'make installcheck-kyua' from
the top-level directory of the project (assuming the 'kyua' binary is
available during the configuration stage of ATF).
* atf-run and atf-report are now in maintenance mode (but *not* deprecated
yet!). Kyua already implements a new, much more reliable runtime engine
that provides similar features to these tools. That said, it is not
complete yet so all development efforts should go towards it.
* If GDB is installed, atf-run dumps the stack trace of crashing test
programs in an attempt to aid debugging. Contributed by Antti Kantee.
* Reverted default timeout change in previous release and reset its value
to 5 minutes. This was causing several issues, specially when running
the existing NetBSD test suite in qemu.
* Fixed the 'match' output checker in atf-check to properly validate the
last line of a file even if it does not have a newline.
* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
check for the presence (or lack thereof) of an element in a collection.
* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
when the cleanup of a test case triggered asynchronous modifications to
its work directory (e.g. killing a daemon process that cleans up a pid
file in the work directory).
* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
instead of just listing them as having 0 test cases.
Retry calls that raise file system errors during cleanup
If a test case mounts user-space (puffs/fuse) file systems or spawns
server processes that create pid files, the termination of the
corresponding processes does not guarantee that the file system is
left in a consistent state immediately. The cleanup routines of both
components (file systems and daemons) may still be running.
This situation causes a race condition between the termination of the
auxiliary processes and our own file system cleanup: the file system
calls performed from within the cleanup routine may raise errors
because the file system is still changing underneath. (E.g. we first
enumerate the contents of a directory and get file X, but when we
attempt to delete file X, it may be gone.)
Deal with this by retrying failing file system calls a few times and
ignoring "expected" errors before giving up.
Really kill subprocesses of a test case before waiting for its completion
Before waiting for any leftover output of the test case after it terminates,
we must ensure that all of its subprocess are really, really dead. Otherwise,
these subprocesses may be sharing the stdout of the test case, in which case
our wait will block (potentially indefinitely).
This finally (well, hopefully) fixes some random lockups exposed by the
NetBSD test suite. Reported by Antti Kantee after
tests/fs/vfs/t_full:p2k_ffs_fillfs was exposing this problem in a pretty
reproducible manner.
the cpu name and the latter the port name. They had been reversed until
now due to some "smart" stupidity^Wlogic in the upstream configure script,
which is now gone.
This is a pullup of revision f9329ca68da7e8557e0803b5747a12f8c10b1258
plus the corresponding reachover build changes.
Addresses PR bin/44305.
Revision: 869e092e4986eb5dce90331ca9a64e125d7796eb
Parent: cca40eb08e7469dfe9d6ca982613458f24c1de28
Author: jmmv@NetBSD.org
Date: 12/27/10 21:19:19
Branch: org.NetBSD.atf.src
Changelog:
Recognize sigabrt in the signal checker
Problem found by Paul Goyette.
Changes against parent cca40eb08e7469dfe9d6ca982613458f24c1de28
patched atf-sh/atf-check.cpp
This reverts the default timeout for test cases back to 300 seconds.
The change in the release was quite blind because it did not anticipate
many existing tests to be slow enough to overflow the modified timeout
(30 seconds), specially in anita.
My plan to really fix this is to let test cases specify their sizes in
a declarative way instead of specifying timeouts in seconds (the timeout
being defined by atf-run on a size basis), so I'm not going to bother to
go over all existing tests trying to figure out which ones need a higher
timeout for now. It is just easier to revert.
Experimental version released on November 7th, 2010.
* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as
ATF_REQUIRE_THROW but allows checking for the validity of the exception's
error message by means of a regular expression.
* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a
regular expression match in a string.
* Changed the default timeout for test cases from 5 minutes to 30 seconds.
30 seconds is long enough for virtually all tests to complete, and 5
minutes is a way too long pause in a test suite where a single test case
stalls.
* Deprecated the use.fs property. While this seemed like a good idea in
the first place to impose more control on what test cases can do, it
turns out to be bad. First, use.fs=false prevents bogus test cases
from dumping core so after-the-fact debugging is harder. Second,
supporting use.fs adds a lot of unnecessary complexity. atf-run will
now ignore any value provided to use.fs and will allow test cases to
freely access the file system if they wish to.
* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c
library. The 'text' module became private in 0.11 but was being used
externally to simplify the parsing of configuration variables.
* Made atf-run recognize the 'unprivileged-user' configuration variable
and automatically drop root privileges when a test case sets
require.user=unprivileged. Note that this is, by no means, done for
security purposes; this is just for user convenience; tests should, in
general, not be blindly run as root in the first place.
atf-run is not twice as large as before. This is a pull-up of
699284e5c0d0a375958293e578af4e02d68d1182.
(I don't think it's reasonable to intentionally cripple tests as I have
just done here. In the future I would like to only report the output of
failed test cases, which would allow us to undo this, but not there yet.)
Experimental version released on October 20th, 2010.
* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match
their counterparts in atf-c.
* Clearly separated the modules in atf-c that are supposed to be public
from those that are implementation details. The header files for the
internal modules are not installed any more.
* Made the atf-check tool private. It is only required by atf-sh and being
public has the danger of causing confusion. Also, making it private
simplifies the public API of atf.
* Changed atf-sh to enable per-command error checking (set -e) by default.
This catches many cases in which a test case is broken but it is not
reported as such because execution continues.
* Fixed the XSTL and CSS stylesheets to support expected failures.
996f9c26e07a86607f373c8f0243d57329c11543
ef98529abaf16e40a2e684496bf3da8f9ff0d09c
These prevent atf-run from stalling/crashing when a subprocess of a test
case stays around after the test case itself exits.
Reported, and verified working, by pooka@.
Catch and report unhandled exceptions before they propagate to C land
The C++ interface to run test cases goes like this:
1) C++ run function -> 2) C run function -> 3) C++ wrapper for
test case -> 4) test case head/body/cleanup
The previous code caught and reported unhandled exceptions in 1).
However, such approach does not seem to work everywhere. It fails,
for example, in NetBSD/i386 but works in NetBSD/amd64. I am not sure
which platform implementation is correct nor if there even _is_ a
defined behavior. No matter what, it feels wrong and clunky.
Move the last-resort exception catching to happen in 3) so that
exceptions don't propagate back to C.
Fixes the test-programs/result_test:result_exception test case in
NetBSD/i386 5.99.34.
Miscellaneous features
* Added expected failures support to test cases and atf-run. These
include, for example, expected clean exits, expected reception of fatal
signals, expected timeouts and expected errors in condition checks.
These statuses can be used to denote test cases that are known to fail
due to a bug in the code they are testing. atf-report reports these
tests separately but they do not count towards the failed test cases
amount.
* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to
allow easy checking of call failures that update errno.
* Added the has.cleanup meta-data property to test caes that specifies
whether the test case has a cleanup routine or not; its value is
automatically set. This property is read by atf-run to know if it has to
run the cleanup routine; skipping this run for every test case
significantly speeds up the run time of test suites.
* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to
take the expected exception as the first argument and the statement to
execute as the second argument.
Changes in atf-check
* Changed atf-check to support negating the status and output checks by
prefixing them with not- and added support to specify multiple checkers
for stdout and stderr, not only one.
* Added the match output checker to atf-check to look for regular
expressions in the stdout and stderr of commands.
* Modified the exit checks in atf-check to support checking for the
reception of signals.
Code simplifications and cleanups
* Removed usage messages from test programs to simplify the
implementation of every binding by a significant amount. They just now
refer the user to the appropriate manual page and do not attempt to wrap
lines on terminal boundaries. Test programs are not supposed to be run
by users directly so this minor interface regression is not important.
* Removed the atf-format internal utility, which is unused after the
change documented above.
* Removed the atf-cleanup internal utility. It has been unused since the
test case isolation was moved to atf-run in 0.8
* Splitted the Makefile.am into smaller files for easier maintenance and
dropped the use of M4. Only affects users building from the repository
sources.
* Intermixed tests with the source files in the source tree to provide
them more visibility and easier access. The tests directory is gone from
the source tree and tests are now suffixed by _test, not prefixed by t_.
* Simplifications to the atf-c library: removed the io, tcr and ui
modules as they had become unnecessary after all simplifications
introduced since the 0.8 release.
* Removed the application/X-atf-tcr format introduced in 0.8 release.
Tests now print a much simplified format that is easy to parse and nicer
to read by end users. As a side effect, the default for test cases is
now to print their results to stdout unless otherwise stated by providing
the -r flag.
* Removed XML distribution documents and replaced them with plain-text
documents. They provided little value and introduced a lot of complexity
to the build system.
* Simplified the output of atf-version by not attempting to print a
revision number when building form a distfile. Makes the build system
easier to maintain.
which are known to fail, e.g.:
atf_tc_set_md_var(tc, "xfail", "PR kern/43456");
Expected failures do not count towards the ultimate pass/fail result
from the test run:
pain-rustique:39:~/<2>src/tests/fs/ptyfs> atf-run t_nullpts | atf-report
Tests root: /home/pooka/src/wholesrc2/src/tests/fs/ptyfs
t_nullpts (1/1): 1 test cases
nullrevoke: Expected failure: PR kern/43456
Summary for 1 test programs:
0 passed test cases.
0 failed test cases.
1 expected failures.
0 skipped test cases.
pain-rustique:40:~/<2>src/tests/fs/ptyfs> echo $?
0
However, an xfail test which passes will count as a failure, i.e.
xfail inverts test case success/fail. This way we can get a better
sense from the ultimate verdict of the NetBSD atf run by seeing if
there were any unexpected failures, i.e. new regressions.
This feature will be present in the upcoming atf 0.10 release,
possibly with finer grained control.
patch reviewed by jmmv
----------
Fix detection of crashed test cases
Prevent cross-test-case contamination that occured when a first test case
passed and the second crashed. The second could pick up the result of the
first test case and not be reported as failed.
Similarly, change the way timed out test cases are reported back to the
caller. The creation of a temporary results file was just a really stupid
way of passing information around and introduced false positives if the
test case creates a results file before timing out.
Fixes ticket #35.
----------
Problem originally reported by pooka@.
jmmv