Import atf-0.13:

Experimental version released on March 31st, 2011.

This is the first release after the creation of the Kyua project, a more
modular and reliable replacement for ATF.  From now on, ATF will change to
accomodate the transition to this new codebase, but ATF will still continue
to see development in the short/medium term.  Check out the project page at
http://code.google.com/p/kyua/ for more details.

The changes in this release are:

* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
  new package that aims to replace atf-run and atf-report.  The ATF tests
  can be run with the new system by issuing a 'make installcheck-kyua' from
  the top-level directory of the project (assuming the 'kyua' binary is
  available during the configuration stage of ATF).

* atf-run and atf-report are now in maintenance mode (but *not* deprecated
  yet!).  Kyua already implements a new, much more reliable runtime engine
  that provides similar features to these tools.  That said, it is not
  complete yet so all development efforts should go towards it.

* If GDB is installed, atf-run dumps the stack trace of crashing test
  programs in an attempt to aid debugging.  Contributed by Antti Kantee.

* Reverted default timeout change in previous release and reset its value
  to 5 minutes.  This was causing several issues, specially when running
  the existing NetBSD test suite in qemu.

* Fixed the 'match' output checker in atf-check to properly validate the
  last line of a file even if it does not have a newline.

* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
  check for the presence (or lack thereof) of an element in a collection.

* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
  when the cleanup of a test case triggered asynchronous modifications to
  its work directory (e.g. killing a daemon process that cleans up a pid
  file in the work directory).

* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
  instead of just listing them as having 0 test cases.
This commit is contained in:
jmmv 2011-03-31 16:41:07 +00:00
parent 35a190a334
commit eb215e31ca
28 changed files with 355 additions and 10 deletions

View File

@ -5,7 +5,7 @@ Redistribution terms Automated Testing Framework
License
*******
Copyright (c) 2007, 2008, 2009, 2010 The NetBSD Foundation, Inc.
Copyright (c) 2007, 2008, 2009, 2010, 2011 The NetBSD Foundation, Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without

11
external/bsd/atf/dist/Kyuafile vendored Normal file
View File

@ -0,0 +1,11 @@
syntax("kyuafile", 1)
test_suite("atf")
include("atf-c/Kyuafile")
include("atf-c++/Kyuafile")
include("atf-sh/Kyuafile")
include("test-programs/Kyuafile")
include("atf-config/Kyuafile")
include("atf-report/Kyuafile")
include("atf-run/Kyuafile")

View File

@ -2,6 +2,52 @@ Major changes between releases Automated Testing Framework
===========================================================================
Changes in version 0.13
***********************
Experimental version released on March 31st, 2011.
This is the first release after the creation of the Kyua project, a more
modular and reliable replacement for ATF. From now on, ATF will change to
accomodate the transition to this new codebase, but ATF will still continue
to see development in the short/medium term. Check out the project page at
http://code.google.com/p/kyua/ for more details.
The changes in this release are:
* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
new package that aims to replace atf-run and atf-report. The ATF tests
can be run with the new system by issuing a 'make installcheck-kyua' from
the top-level directory of the project (assuming the 'kyua' binary is
available during the configuration stage of ATF).
* atf-run and atf-report are now in maintenance mode (but *not* deprecated
yet!). Kyua already implements a new, much more reliable runtime engine
that provides similar features to these tools. That said, it is not
complete yet so all development efforts should go towards it.
* If GDB is installed, atf-run dumps the stack trace of crashing test
programs in an attempt to aid debugging. Contributed by Antti Kantee.
* Reverted default timeout change in previous release and reset its value
to 5 minutes. This was causing several issues, specially when running
the existing NetBSD test suite in qemu.
* Fixed the 'match' output checker in atf-check to properly validate the
last line of a file even if it does not have a newline.
* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
check for the presence (or lack thereof) of an element in a collection.
* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
when the cleanup of a test case triggered asynchronous modifications to
its work directory (e.g. killing a daemon process that cleans up a pid
file in the work directory).
* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
instead of just listing them as having 0 test cases.
Changes in version 0.12
***********************

14
external/bsd/atf/dist/atf-c++/Kyuafile vendored Normal file
View File

@ -0,0 +1,14 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="atf_c++_test"}
atf_test_program{name="build_test"}
atf_test_program{name="check_test"}
atf_test_program{name="config_test"}
atf_test_program{name="macros_test"}
atf_test_program{name="pkg_config_test"}
atf_test_program{name="tests_test"}
atf_test_program{name="utils_test"}
include("detail/Kyuafile")

View File

@ -26,7 +26,7 @@
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd October 21, 2010
.Dd December 10, 2010
.Dt ATF-C++-API 3
.Os
.Sh NAME
@ -38,7 +38,9 @@
.Nm ATF_REQUIRE ,
.Nm ATF_REQUIRE_EQ ,
.Nm ATF_REQUIRE_ERRNO ,
.Nm ATF_REQUIRE_IN ,
.Nm ATF_REQUIRE_MATCH ,
.Nm ATF_REQUIRE_NOT_IN ,
.Nm ATF_REQUIRE_THROW ,
.Nm ATF_REQUIRE_THROW_RE ,
.Nm ATF_SKIP ,
@ -59,7 +61,9 @@
.Fn ATF_REQUIRE "expression"
.Fn ATF_REQUIRE_EQ "expression_1" "expression_2"
.Fn ATF_REQUIRE_ERRNO "exp_errno" "bool_expression"
.Fn ATF_REQUIRE_IN "element" "collection"
.Fn ATF_REQUIRE_MATCH "regexp" "string_expression"
.Fn ATF_REQUIRE_NOT_IN "element" "collection"
.Fn ATF_REQUIRE_THROW "expected_exception" "statement"
.Fn ATF_REQUIRE_THROW_RE "expected_exception" "regexp" "statement"
.Fn ATF_SKIP "reason"
@ -313,10 +317,18 @@ takes an expression and raises a failure if it evaluates to false.
takes two expressions and raises a failure if the two do not evaluate to
the same exact value.
.Pp
.Fn ATF_REQUIRE_IN
takes an element and a collection and validates that the element is present in
the collection.
.Pp
.Fn ATF_REQUIRE_MATCH
takes a regular expression and a string and raises a failure if the regular
expression does not match the string.
.Pp
.Fn ATF_REQUIRE_NOT_IN
takes an element and a collection and validates that the element is not present
in the collection.
.Pp
.Fn ATF_REQUIRE_THROW
takes the name of an exception and a statement and raises a failure if
the statement does not throw the specified exception.

View File

@ -0,0 +1,13 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="application_test"}
atf_test_program{name="env_test"}
atf_test_program{name="exceptions_test"}
atf_test_program{name="expand_test"}
atf_test_program{name="fs_test"}
atf_test_program{name="parser_test"}
atf_test_program{name="sanity_test"}
atf_test_program{name="text_test"}
atf_test_program{name="ui_test"}

View File

@ -1,7 +1,7 @@
//
// Automated Testing Framework (atf)
//
// Copyright (c) 2007, 2008, 2010 The NetBSD Foundation, Inc.
// Copyright (c) 2007, 2008, 2010, 2011 The NetBSD Foundation, Inc.
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
@ -178,6 +178,7 @@ impl::app::process_options(void)
}
int ch;
const int old_opterr = ::opterr;
::opterr = 0;
while ((ch = ::getopt(m_argc, m_argv, optstr.c_str())) != -1) {
switch (ch) {
@ -201,6 +202,7 @@ impl::app::process_options(void)
m_argv += ::optind;
// Clear getopt state just in case the test wants to use it.
opterr = old_opterr;
optind = 1;
#if defined(HAVE_OPTRESET)
optreset = 1;

View File

@ -34,6 +34,7 @@
#include <cstdarg>
#include <cstdio>
#include <cstring>
#include <new>
extern "C" {
#include "../../atf-c/error.h"

View File

@ -31,6 +31,7 @@
#define _ATF_CXX_EXCEPTIONS_HPP_
#include <stdexcept>
#include <string>
extern "C" {
struct atf_error;

View File

@ -32,6 +32,7 @@ extern "C" {
}
#include <cstdio>
#include <new>
#include "../macros.hpp"

View File

@ -31,6 +31,7 @@ extern "C" {
#include <sys/ioctl.h>
#include <termios.h>
#include <unistd.h>
}
#include <sstream>

View File

@ -101,6 +101,12 @@
} \
} while (false)
#define ATF_REQUIRE_IN(element, collection) \
ATF_REQUIRE((collection).find(element) != (collection).end())
#define ATF_REQUIRE_NOT_IN(element, collection) \
ATF_REQUIRE((collection).find(element) == (collection).end())
#define ATF_REQUIRE_MATCH(regexp, string) \
do { \
if (!atf::tests::detail::match(regexp, string)) { \

View File

@ -125,6 +125,25 @@ ATF_TEST_CASE_BODY(h_require_eq)
create_ctl_file(*this, "after");
}
ATF_TEST_CASE(h_require_in);
ATF_TEST_CASE_HEAD(h_require_in)
{
set_md_var("descr", "Helper test case");
}
ATF_TEST_CASE_BODY(h_require_in)
{
const std::string element = get_config_var("value");
std::set< std::string > collection;
collection.insert("foo");
collection.insert("bar");
collection.insert("baz");
create_ctl_file(*this, "before");
ATF_REQUIRE_IN(element, collection);
create_ctl_file(*this, "after");
}
ATF_TEST_CASE(h_require_match);
ATF_TEST_CASE_HEAD(h_require_match)
{
@ -140,6 +159,25 @@ ATF_TEST_CASE_BODY(h_require_match)
create_ctl_file(*this, "after");
}
ATF_TEST_CASE(h_require_not_in);
ATF_TEST_CASE_HEAD(h_require_not_in)
{
set_md_var("descr", "Helper test case");
}
ATF_TEST_CASE_BODY(h_require_not_in)
{
const std::string element = get_config_var("value");
std::set< std::string > collection;
collection.insert("foo");
collection.insert("bar");
collection.insert("baz");
create_ctl_file(*this, "before");
ATF_REQUIRE_NOT_IN(element, collection);
create_ctl_file(*this, "after");
}
ATF_TEST_CASE(h_require_throw);
ATF_TEST_CASE_HEAD(h_require_throw)
{
@ -373,6 +411,50 @@ ATF_TEST_CASE_BODY(require_eq)
}
}
ATF_TEST_CASE(require_in);
ATF_TEST_CASE_HEAD(require_in)
{
set_md_var("descr", "Tests the ATF_REQUIRE_IN macro");
}
ATF_TEST_CASE_BODY(require_in)
{
struct test {
const char *value;
bool ok;
} *t, tests[] = {
{ "foo", true },
{ "bar", true },
{ "baz", true },
{ "xxx", false },
{ "fooa", false },
{ "foo ", false },
{ NULL, false }
};
const atf::fs::path before("before");
const atf::fs::path after("after");
for (t = &tests[0]; t->value != NULL; t++) {
atf::tests::vars_map config;
config["value"] = t->value;
run_h_tc< ATF_TEST_CASE_NAME(h_require_in) >(config);
ATF_REQUIRE(atf::fs::exists(before));
if (t->ok) {
ATF_REQUIRE(grep_file("result", "^passed"));
ATF_REQUIRE(atf::fs::exists(after));
} else {
ATF_REQUIRE(grep_file("result", "^failed: "));
ATF_REQUIRE(!atf::fs::exists(after));
}
atf::fs::remove(before);
if (t->ok)
atf::fs::remove(after);
}
}
ATF_TEST_CASE(require_match);
ATF_TEST_CASE_HEAD(require_match)
{
@ -419,6 +501,50 @@ ATF_TEST_CASE_BODY(require_match)
}
}
ATF_TEST_CASE(require_not_in);
ATF_TEST_CASE_HEAD(require_not_in)
{
set_md_var("descr", "Tests the ATF_REQUIRE_NOT_IN macro");
}
ATF_TEST_CASE_BODY(require_not_in)
{
struct test {
const char *value;
bool ok;
} *t, tests[] = {
{ "foo", false },
{ "bar", false },
{ "baz", false },
{ "xxx", true },
{ "fooa", true },
{ "foo ", true },
{ NULL, false }
};
const atf::fs::path before("before");
const atf::fs::path after("after");
for (t = &tests[0]; t->value != NULL; t++) {
atf::tests::vars_map config;
config["value"] = t->value;
run_h_tc< ATF_TEST_CASE_NAME(h_require_not_in) >(config);
ATF_REQUIRE(atf::fs::exists(before));
if (t->ok) {
ATF_REQUIRE(grep_file("result", "^passed"));
ATF_REQUIRE(atf::fs::exists(after));
} else {
ATF_REQUIRE(grep_file("result", "^failed: "));
ATF_REQUIRE(!atf::fs::exists(after));
}
atf::fs::remove(before);
if (t->ok)
atf::fs::remove(after);
}
}
ATF_TEST_CASE(require_throw);
ATF_TEST_CASE_HEAD(require_throw)
{
@ -635,7 +761,9 @@ ATF_INIT_TEST_CASES(tcs)
ATF_ADD_TEST_CASE(tcs, check_errno);
ATF_ADD_TEST_CASE(tcs, require);
ATF_ADD_TEST_CASE(tcs, require_eq);
ATF_ADD_TEST_CASE(tcs, require_in);
ATF_ADD_TEST_CASE(tcs, require_match);
ATF_ADD_TEST_CASE(tcs, require_not_in);
ATF_ADD_TEST_CASE(tcs, require_throw);
ATF_ADD_TEST_CASE(tcs, require_throw_re);
ATF_ADD_TEST_CASE(tcs, require_errno);

16
external/bsd/atf/dist/atf-c/Kyuafile vendored Normal file
View File

@ -0,0 +1,16 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="atf_c_test"}
atf_test_program{name="build_test"}
atf_test_program{name="check_test"}
atf_test_program{name="config_test"}
atf_test_program{name="error_test"}
atf_test_program{name="macros_test"}
atf_test_program{name="pkg_config_test"}
atf_test_program{name="tc_test"}
atf_test_program{name="tp_test"}
atf_test_program{name="utils_test"}
include("detail/Kyuafile")

View File

@ -26,7 +26,7 @@
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd November 1, 2010
.Dd December 26, 2010
.Dt ATF-C-API 3
.Os
.Sh NAME
@ -56,6 +56,12 @@
.Nm ATF_TC_WITHOUT_HEAD ,
.Nm ATF_TP_ADD_TC ,
.Nm ATF_TP_ADD_TCS ,
.Nm atf_tc_get_config_var ,
.Nm atf_tc_get_config_var_wd ,
.Nm atf_tc_get_config_var_as_bool ,
.Nm atf_tc_get_config_var_as_bool_wd ,
.Nm atf_tc_get_config_var_as_long ,
.Nm atf_tc_get_config_var_as_long_wd ,
.Nm atf_no_error ,
.Nm atf_tc_expect_death ,
.Nm atf_tc_expect_exit ,
@ -96,6 +102,12 @@
.Fn ATF_TC_WITHOUT_HEAD "name"
.Fn ATF_TP_ADD_TC "tp_name" "tc_name"
.Fn ATF_TP_ADD_TCS "tp_name"
.Fn atf_tc_get_config_var "tc" "varname"
.Fn atf_tc_get_config_var_wd "tc" "variable_name" "default_value"
.Fn atf_tc_get_config_var_as_bool "tc" "variable_name"
.Fn atf_tc_get_config_var_as_bool_wd "tc" "variable_name" "default_value"
.Fn atf_tc_get_config_var_as_long "tc" "variable_name"
.Fn atf_tc_get_config_var_as_long_wd "tc" "variable_name" "default_value"
.Fn atf_no_error
.Fn atf_tc_expect_death "reason" "..."
.Fn atf_tc_expect_exit "exitcode" "reason" "..."
@ -248,6 +260,16 @@ and the
.Ft long
.Fn atf_tc_get_config_var_as_long_wd
functions, which can be called in any of the three parts of a test case.
.Pp
The
.Sq _wd
variants take a default value for the variable which is returned if the
variable is not defined.
The other functions without the
.Sq _wd
suffix
.Em require
the variable to be defined.
.Ss Access to the source directory
It is possible to get the path to the test case's source directory from any
of its three components by querying the

View File

@ -0,0 +1,14 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="dynstr_test"}
atf_test_program{name="env_test"}
atf_test_program{name="fs_test"}
atf_test_program{name="list_test"}
atf_test_program{name="map_test"}
atf_test_program{name="process_test"}
atf_test_program{name="sanity_test"}
atf_test_program{name="test_helpers_test"}
atf_test_program{name="text_test"}
atf_test_program{name="user_test"}

View File

@ -1,7 +1,7 @@
/*
* Automated Testing Framework (atf)
*
* Copyright (c) 2008, 2009, 2010 The NetBSD Foundation, Inc.
* Copyright (c) 2008, 2009, 2010, 2011 The NetBSD Foundation, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@ -322,11 +322,13 @@ process_params(int argc, char **argv, struct params *p)
{
atf_error_t err;
int ch;
int old_opterr;
err = params_init(p, argv[0]);
if (atf_is_error(err))
goto out;
old_opterr = opterr;
opterr = 0;
while (!atf_is_error(err) &&
(ch = getopt(argc, argv, GETOPT_POSIX ":lr:s:v:")) != -1) {
@ -360,6 +362,7 @@ process_params(int argc, char **argv, struct params *p)
argv += optind;
/* Clear getopt state just in case the test wants to use it. */
opterr = old_opterr;
optind = 1;
#if defined(HAVE_OPTRESET)
optreset = 1;

View File

@ -0,0 +1,5 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="integration_test"}

View File

@ -0,0 +1,6 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="integration_test"}
atf_test_program{name="reader_test"}

13
external/bsd/atf/dist/atf-run/Kyuafile vendored Normal file
View File

@ -0,0 +1,13 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="atffile_test"}
atf_test_program{name="config_test"}
atf_test_program{name="fs_test"}
atf_test_program{name="integration_test"}
atf_test_program{name="io_test"}
atf_test_program{name="requirements_test"}
atf_test_program{name="signals_test"}
atf_test_program{name="test_program_test"}
atf_test_program{name="user_test"}

View File

@ -165,7 +165,7 @@ The purpose of this hook is to write additional
stanzas to the top of the output report; these are defined by the
.Sq application/X-atf-tps format
described in
.Xr atf-formats 1 .
.Xr atf-formats 5 .
Always use the
.Sq atf_tps_writer_info
function to print these.

View File

@ -37,6 +37,7 @@ extern "C" {
#include <cstdlib>
#include <fstream>
#include <iomanip>
#include <ios>
#include <iostream>
#include <string>

View File

@ -1,7 +1,7 @@
#
# Automated Testing Framework (atf)
#
# Copyright (c) 2007, 2008 The NetBSD Foundation, Inc.
# Copyright (c) 2007, 2008, 2011 The NetBSD Foundation, Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@ -62,7 +62,7 @@ default_info_start_hook()
set -- $(env)
val=${1}; shift
while [ ${#} -gt 0 ]; do
if echo "${1}" | grep '^[a-zA-Z_][a-zA-Z_]*=' >/dev/null; then
if echo "${1}" | grep '^[a-zA-Z0-0_][a-zA-Z0-9_]*=' >/dev/null; then
atf_tps_writer_info "env" "${val}"
val="${1}"
else

View File

@ -31,11 +31,13 @@ extern "C" {
#include <sys/types.h>
#include <pwd.h>
#include <unistd.h>
#include "../atf-c/detail/user.h"
}
#include <stdexcept>
#include <string>
#include "../atf-c++/detail/sanity.hpp"

11
external/bsd/atf/dist/atf-sh/Kyuafile vendored Normal file
View File

@ -0,0 +1,11 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="tc_test"}
atf_test_program{name="tp_test"}
atf_test_program{name="normalize_test"}
atf_test_program{name="config_test"}
atf_test_program{name="atf-check_test"}
atf_test_program{name="atf_check_test"}
atf_test_program{name="integration_test"}

View File

@ -65,10 +65,12 @@ construct_script(const char* filename)
{
const std::string libexecdir = atf::config::get("atf_libexecdir");
const std::string pkgdatadir = atf::config::get("atf_pkgdatadir");
const std::string shell = atf::config::get("atf_shell");
std::string* command = new std::string();
command->reserve(512);
(*command) += ("Atf_Check='" + libexecdir + "/atf-check' ; " +
"Atf_Shell='" + shell + "' ; " +
". " + pkgdatadir + "/libatf-sh.subr ; " +
". " + fix_plain_name(filename) + " ; " +
"main \"${@}\"");

View File

@ -1,7 +1,7 @@
.\"
.\" Automated Testing Framework (atf)
.\"
.\" Copyright (c) 2007, 2008, 2010 The NetBSD Foundation, Inc.
.\" Copyright (c) 2007, 2008, 2010, 2011 The NetBSD Foundation, Inc.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
@ -26,7 +26,7 @@
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd June 18, 2010
.Dd February 6, 2011
.Dt ATF-TEST-PROGRAM 1
.Os
.Sh NAME
@ -43,6 +43,10 @@
.Sh DESCRIPTION
Test programs written using the ATF libraries all share a common user
interface, which is what this manual page describes.
.Em NOTE: There is no binary known as
.Nm ;
.Em what is described in this manual page is the command-line interface
.Em exposed by the atf-c, atf-c++ and atf-sh bindings .
.Pp
In the first synopsis form, the test program will execute the provided
test case and print its results to the standard output, unless otherwise

View File

@ -0,0 +1,10 @@
syntax("kyuafile", 1)
test_suite("atf")
atf_test_program{name="config_test"}
atf_test_program{name="expect_test"}
atf_test_program{name="fork_test"}
atf_test_program{name="meta_data_test"}
atf_test_program{name="srcdir_test"}
atf_test_program{name="result_test"}