From 3c6dee26b1e8379a511b5273105e7ab042ef66b7 Mon Sep 17 00:00:00 2001 From: pooka Date: Wed, 18 Aug 2010 21:28:03 +0000 Subject: [PATCH] new tests should use atf -> new tests must use atf --- regress/README | 61 +++----------------------------------------------- 1 file changed, 3 insertions(+), 58 deletions(-) diff --git a/regress/README b/regress/README index b861eee7c2ec..8183aab29d68 100644 --- a/regress/README +++ b/regress/README @@ -1,61 +1,6 @@ -$NetBSD: README,v 1.2 2009/09/14 17:15:32 apb Exp $ + $NetBSD: README,v 1.3 2010/08/18 21:28:03 pooka Exp $ -NOTE: New tests should use the ATF framework; see atf(7) +New tests must use the ATF framework; see atf(7) and the src/tests directory. ------------------------------------------------------------------------- -This part of the source tree contains regression tests. There are special -make targets and rules to follow. Most of these, however, are currently not -enforced, and most tests available are not conforming. - -We hope to fix this someday. If you add new tests, please try to be conforming. - -What is a regression test? - - A regression test is run by a makefile in a test directory (see below). - Each makefile may run multiple tests. - -What is a test directory? - - A directory in this part of the tree is a regression test directory. It - contains a Makefile which implements the additional "regress" target, - and runs all it's regression tests during this target. - -May the make progress be stopped on failures? - - No, the "make regress" target should succeed, unless some regression - binaries could not be build, disk is full or other catastrophic failures - outside of the tested subsystem happen. A failing regression test should - log the failure (see below), but not make the target itself fail. - -What are the possible results of a regression test? - - A test may either - - - succeed, in which case it logs "PASSED" (see below for logging details) - - fail, in which case it logs "FAILED" - - not be able to run, in which case it logs "SKIPPED" and the reason - for the skip in the comment field (see below) - - Typical reasons for tests to not being run are missing kernel options, - or missing privileges (test needs root, "make regress" is invoked by - mere mortal or vice versa). A test may not fail because of such - environmental issues, it must detect and properly log the problem. - - If a test directory contains tests that may be skipped, it should have - a README file explaining the prerequisites (e.g. needed kernel options) - - In future, we will mark affected makefiles and optimize run/skipped test - during repeated runs with differing privileges - but currently there is no - make framework in place to handle this. - -How and when does a test log results? - - If the make/environment variable ${REGRESS_LOG} is defined, the final - results (and only those) should be logged to the file named by that - variable. We will, in the future, add make targets for this purpose. - The log format is line oriented, one line used per test. Each line - consists of the directory where the Makefile lives, followed by the - test name and the result (see above: PASSED, FAILED, SKIPPED). - Following this an optional comment may be added. For SKIPPED tests the - comment is not optional. Fields are separated by spaces. +All exceptions need prior approval from core.