NetBSD/regress
martin ed3f875e0a gcc constant folding at compile time works around the real points of these
tests - so enforce -O0.
2007-11-07 00:08:50 +00:00
..
bin Fix typo in shell-bang line. Not really needed since the script is not 2007-07-15 09:23:53 +00:00
games
include Add a check for correct definition of UCHAR_MAX/USHRT_MAX and related 2007-08-31 16:20:50 +00:00
lib gcc constant folding at compile time works around the real points of these 2007-11-07 00:08:50 +00:00
libexec Descending into ld.elf_so only makes sense if ${MKPIC} != "no". 2007-05-23 09:39:53 +00:00
sys Fix duplicate year in copyright notice. 2007-07-24 11:29:16 +00:00
usr.bin Add two very simple regression tests for the "toupper" and "tolower" 2007-10-25 14:51:18 +00:00
Makefile Regression test for RTLD expansion code. 2007-05-18 20:28:11 +00:00
README

README

$NetBSD: README,v 1.1 2006/03/14 09:46:34 martin Exp $

This part of the source tree contains regression tests. There are special
make targets and rules to follow. Most of these, however, are currently not
enforced, and most tests available are not conforming.

We hope to fix this someday. If you add new tests, please try to be conforming.

What is a regression test?

  A regression test is run by a makefile in a test directory (see below).
  Each makefile may run multiple tests.

What is a test directory?

  A directory in this part of the tree is a regression test directory. It
  contains a Makefile which implements the additional "regress" target,
  and runs all it's regression tests during this target.

May the make progress be stopped on failures?

  No, the "make regress" target should succeed, unless some regression 
  binaries could not be build, disk is full or other catastrophic failures
  outside of the tested subsystem happen. A failing regression test should
  log the failure (see below), but not make the target itself fail.

What are the possible results of a regression test?

  A test may either

  - succeed, in which case it logs "PASSED" (see below for logging details)
  - fail, in which case it logs "FAILED"
  - not be able to run, in which case it logs "SKIPPED" and the reason
    for the skip in the comment field (see below)

  Typical reasons for tests to not being run are missing kernel options,
  or missing privileges (test needs root, "make regress" is invoked by
  mere mortal or vice versa). A test may not fail because of such
  environmental issues, it must detect and properly log the problem.

  If a test directory contains tests that may be skipped, it should have
  a README file explaining the prerequisites (e.g. needed kernel options)

  In future, we will mark affected makefiles and optimize run/skipped test
  during repeated runs with differing privileges - but currently there is no
  make framework in place to handle this.

How and when does a test log results?

  If the make/environment variable ${REGRESS_LOG} is defined, the final
  results (and only those) should be logged to the file named by that
  variable. We will, in the future, add make targets for this purpose.
  The log format is line oriented, one line used per test. Each line
  consists of the directory where the Makefile lives, followed by the
  test name and the result (see above: PASSED, FAILED, SKIPPED).
  Following this an optional comment may be added. For SKIPPED tests the
  comment is not optional. Fields are separated by spaces.