249 lines
7.9 KiB
Groff
249 lines
7.9 KiB
Groff
.\" $NetBSD: tests.kyua.7,v 1.7 2015/07/30 03:21:33 jnemeth Exp $
|
|
.\"
|
|
.\" Copyright (c) 2010 The NetBSD Foundation, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.Dd July 29, 2015
|
|
.Dt TESTS 7
|
|
.Os
|
|
.Sh NAME
|
|
.Nm tests
|
|
.Nd introduction to the NetBSD test suite
|
|
.Sh DESCRIPTION
|
|
The
|
|
.Nx
|
|
test suite provides a collection of automated tests for two major purposes.
|
|
On the one hand, the test suite aids
|
|
.Em developers
|
|
in catching bugs and regressions in the code when they performing modifications
|
|
to the source tree.
|
|
On the other hand, the test suite allows
|
|
.Em end users
|
|
(and, in particular, system administrators) to verify that fresh installations
|
|
of the
|
|
.Nx
|
|
operating system behave correctly in their hardware platform and also to ensure
|
|
that the system does not suffer from regressions during regular system
|
|
operation and maintenance.
|
|
.Pp
|
|
The
|
|
.Nx
|
|
test suite is distributed as a separate installation set, named
|
|
.Pa tests.tgz ,
|
|
and the test programs are all installed under the
|
|
.Pa /usr/tests
|
|
hierarchy.
|
|
.Pp
|
|
This manual page describes how to execute the test suite and how to configure
|
|
some of its optional features.
|
|
.Ss When to run the tests?
|
|
Before diving into the details of how to run the test suite, here are some
|
|
scenarios in which you should be running them:
|
|
.Bl -bullet -offset indent
|
|
.It
|
|
After a fresh installation of
|
|
.Nx
|
|
to ensure that the system works correctly on your hardware platform.
|
|
.It
|
|
After an upgrade of
|
|
.Nx
|
|
to a different version to ensure that the new code works well on your
|
|
hardware platform and that the upgrade did not introduce regressions in your
|
|
configuration.
|
|
.It
|
|
After performing changes to the source tree to catch any bugs and/or regressions
|
|
introduced by the modifications.
|
|
.It
|
|
Periodically, maybe from a
|
|
.Xr cron 8
|
|
job, to ensure that any changes to the system (such as the installation of
|
|
third-party packages or manual modifications to configuration files) do not
|
|
introduce unexpected failures.
|
|
.El
|
|
.Ss Installing the tests
|
|
If you chose to install the
|
|
.Pa tests.tgz
|
|
distribution set while setting up your
|
|
.Nx
|
|
system, the tests are already available in
|
|
.Pa /usr/tests .
|
|
Otherwise, install the set now by running:
|
|
.Bd -literal -offset indent
|
|
# cd /
|
|
# tar xzpf /path/to/tests.tgz
|
|
.Ed
|
|
.Ss Running the tests
|
|
Use the following command to run the whole test suite:
|
|
.Bd -literal -offset indent
|
|
$ kyua test -k /usr/tests/Kyuafile
|
|
.Ed
|
|
.Pp
|
|
The above will go through all test programs in
|
|
.Pa /usr/tests
|
|
recursively, execute them, store their results and debugging data in Kyua
|
|
database (by default in
|
|
.Pa ~/.kyua/store.db ) ,
|
|
and print a summary of the results.
|
|
This summary includes a brief count of all total tests run and how many of
|
|
them failed.
|
|
.Pp
|
|
It is possible to restrict which tests to run by providing their names in
|
|
the command line.
|
|
For example, this would execute the tests for the
|
|
.Xr cp 1
|
|
and
|
|
.Xr cut 1
|
|
utilities:
|
|
.Bd -literal -offset indent
|
|
$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut
|
|
.Ed
|
|
.Ss Obtaining reports of the tests execution
|
|
Additional information of the results of the execution can be later extracted
|
|
from the database by using the various reporting commands of Kyua.
|
|
For example, the following would extract a plain-text report of the executed
|
|
tests and show which ones failed:
|
|
.Bd -literal -offset indent
|
|
$ kyua report
|
|
.Ed
|
|
.Pp
|
|
This other example would generate an HTML report ready to be published on a
|
|
web server, possibly the built-in
|
|
.Xr httpd 8 :
|
|
.Bd -literal -offset indent
|
|
$ kyua report-html --output ~/public_html/tests
|
|
.Ed
|
|
.Pp
|
|
For further details on the command-line interface of Kyua, please refer
|
|
to its manual page
|
|
.Xr kyua 1 .
|
|
.Ss Test environment considerations
|
|
Tests can be invoked as an unprivileged user, in which case tests that
|
|
require privileges will be skipped.
|
|
If run as root, an unprivileged user will be used for tests that
|
|
do not require privileges.
|
|
For maximal coverage, the standard approach is to invoke tests as root.
|
|
.Pp
|
|
Ideally, tests are self-contained and do not either depend on or
|
|
perturb the host environment, aside from skipping tests when optional
|
|
facilities are not available.
|
|
In reality, tests load and unload modules, and do other things that
|
|
might cause problems.
|
|
While it is not entirely safe to run tests on a multi-user system,
|
|
permanent problems or crashes from doing so are viewed as bugs and
|
|
should be reported.
|
|
.Ss Configuring the tests
|
|
Some test cases in the
|
|
.Nx
|
|
test suite require the administrator to manually set up some configuration
|
|
properties before they can run.
|
|
Unless these properties are defined, the tests that require them will be marked
|
|
as skipped and thus they will not be really executed.
|
|
.Pp
|
|
Test suites are configured by defining the values to their configuration
|
|
variables in
|
|
.Pa /etc/kyua/kyua.conf .
|
|
The format of this file is detailed in
|
|
.Xr kyua.conf 5 .
|
|
.Pp
|
|
The following configuration variables are available in the
|
|
.Nx
|
|
test suite:
|
|
.Bl -tag -width "fstype"
|
|
.It fstype
|
|
When set to a filesystem type, restrict tests programs from the
|
|
.Pa /usr/tests/fs/vfs/ .
|
|
.El
|
|
.Ss What to do if something fails?
|
|
If there is
|
|
.Em any failure
|
|
during the execution of the test suite, please considering reporting it to the
|
|
.Nx
|
|
developers so that the failure can be analyzed and fixed.
|
|
To do so, either send a message to the appropriate mailing list or file a
|
|
problem report.
|
|
For more details please refer to:
|
|
.Bl -bullet -offset indent -compact
|
|
.It
|
|
.Lk http://www.netbsd.org/mailinglists/ "NetBSD mailing lists"
|
|
.It
|
|
.Lk http://www.netbsd.org/support/send-pr.html "NetBSD Problem Reports"
|
|
.El
|
|
.Sh FILES
|
|
.Bl -tag -compact -width etcXatfXNetBSDXconfXX
|
|
.It Pa /etc/kyua/kyua.conf
|
|
System-wide configuration file for
|
|
.Xr kyua 1 .
|
|
.It Pa ~/.kyua/kyua.conf
|
|
User-specific configuration file for
|
|
.Xr kyua 1 ;
|
|
overrides the system file.
|
|
.It Pa ~/.kyua/store.db
|
|
Default database used by Kyua to maintain the data of the executed tests.
|
|
.It Pa /usr/tests/
|
|
Location of the
|
|
.Nx
|
|
test suite.
|
|
.It Pa /usr/tests/Kyuafile
|
|
Top-level test suite definition file.
|
|
.El
|
|
.Sh SEE ALSO
|
|
.Xr kyua 1 ,
|
|
.Xr kyua-report 1 ,
|
|
.Xr kyua-test 1
|
|
.Sh HISTORY
|
|
The collection of test programs in
|
|
.Pa /usr/tests
|
|
first appeared in
|
|
.Nx 5.0
|
|
and has been growing since then.
|
|
.Pp
|
|
The
|
|
.Nm
|
|
manual page first appeared in
|
|
.Nx 6.0
|
|
and was updated in
|
|
.Nx 7.0
|
|
to describe the execution of the tests with Kyua rather than with ATF.
|
|
.Pp
|
|
The ATF testing framework was first distributed with
|
|
.Nx 5.0
|
|
and the runtime tools of this framework are being phased out in
|
|
.Nx 7.0 .
|
|
Note that the
|
|
.Em libraries
|
|
that ship with ATF are still in active use and are not deprecated.
|
|
.Pp
|
|
The Kyua testing toolkit was first distributed with
|
|
.Nx 7.0 .
|
|
The
|
|
.Xr atf-run 1
|
|
and
|
|
.Xr atf-report 1
|
|
tools were replaced as part of this import to be backwards-compatibility
|
|
wrappers around
|
|
.Xr kyua 1 .
|
|
.Sh AUTHORS
|
|
.An Julio Merino Aq Mt jmmv@NetBSD.org
|