866b001ea5
Initial import of the Automated Testing Framework, version 0.3, a project that provides a framework to easily implement test cases for the NetBSD operating system and some tools to run them and generate reports with the results. Note that this is just the framework (libraries and tools), which is and will be maintained externally. The tests themselves will come later, will be put under the 'tests' hierarchy and will be managed exclusively under the NetBSD CVS tree given that they are tied to the operating system. The work done until version 0.1 was sponsored by the Google Summer of Code 2007 program and mentored by martin@.
332 lines
11 KiB
Groff
332 lines
11 KiB
Groff
.\"
|
|
.\" Automated Testing Framework (atf)
|
|
.\"
|
|
.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\" 3. All advertising materials mentioning features or use of this
|
|
.\" software must display the following acknowledgement:
|
|
.\" This product includes software developed by the NetBSD
|
|
.\" Foundation, Inc. and its contributors.
|
|
.\" 4. Neither the name of The NetBSD Foundation nor the names of its
|
|
.\" contributors may be used to endorse or promote products derived
|
|
.\" from this software without specific prior written permission.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.Dd November 4, 2007
|
|
.Dt ATF-FORMATS 5
|
|
.Os
|
|
.Sh NAME
|
|
.Nm atf-formats
|
|
.Nd machine-parseable data formats used by ATF
|
|
.Sh DESCRIPTION
|
|
This manual page describes the multiple data formats used in ATF.
|
|
These formats affect configuration files, control files and any data that
|
|
is externalized or internalized by the tools.
|
|
.Pp
|
|
Data files are always organized as follows:
|
|
.Bd -literal -offset indent
|
|
Header1: Value1 \\
|
|
... | head
|
|
HeaderN: ValueN /
|
|
mandatory blank line
|
|
Free-form text contents \\
|
|
... | body
|
|
... /
|
|
.Ed
|
|
.Pp
|
|
A file must always contain a
|
|
.Sq Content-Type
|
|
header and must always separate that header from the body with a blank
|
|
line, even if the body is empty.
|
|
.Pp
|
|
The
|
|
.Sq Content-Type
|
|
is always of the form:
|
|
.Bd -literal -offset indent
|
|
Content-Type: application/X-atf-<subtype>; version="<version>"
|
|
.Ed
|
|
.Pp
|
|
where
|
|
.Sq subtype
|
|
indicates the specific file format and
|
|
.Sq version
|
|
its format version.
|
|
This header must be the first one of the file.
|
|
.Pp
|
|
The main purpose of the
|
|
.Sq Content-Type
|
|
header, aside from determining the format used in the file, is to allow
|
|
future changes to a given format.
|
|
Whenever an incompatible change is made, the version is bumped by one.
|
|
By keeping the header in the first line, future versions may even remove
|
|
the need for such a header -- e.g. if some format was replaced by XML
|
|
files, which have their own mandatory header.
|
|
.Pp
|
|
The rest of this document details the different format types.
|
|
.Ss Format: application/X-atf-atffile, version: 1
|
|
Atffiles are logically divided into three sections:
|
|
.Bl -bullet
|
|
.It
|
|
Test programs: the list of test programs that define the test suite
|
|
described by the Atffile.
|
|
.It
|
|
Meta-data properties: these define some constant values applicable to
|
|
all the test programs defined in the file.
|
|
In some sense they define the properties that describe the test suite.
|
|
.It
|
|
Configuration variables: defaults for configuration variables that
|
|
can be overridden through configuration files or the command line.
|
|
.El
|
|
.Pp
|
|
The grammar for Atffiles is the following:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
|
|
CONF ::= 'conf:' WORD EQUAL STRING
|
|
PROP ::= 'prop:' WORD EQUAL STRING
|
|
TP ::= TPFILE | TPGLOB
|
|
TPFILE ::= 'tp: ' STRING
|
|
TPGLOB ::= 'tp-glob: ' STRING
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPGLOBXX
|
|
.It CONF
|
|
Definition of a configuration variable.
|
|
.It PROP
|
|
Definition of a meta-data property.
|
|
.It TPFILE
|
|
Addition of a test program into the test suite.
|
|
The string is taken literally as the program's name, and this program
|
|
must exist.
|
|
.It TPGLOB
|
|
Addition of multiple test programs into the test suite.
|
|
The string is taken as a glob pattern, which may have or not have any
|
|
matches in the current directory.
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
prop: test-suite = utilities
|
|
|
|
conf: unprivileged-user = nobody
|
|
|
|
tp: t_cp
|
|
tp: t_mv
|
|
tp: t_df
|
|
tp-glob: t_dir_*
|
|
.Ed
|
|
.Ss Format: application/X-atf-config, version: 1
|
|
Configuration files are very simple: they only contain a list of variable
|
|
name/variable value pairs.
|
|
Their grammar is:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
|
|
VAR ::= WORD EQUAL STRING
|
|
COMMENT ::= HASH WORD*
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
# This is the system-wide configuration file for ATF.
|
|
# The above and this line are comments placed on their own line.
|
|
|
|
var1 = this is a variable value
|
|
var2 = this is another one # Optional comment at the end.
|
|
.Ed
|
|
.Ss Format: application/X-atf-tcs, version: 1
|
|
The
|
|
.Sq application/X-atf-tcs
|
|
format is used to describe the results of a collection of test cases;
|
|
in other words, it represents
|
|
.Em the output of a test program .
|
|
Unfortunately, it is not easy to control, from inside a test program, what
|
|
it prints to both its standard output and standard error streams.
|
|
This is specially the case of test programs written in the POSIX shell
|
|
language, because they are constantly executing external tools that may
|
|
print unexpected messages at all times.
|
|
Due to this, ATF imposes no restrictions on what a test program can send to
|
|
these two channels; in fact, they are encouraged to print as much useful
|
|
information as possible to aid in the debugging of test failures.
|
|
.Pp
|
|
Because we have no control over the two standard streams, the
|
|
.Sq application/X-atf-tcs
|
|
format describes the structure of a third stream, known as the
|
|
.Em results output ,
|
|
that test programs must generate.
|
|
(Note that test programs send, by default, the results output to the
|
|
standard output; use their
|
|
.Fl r
|
|
flag to change this whenever you need to parse the data.)
|
|
This stream is decoupled from the other two and has the following grammar:
|
|
.Bd -literal -offset indent
|
|
DATA ::= TCS-COUNT TC-STANZA* EOF
|
|
TCS-COUNT ::= 'tcs-count' COLON POSITIVE-NUMBER NEWLINE
|
|
TC-STANZA ::= TC-START TC-END
|
|
TC-START ::= 'tc-start' COLON STRING NEWLINE
|
|
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
|
|
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TCSXCOUNTXX
|
|
.It TCS-COUNT
|
|
Indicates the number of test cases that will be executed.
|
|
There will be this exact amount of
|
|
.Sq TC-STANZA
|
|
constructions following it.
|
|
.It TC-START
|
|
Indicates the beginning of a test case.
|
|
This is accompanied by the test case's name.
|
|
.It TC-END
|
|
Indicates the completion of a test case.
|
|
This is accompanied by the test case's name, its result and the reason
|
|
associated with this result (if applicable).
|
|
.El
|
|
.Pp
|
|
There are multiple reasons behind this design:
|
|
.Bl -bullet
|
|
.It
|
|
The reader of this format must be able to show real-time progress to the
|
|
user as the test cases are processed.
|
|
Therefore, the
|
|
.Sq TC-START
|
|
construction tells the reader
|
|
.Em when
|
|
a test case has started to process data.
|
|
.It
|
|
The reader of this format has to be able to provide useful statistics to
|
|
the user without having to wait for the end of the file.
|
|
Hence, the existence of the
|
|
.Sq TCS-COUNT
|
|
construction located at the beginning of the file.
|
|
.It
|
|
Text-based tools have to be able to easily look for the results of a given
|
|
test case.
|
|
This is why the
|
|
.Sq TC-END
|
|
construction duplicate the test case name already provided in
|
|
.Sq TC-START .
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
tcs-count: 2
|
|
tc-start: add
|
|
tc-end: add, passed
|
|
tc-start: subtract
|
|
tc-end: subtract, failed, Calculated an unexpected value
|
|
.Ed
|
|
.Pp
|
|
Going back to the standard output and standard error streams, the reader
|
|
has to be able to match the messages in those two streams to the test cases
|
|
they belong to.
|
|
To do this, these two streams must print a magic string that separates the
|
|
output of test cases from each other, which is enough to synchronize their
|
|
contents with the results output.
|
|
This string is
|
|
.Sq __atf_tc_separator__
|
|
and it must printed on a line of its own.
|
|
The last test case should not be followed by this line because the end of
|
|
file marker takes its role.
|
|
.Ss Format: application/X-atf-tps, version: 2
|
|
The
|
|
.Sq application/X-atf-tps
|
|
format multiplexes the standard output, standard error and results output
|
|
streams from multiple test programs into a single data file.
|
|
This format is used by
|
|
.Xr atf-run 1
|
|
to report the execution of several test programs and is later parsed by
|
|
.Xr atf-report 1
|
|
to inform the user of this process.
|
|
It has the following grammar:
|
|
.Bd -literal -offset indent
|
|
DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF
|
|
INFO ::= 'info' COLON STRING COMMA STRING NEWLINE
|
|
TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
|
|
TP-STANZA ::= TP-START TC-STANZA* TC-END
|
|
TP-START ::= 'tp-start' COLON STRING COMMA POSITIVE-NUMBER NEWLINE
|
|
TP-END ::= 'tc-end' COLON STRING (COMMA STRING)?
|
|
TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
|
|
TC-START ::= 'tc-start' COLON STRING NEWLINE
|
|
TC-SO ::= 'tc-so' COLON STRING NEWLINE
|
|
TC-SE ::= 'tc-se' COLON STRING NEWLINE
|
|
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
|
|
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPSXCOUNTXX
|
|
.It TPS-COUNT
|
|
Indicates the number of test programs that will be executed.
|
|
There will be this exact amount of
|
|
.Sq TP-STANZA
|
|
constructions following it.
|
|
.It TP-START
|
|
Indicates the beginning of a test program.
|
|
This includes the program's name and the amount of test cases that
|
|
will follow.
|
|
.It TP-END
|
|
Indicates the completion of a test program.
|
|
This is followed by the program's name and, if the program ended
|
|
prematurely, an error message indicating the reason of its failure.
|
|
A successful execution of a test program (regardless of the status of its
|
|
test cases) must not be accompanied by any reason.
|
|
.It TC-START
|
|
Indicates the beginning of a test case.
|
|
This is accompanied by the test case's name.
|
|
.It TC-SO
|
|
Contains a text line sent to the standard output stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-SE
|
|
Contains a text line sent to the standard error stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-END
|
|
Indicates the completion of a test case.
|
|
This is accompanied by the test case's name, its result and the reason
|
|
associated with this result (if applicable).
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
tps-count: 2
|
|
tp-start: calculator, 2
|
|
tc-start: add
|
|
tc-end: add, passed
|
|
tc-start: subtract
|
|
tc-so: 3-2 expected to return 1 but got 0
|
|
tc-end: subtract, failed, Calculated an unexpected value
|
|
tp-end: calculator
|
|
tp-start: files, 1
|
|
tc-start: copy
|
|
tc-se: could not find the cp(1) utility
|
|
tc-end: copy, skipped
|
|
tp-end: files
|
|
.Ed
|
|
.Sh SEE ALSO
|
|
.Xr atf 7
|