325 lines
11 KiB
Groff
325 lines
11 KiB
Groff
.\"
|
|
.\" Automated Testing Framework (atf)
|
|
.\"
|
|
.\" Copyright (c) 2007, 2008 The NetBSD Foundation, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.Dd November 4, 2007
|
|
.Dt ATF-FORMATS 5
|
|
.Os
|
|
.Sh NAME
|
|
.Nm atf-formats
|
|
.Nd machine-parseable data formats used by ATF
|
|
.Sh DESCRIPTION
|
|
This manual page describes the multiple data formats used in ATF.
|
|
These formats affect configuration files, control files and any data that
|
|
is externalized or internalized by the tools.
|
|
.Pp
|
|
Data files are always organized as follows:
|
|
.Bd -literal -offset indent
|
|
Header1: Value1 \\
|
|
... | head
|
|
HeaderN: ValueN /
|
|
mandatory blank line
|
|
Free-form text contents \\
|
|
... | body
|
|
... /
|
|
.Ed
|
|
.Pp
|
|
A file must always contain a
|
|
.Sq Content-Type
|
|
header and must always separate that header from the body with a blank
|
|
line, even if the body is empty.
|
|
.Pp
|
|
The
|
|
.Sq Content-Type
|
|
is always of the form:
|
|
.Bd -literal -offset indent
|
|
Content-Type: application/X-atf-<subtype>; version="<version>"
|
|
.Ed
|
|
.Pp
|
|
where
|
|
.Sq subtype
|
|
indicates the specific file format and
|
|
.Sq version
|
|
its format version.
|
|
This header must be the first one of the file.
|
|
.Pp
|
|
The main purpose of the
|
|
.Sq Content-Type
|
|
header, aside from determining the format used in the file, is to allow
|
|
future changes to a given format.
|
|
Whenever an incompatible change is made, the version is bumped by one.
|
|
By keeping the header in the first line, future versions may even remove
|
|
the need for such a header -- e.g. if some format was replaced by XML
|
|
files, which have their own mandatory header.
|
|
.Pp
|
|
The rest of this document details the different format types.
|
|
.Ss Format: application/X-atf-atffile, version: 1
|
|
Atffiles are logically divided into three sections:
|
|
.Bl -bullet
|
|
.It
|
|
Test programs: the list of test programs that define the test suite
|
|
described by the Atffile.
|
|
.It
|
|
Meta-data properties: these define some constant values applicable to
|
|
all the test programs defined in the file.
|
|
In some sense they define the properties that describe the test suite.
|
|
.It
|
|
Configuration variables: defaults for configuration variables that
|
|
can be overridden through configuration files or the command line.
|
|
.El
|
|
.Pp
|
|
The grammar for Atffiles is the following:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
|
|
CONF ::= 'conf:' WORD EQUAL STRING
|
|
PROP ::= 'prop:' WORD EQUAL STRING
|
|
TP ::= TPFILE | TPGLOB
|
|
TPFILE ::= 'tp: ' STRING
|
|
TPGLOB ::= 'tp-glob: ' STRING
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPGLOBXX
|
|
.It CONF
|
|
Definition of a configuration variable.
|
|
.It PROP
|
|
Definition of a meta-data property.
|
|
.It TPFILE
|
|
Addition of a test program into the test suite.
|
|
The string is taken literally as the program's name, and this program
|
|
must exist.
|
|
.It TPGLOB
|
|
Addition of multiple test programs into the test suite.
|
|
The string is taken as a glob pattern, which may have or not have any
|
|
matches in the current directory.
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
prop: test-suite = utilities
|
|
|
|
conf: unprivileged-user = nobody
|
|
|
|
tp: t_cp
|
|
tp: t_mv
|
|
tp: t_df
|
|
tp-glob: t_dir_*
|
|
.Ed
|
|
.Ss Format: application/X-atf-config, version: 1
|
|
Configuration files are very simple: they only contain a list of variable
|
|
name/variable value pairs.
|
|
Their grammar is:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
|
|
VAR ::= WORD EQUAL STRING
|
|
COMMENT ::= HASH WORD*
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
# This is the system-wide configuration file for ATF.
|
|
# The above and this line are comments placed on their own line.
|
|
|
|
var1 = this is a variable value
|
|
var2 = this is another one # Optional comment at the end.
|
|
.Ed
|
|
.Ss Format: application/X-atf-tcs, version: 1
|
|
The
|
|
.Sq application/X-atf-tcs
|
|
format is used to describe the results of a collection of test cases;
|
|
in other words, it represents
|
|
.Em the output of a test program .
|
|
Unfortunately, it is not easy to control, from inside a test program, what
|
|
it prints to both its standard output and standard error streams.
|
|
This is specially the case of test programs written in the POSIX shell
|
|
language, because they are constantly executing external tools that may
|
|
print unexpected messages at all times.
|
|
Due to this, ATF imposes no restrictions on what a test program can send to
|
|
these two channels; in fact, they are encouraged to print as much useful
|
|
information as possible to aid in the debugging of test failures.
|
|
.Pp
|
|
Because we have no control over the two standard streams, the
|
|
.Sq application/X-atf-tcs
|
|
format describes the structure of a third stream, known as the
|
|
.Em results output ,
|
|
that test programs must generate.
|
|
(Note that test programs send, by default, the results output to the
|
|
standard output; use their
|
|
.Fl r
|
|
flag to change this whenever you need to parse the data.)
|
|
This stream is decoupled from the other two and has the following grammar:
|
|
.Bd -literal -offset indent
|
|
DATA ::= TCS-COUNT TC-STANZA* EOF
|
|
TCS-COUNT ::= 'tcs-count' COLON POSITIVE-NUMBER NEWLINE
|
|
TC-STANZA ::= TC-START TC-END
|
|
TC-START ::= 'tc-start' COLON STRING NEWLINE
|
|
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
|
|
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TCSXCOUNTXX
|
|
.It TCS-COUNT
|
|
Indicates the number of test cases that will be executed.
|
|
There will be this exact amount of
|
|
.Sq TC-STANZA
|
|
constructions following it.
|
|
.It TC-START
|
|
Indicates the beginning of a test case.
|
|
This is accompanied by the test case's name.
|
|
.It TC-END
|
|
Indicates the completion of a test case.
|
|
This is accompanied by the test case's name, its result and the reason
|
|
associated with this result (if applicable).
|
|
.El
|
|
.Pp
|
|
There are multiple reasons behind this design:
|
|
.Bl -bullet
|
|
.It
|
|
The reader of this format must be able to show real-time progress to the
|
|
user as the test cases are processed.
|
|
Therefore, the
|
|
.Sq TC-START
|
|
construction tells the reader
|
|
.Em when
|
|
a test case has started to process data.
|
|
.It
|
|
The reader of this format has to be able to provide useful statistics to
|
|
the user without having to wait for the end of the file.
|
|
Hence, the existence of the
|
|
.Sq TCS-COUNT
|
|
construction located at the beginning of the file.
|
|
.It
|
|
Text-based tools have to be able to easily look for the results of a given
|
|
test case.
|
|
This is why the
|
|
.Sq TC-END
|
|
construction duplicate the test case name already provided in
|
|
.Sq TC-START .
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
tcs-count: 2
|
|
tc-start: add
|
|
tc-end: add, passed
|
|
tc-start: subtract
|
|
tc-end: subtract, failed, Calculated an unexpected value
|
|
.Ed
|
|
.Pp
|
|
Going back to the standard output and standard error streams, the reader
|
|
has to be able to match the messages in those two streams to the test cases
|
|
they belong to.
|
|
To do this, these two streams must print a magic string that separates the
|
|
output of test cases from each other, which is enough to synchronize their
|
|
contents with the results output.
|
|
This string is
|
|
.Sq __atf_tc_separator__
|
|
and it must printed on a line of its own.
|
|
The last test case should not be followed by this line because the end of
|
|
file marker takes its role.
|
|
.Ss Format: application/X-atf-tps, version: 2
|
|
The
|
|
.Sq application/X-atf-tps
|
|
format multiplexes the standard output, standard error and results output
|
|
streams from multiple test programs into a single data file.
|
|
This format is used by
|
|
.Xr atf-run 1
|
|
to report the execution of several test programs and is later parsed by
|
|
.Xr atf-report 1
|
|
to inform the user of this process.
|
|
It has the following grammar:
|
|
.Bd -literal -offset indent
|
|
DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF
|
|
INFO ::= 'info' COLON STRING COMMA STRING NEWLINE
|
|
TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
|
|
TP-STANZA ::= TP-START TC-STANZA* TC-END
|
|
TP-START ::= 'tp-start' COLON STRING COMMA POSITIVE-NUMBER NEWLINE
|
|
TP-END ::= 'tc-end' COLON STRING (COMMA STRING)?
|
|
TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
|
|
TC-START ::= 'tc-start' COLON STRING NEWLINE
|
|
TC-SO ::= 'tc-so' COLON STRING NEWLINE
|
|
TC-SE ::= 'tc-se' COLON STRING NEWLINE
|
|
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
|
|
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPSXCOUNTXX
|
|
.It TPS-COUNT
|
|
Indicates the number of test programs that will be executed.
|
|
There will be this exact amount of
|
|
.Sq TP-STANZA
|
|
constructions following it.
|
|
.It TP-START
|
|
Indicates the beginning of a test program.
|
|
This includes the program's name and the amount of test cases that
|
|
will follow.
|
|
.It TP-END
|
|
Indicates the completion of a test program.
|
|
This is followed by the program's name and, if the program ended
|
|
prematurely, an error message indicating the reason of its failure.
|
|
A successful execution of a test program (regardless of the status of its
|
|
test cases) must not be accompanied by any reason.
|
|
.It TC-START
|
|
Indicates the beginning of a test case.
|
|
This is accompanied by the test case's name.
|
|
.It TC-SO
|
|
Contains a text line sent to the standard output stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-SE
|
|
Contains a text line sent to the standard error stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-END
|
|
Indicates the completion of a test case.
|
|
This is accompanied by the test case's name, its result and the reason
|
|
associated with this result (if applicable).
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
tps-count: 2
|
|
tp-start: calculator, 2
|
|
tc-start: add
|
|
tc-end: add, passed
|
|
tc-start: subtract
|
|
tc-so: 3-2 expected to return 1 but got 0
|
|
tc-end: subtract, failed, Calculated an unexpected value
|
|
tp-end: calculator
|
|
tp-start: files, 1
|
|
tc-start: copy
|
|
tc-se: could not find the cp(1) utility
|
|
tc-end: copy, skipped
|
|
tp-end: files
|
|
.Ed
|
|
.Sh SEE ALSO
|
|
.Xr atf 7
|