Doesn't seem to be much point in keeping this README up to date anymore,
since it's completely redundant with regress.sgml. I think we agreed to remove it awhile back, actually, but no one got around to doing it.
This commit is contained in:
Tom Lane
parent
af3487e9bf
commit
2f4759d5ce
@ -1,270 +0,0 @@
|
|||||||
REGRESSION TESTS
|
|
||||||
|
|
||||||
The regression tests are a comprehensive set of tests for the SQL
|
|
||||||
implementation in PostgreSQL. They test standard SQL operations as
|
|
||||||
well as the extended capabilities of PostgreSQL. The test suite was
|
|
||||||
originally developed by Jolly Chen and Andrew Yu, and was extensively
|
|
||||||
revised and repackaged by Marc Fournier and Thomas Lockhart. From
|
|
||||||
PostgreSQL 6.1 onward the regression tests are current for every
|
|
||||||
official release.
|
|
||||||
|
|
||||||
The regression test can be run against an already installed and
|
|
||||||
running server, or using a temporary installation within the build
|
|
||||||
tree. Furthermore, there is a "parallel" and a "sequential" mode for
|
|
||||||
running the tests. The sequential method runs each test script in
|
|
||||||
turn, whereas the parallel method starts up multiple server processes
|
|
||||||
to run groups of tests in parallel. Parallel testing gives confidence
|
|
||||||
that interprocess communication and locking are working correctly. For
|
|
||||||
historical reasons, the sequential test is usually run against an
|
|
||||||
existing installation and the parallel method "stand-alone", but there
|
|
||||||
are technical reasons for this.
|
|
||||||
|
|
||||||
To run the regression tests after building but before installation,
|
|
||||||
type
|
|
||||||
|
|
||||||
$ gmake check
|
|
||||||
|
|
||||||
in the top-level directory. (Or you can change to src/test/regress and
|
|
||||||
run the command there.) This will first build several auxiliary files,
|
|
||||||
such as platform-dependent "expected" files and some sample
|
|
||||||
user-defined trigger functions, and then run the test driver
|
|
||||||
script. At the end you should see something like
|
|
||||||
|
|
||||||
======================
|
|
||||||
All 76 tests passed.
|
|
||||||
======================
|
|
||||||
|
|
||||||
or otherwise a note about what tests failed. See the section called
|
|
||||||
Test Evaluation below for more.
|
|
||||||
|
|
||||||
Note: Because this test method runs a temporary server, it will
|
|
||||||
not work when you are the root user (the server will not start as
|
|
||||||
root). If you already did the build as root, you do not have to
|
|
||||||
start all over. Instead, make the regression test directory
|
|
||||||
writable by some other user, log in as that user, and restart the
|
|
||||||
tests. For example,
|
|
||||||
|
|
||||||
root# chmod -R a+w src/test/regress
|
|
||||||
root# su - joeuser
|
|
||||||
joeuser$ gmake check
|
|
||||||
|
|
||||||
(The only possible "security risk" here is that other users might
|
|
||||||
be able to alter the regression test results behind your back. Use
|
|
||||||
common sense when managing user permissions.)
|
|
||||||
|
|
||||||
Alternatively, run the tests after installation.
|
|
||||||
|
|
||||||
Tip: On some systems, the default Bourne-compatible shell
|
|
||||||
(/bin/sh) gets confused when it has to manage too many child
|
|
||||||
processes in parallel. This may cause the parallel test run to
|
|
||||||
lock up or fail. In such cases, specify a different
|
|
||||||
Bourne-compatible shell on the command line, for example:
|
|
||||||
|
|
||||||
$ gmake SHELL=/bin/ksh check
|
|
||||||
|
|
||||||
To run the tests after installation, initialize a data area and start
|
|
||||||
the server, then type
|
|
||||||
|
|
||||||
$ gmake installcheck
|
|
||||||
|
|
||||||
The tests will expect to contact the server at the local host and the
|
|
||||||
default port number, unless directed otherwise by PGHOST and PGPORT
|
|
||||||
environment variables.
|
|
||||||
|
|
||||||
Test Evaluation
|
|
||||||
|
|
||||||
Some properly installed and fully functional PostgreSQL installations
|
|
||||||
can "fail" some of these regression tests due to platform-specific
|
|
||||||
artifacts such as varying floating point representation and time zone
|
|
||||||
support. The tests are currently evaluated using a simple diff
|
|
||||||
comparison against the outputs generated on a reference system, so the
|
|
||||||
results are sensitive to small system differences. When a test is
|
|
||||||
reported as "failed", always examine the differences between expected
|
|
||||||
and actual results; you may well find that the differences are not
|
|
||||||
significant. Nonetheless, we still strive to maintain accurate reference
|
|
||||||
files across all supported platforms, so it can be expected that all
|
|
||||||
tests pass.
|
|
||||||
|
|
||||||
The actual outputs of the regression tests are in files in the
|
|
||||||
src/test/regress/results directory. The test script uses diff to
|
|
||||||
compare each output file against the reference outputs stored in the
|
|
||||||
src/test/regress/expected directory. Any differences are saved for
|
|
||||||
your inspection in src/test/regress/regression.diffs. (Or you can run
|
|
||||||
diff yourself, if you prefer.)
|
|
||||||
|
|
||||||
Error message differences
|
|
||||||
|
|
||||||
Some of the regression tests involve intentional invalid input
|
|
||||||
values. Error messages can come from either the PostgreSQL code or
|
|
||||||
from the host platform system routines. In the latter case, the
|
|
||||||
messages may vary between platforms, but should reflect similar
|
|
||||||
information. These differences in messages will result in a "failed"
|
|
||||||
regression test which can be validated by inspection.
|
|
||||||
|
|
||||||
Locale differences
|
|
||||||
|
|
||||||
The tests expect to run in plain "C" locale. This should not cause any
|
|
||||||
problems when you run the tests against a temporary installation, since
|
|
||||||
the regression test driver takes care to start the server in C locale.
|
|
||||||
However, if you run the tests against an already-installed server that
|
|
||||||
is using non-C locale settings, you may see differences caused by
|
|
||||||
varying rules for string sort order, formatting of numeric and monetary
|
|
||||||
values, and so forth.
|
|
||||||
|
|
||||||
In some locales the resulting differences are small and easily checked by
|
|
||||||
inspection. However, in a locale that changes the rules for formatting
|
|
||||||
of numeric values (typically by swapping the usage of commas and
|
|
||||||
decimal points), entry of some data values will fail, resulting in
|
|
||||||
extensive differences later in the tests where the missing data values
|
|
||||||
are supposed to be used.
|
|
||||||
|
|
||||||
Date and time differences
|
|
||||||
|
|
||||||
Most of the date and time results are dependent on the time zone
|
|
||||||
environment. The reference files are generated for time zone PST8PDT
|
|
||||||
(Berkeley, California) and there will be apparent failures if the
|
|
||||||
tests are not run with that time zone setting. The regression test
|
|
||||||
driver sets environment variable PGTZ to PST8PDT to ensure proper
|
|
||||||
results. However, your system must provide library support for the
|
|
||||||
PST8PDT time zone, or the time zone-dependent tests will fail. To
|
|
||||||
verify that your machine does have this support, type the following:
|
|
||||||
|
|
||||||
$ env TZ=PST8PDT date
|
|
||||||
|
|
||||||
The command above should have returned the current system time in the
|
|
||||||
PST8PDT time zone. If the PST8PDT database is not available, then your
|
|
||||||
system may have returned the time in GMT. If the PST8PDT time zone is
|
|
||||||
not available, you can set the time zone rules explicitly:
|
|
||||||
|
|
||||||
PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
|
|
||||||
|
|
||||||
There appear to be some systems which do not accept the recommended
|
|
||||||
syntax for explicitly setting the local time zone rules; you may need
|
|
||||||
to use a different PGTZ setting on such machines.
|
|
||||||
|
|
||||||
Some systems using older time zone libraries fail to apply
|
|
||||||
daylight-savings corrections to dates before 1970, causing pre-1970
|
|
||||||
PDT times to be displayed in PST instead. This will result in
|
|
||||||
localized differences in the test results.
|
|
||||||
|
|
||||||
Some of the queries in the "timestamp" test will fail if you run the
|
|
||||||
test on the day of a daylight-savings time changeover, or the day
|
|
||||||
before or after one. These queries assume that the intervals between
|
|
||||||
midnight yesterday, midnight today and midnight tomorrow are exactly
|
|
||||||
twenty-four hours -- which is wrong if daylight-savings time went into
|
|
||||||
or out of effect meanwhile.
|
|
||||||
|
|
||||||
Floating point differences
|
|
||||||
|
|
||||||
Some of the tests involve computing 64-bit (double precision) numbers
|
|
||||||
from table columns. Differences in results involving mathematical
|
|
||||||
functions of double precision columns have been observed. The float8
|
|
||||||
and geometry tests are particularly prone to small differences across
|
|
||||||
platforms, or even with different compiler optimization options. Human
|
|
||||||
eyeball comparison is needed to determine the real significance of
|
|
||||||
these differences which are usually 10 places to the right of the
|
|
||||||
decimal point.
|
|
||||||
|
|
||||||
Some systems signal errors from pow() and exp() differently from the
|
|
||||||
mechanism expected by the current PostgreSQL code.
|
|
||||||
|
|
||||||
Polygon differences
|
|
||||||
|
|
||||||
Several of the tests involve operations on geographic data about the
|
|
||||||
Oakland/Berkeley, CA street map. The map data is expressed as polygons
|
|
||||||
whose vertices are represented as pairs of double precision numbers
|
|
||||||
(decimal latitude and longitude). Initially, some tables are created
|
|
||||||
and loaded with geographic data, then some views are created which
|
|
||||||
join two tables using the polygon intersection operator (##), then a
|
|
||||||
select is done on the view.
|
|
||||||
|
|
||||||
When comparing the results from different platforms, differences occur
|
|
||||||
in the 2nd or 3rd place to the right of the decimal point. The SQL
|
|
||||||
statements where these problems occur are the following:
|
|
||||||
|
|
||||||
SELECT * from street;
|
|
||||||
SELECT * from iexit;
|
|
||||||
|
|
||||||
Tuple ordering differences
|
|
||||||
|
|
||||||
You might see differences in which the same tuples are output in a
|
|
||||||
different order than what appears in the expected file. In most cases
|
|
||||||
this is not, strictly speaking, a bug. Most of the regression test
|
|
||||||
scripts are not so pedantic as to use an ORDER BY for every single
|
|
||||||
SELECT, and so their result tuple orderings are not well-defined
|
|
||||||
according to the letter of the SQL spec. In practice, since we are
|
|
||||||
looking at the same queries being executed on the same data by the same
|
|
||||||
software, we usually get the same result ordering on all platforms, and
|
|
||||||
so the lack of ORDER BY isn't a problem. Some queries do exhibit
|
|
||||||
cross-platform ordering differences, however. (Ordering differences
|
|
||||||
can also be triggered by non-C locale settings.)
|
|
||||||
|
|
||||||
Therefore, if you see an ordering difference, it's not something to
|
|
||||||
worry about, unless the query does have an ORDER BY that your result
|
|
||||||
is violating. But please report it anyway, so that we can add an
|
|
||||||
ORDER BY to that particular query and thereby eliminate the bogus
|
|
||||||
"failure" in future releases.
|
|
||||||
|
|
||||||
You might wonder why we don't ORDER all the regress test SELECTs to
|
|
||||||
get rid of this issue once and for all. The reason is that that would
|
|
||||||
make the regression tests less useful, not more, since they'd tend
|
|
||||||
to exercise query plan types that produce ordered results to the
|
|
||||||
exclusion of those that don't.
|
|
||||||
|
|
||||||
The "random" test
|
|
||||||
|
|
||||||
There is at least one case in the "random" test script that is
|
|
||||||
intended to produce random results. This causes random to fail the
|
|
||||||
regression test once in a while (perhaps once in every five to ten
|
|
||||||
trials). Typing
|
|
||||||
|
|
||||||
diff results/random.out expected/random.out
|
|
||||||
|
|
||||||
should produce only one or a few lines of differences. You need not
|
|
||||||
worry unless the random test always fails in repeated attempts. (On
|
|
||||||
the other hand, if the random test is never reported to fail even in
|
|
||||||
many trials of the regress tests, you probably should worry.)
|
|
||||||
|
|
||||||
Platform-specific comparison files
|
|
||||||
|
|
||||||
Since some of the tests inherently produce platform-specific results,
|
|
||||||
we have provided a way to supply platform-specific result comparison
|
|
||||||
files. Frequently, the same variation applies to multiple platforms;
|
|
||||||
rather than supplying a separate comparison file for every platform,
|
|
||||||
there is a mapping file that defines which comparison file to use. So,
|
|
||||||
to eliminate bogus test "failures" for a particular platform, you must
|
|
||||||
choose or make a variant result file, and then add a line to the
|
|
||||||
mapping file, which is "resultmap".
|
|
||||||
|
|
||||||
Each line in the mapping file is of the form
|
|
||||||
|
|
||||||
testname/platformpattern=comparisonfilename
|
|
||||||
|
|
||||||
The test name is just the name of the particular regression test
|
|
||||||
module. The platform pattern is a pattern in the style of expr(1)
|
|
||||||
(that is, a regular expression with an implicit ^ anchor at the start).
|
|
||||||
It is matched against the platform name as printed by config.guess
|
|
||||||
followed by ":gcc" or ":cc", depending on whether you use the GNU compiler
|
|
||||||
or the system's native compiler (on systems where there is a difference).
|
|
||||||
The comparison file name is the name of the substitute result comparison
|
|
||||||
file.
|
|
||||||
|
|
||||||
For example: the int2 regress test includes a deliberate entry of a
|
|
||||||
value that is too large to fit in int2. The specific error message
|
|
||||||
that is produced is platform-dependent; our reference platform emits
|
|
||||||
|
|
||||||
ERROR: pg_atoi: error reading "100000": Numerical result out of range
|
|
||||||
|
|
||||||
but a fair number of other Unix platforms emit
|
|
||||||
|
|
||||||
ERROR: pg_atoi: error reading "100000": Result too large
|
|
||||||
|
|
||||||
Therefore, we provide a variant comparison file, int2-too-large.out,
|
|
||||||
that includes this spelling of the error message. To silence the bogus
|
|
||||||
"failure" message on HPPA platforms, resultmap includes
|
|
||||||
|
|
||||||
int2/hppa=int2-too-large
|
|
||||||
|
|
||||||
which will trigger on any machine for which config.guess's output
|
|
||||||
begins with 'hppa'. Other lines in resultmap select the variant
|
|
||||||
comparison file for other platforms where it's appropriate.
|
|
||||||
Loading…
x
Reference in New Issue
Block a user