Add doc/testrunner.md, for documenting the testrunner.tcl script.
FossilOrigin-Name: 9c69a28401c7273823f2c2b291fd417febeb278afb9ce085a4b944505ca13d23
This commit is contained in:
dan
parent
eef599dbd4
commit
36018803d6
284
doc/testrunner.md
Normal file
284
doc/testrunner.md
Normal file
@ -0,0 +1,284 @@
|
||||
|
||||
|
||||
# The testrunner.tcl Script
|
||||
|
||||
# 1. Overview
|
||||
|
||||
testrunner.tcl is a Tcl script used to run multiple SQLite tests using
|
||||
multiple jobs. It supports the following types of tests:
|
||||
|
||||
* Tcl test scripts.
|
||||
|
||||
* Tests run with [make] commands. Specifically, at time of writing,
|
||||
[make fuzztest], [make mptest], [make sourcetest] and [make threadtest].
|
||||
|
||||
testrunner.tcl pipes the output of all tests and builds run into log file
|
||||
**testrunner.log**, created in the cwd directory. Searching this file for
|
||||
"failed" is a good way to find the output of a failed test.
|
||||
|
||||
testrunner.tcl also populates SQLite database **testrunner.db**. This database
|
||||
contains details of all tests run, running and to be run. A useful query
|
||||
might be:
|
||||
|
||||
```
|
||||
SELECT * FROM script WHERE state='failed'
|
||||
```
|
||||
|
||||
Running the command:
|
||||
|
||||
```
|
||||
./testfixture $(TESTDIR)/testrunner.tcl status
|
||||
```
|
||||
|
||||
in the directory containing the testrunner.db database runs various queries
|
||||
to produce a succinct report on the state of a running testrunner.tcl script.
|
||||
Running:
|
||||
|
||||
```
|
||||
watch ./testfixture $(TESTDIR)/testrunner.tcl status
|
||||
```
|
||||
|
||||
in another terminal is a good way to keep an eye on a long running test.
|
||||
|
||||
Sometimes testrunner.tcl uses the [testfixture] binary that it is run with
|
||||
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
|
||||
other binaries in specific configurations to test (see "Source Tests").
|
||||
|
||||
# 2. Binary Tests
|
||||
|
||||
The commands described in this section all run various combinations of the Tcl
|
||||
test scripts using the [testfixture] binary used to run the testrunner.tcl
|
||||
script (i.e. they do not invoke the compiler to build new binaries, or the
|
||||
[make] command to run tests that are not Tcl scripts). The procedure to run
|
||||
these tests is therefore:
|
||||
|
||||
1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
|
||||
whatever method seems convenient.
|
||||
|
||||
2. Test the binary built in step 1 by running testrunner.tcl with it,
|
||||
perhaps with various options.
|
||||
|
||||
The following sub-sections describe the various options that can be
|
||||
passed to testrunner.tcl to test binary testfixture builds.
|
||||
|
||||
## 2.1. Organization of Tcl Tests
|
||||
|
||||
Tcl tests are stored in files that match the pattern *\*.test*. They are
|
||||
found in both the $TOP/test/ directory, and in the various sub-directories
|
||||
of the $TOP/ext/ directory of the source tree. Not all *\*.test* files
|
||||
contain Tcl tests - a handful are Tcl scripts designed to invoke other
|
||||
*\*.test* files.
|
||||
|
||||
The **veryquick** set of tests is a subset of all Tcl test scripts in the
|
||||
source tree. In includes most tests, but excludes some that are very slow.
|
||||
Almost all fault-injection tests (those that test the response of the library
|
||||
to OOM or IO errors) are excluded. It is defined in source file
|
||||
*test/permutations.test*.
|
||||
|
||||
The **full** set of tests includes all Tcl test scripts in the source tree.
|
||||
To run a "full" test is to run all Tcl test scripts that can be found in the
|
||||
source tree.
|
||||
|
||||
File *permutations.test* defines various test "permutations". A permutation
|
||||
consists of:
|
||||
|
||||
* A subset of Tcl test scripts, and
|
||||
|
||||
* Runtime configuration to apply before running each test script
|
||||
(e.g. enabling auto-vacuum, or disable lookaside).
|
||||
|
||||
Running **all** tests is to run all tests in the full test set, plus a dozen
|
||||
or so permutations. The specific permutations that are run as part of "all"
|
||||
are defined in file *testrunner_data.tcl*.
|
||||
|
||||
## 2.2. Commands to Run Tests
|
||||
|
||||
To run the "veryquick" test set, use either of the following:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl
|
||||
./testfixture $TESTDIR/testrunner.tcl veryquick
|
||||
```
|
||||
|
||||
To run the "full" test suite:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl full
|
||||
```
|
||||
|
||||
To run the subset of the "full" test suite for which the test file name matches
|
||||
a specified pattern (e.g. all tests that start with "fts5"), either of:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl fts5%
|
||||
./testfixture $TESTDIR/testrunner.tcl 'fts5*'
|
||||
```
|
||||
|
||||
To run "all" tests (full + permutations):
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl all
|
||||
```
|
||||
|
||||
<a name=binary_test_failures></a>
|
||||
## 2.3. Investigating Binary Test Failures
|
||||
|
||||
If a test fails, testrunner.tcl reports name of the Tcl test script and, if
|
||||
applicable, the name of the permutation, to stdout. This information can also
|
||||
be retrieved from either *testrunner.log* or *testrunner.db*.
|
||||
|
||||
If there is no permutation, the individual test script may be run with:
|
||||
|
||||
```
|
||||
./testfixture $PATH_TO_SCRIPT
|
||||
```
|
||||
|
||||
Or, if the failure occured as part of a permutation:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
|
||||
```
|
||||
|
||||
TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
|
||||
|
||||
# 3. Source Code Tests
|
||||
|
||||
The commands described in this section invoke the C compiler to build
|
||||
binaries from the source tree, then use those binaries to run Tcl and
|
||||
other tests. The advantages of this are that:
|
||||
|
||||
* it is possible to test multiple build configurations with a single
|
||||
command, and
|
||||
|
||||
* it ensures that tests are always run using binaries created with the
|
||||
same set of compiler options.
|
||||
|
||||
The testrunner.tcl commands described in this section may be run using
|
||||
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
|
||||
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
|
||||
|
||||
TODO: ./configure + Makefile.msc build systems.
|
||||
|
||||
## Commands to Run SQLite Tests
|
||||
|
||||
The **mdevtest** command is equivalent to running the veryquick tests and
|
||||
the [make fuzztest] target once for each of two --enable-all builds - one
|
||||
with debugging enabled and one without:
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl mdevtest
|
||||
```
|
||||
|
||||
In other words, it is equivalent to running:
|
||||
|
||||
```
|
||||
$TOP/configure --enable-all --enable-debug
|
||||
make fuzztest
|
||||
make testfixture
|
||||
./testfixture $TOP/test/testrunner.tcl veryquick
|
||||
|
||||
# Then, after removing files created by the tests above:
|
||||
$TOP/configure --enable-all OPTS="-O0"
|
||||
make fuzztest
|
||||
make testfixture
|
||||
./testfixture $TOP/test/testrunner.tcl veryquick
|
||||
```
|
||||
|
||||
The **sdevtest** command is identical to the mdevtest command, except that the
|
||||
second of the two builds is a sanitizer build. Specifically, this means that
|
||||
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl sdevtest
|
||||
```
|
||||
|
||||
The **release** command runs lots of tests under lots of builds. It runs
|
||||
different combinations of builds and tests depending on whether it is run
|
||||
on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
|
||||
of the specific tests run.
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl release
|
||||
```
|
||||
|
||||
## Running ZipVFS Tests
|
||||
|
||||
testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
|
||||
tests from the Zipvfs project with the following command:
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
|
||||
```
|
||||
|
||||
This can be combined with any of "mdevtest", "sdevtest" or "release" to
|
||||
test both SQLite and Zipvfs with a single command:
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
|
||||
```
|
||||
|
||||
## Investigating Source Code Test Failures
|
||||
|
||||
Investigating a test failure that occurs during source code testing is a
|
||||
two step process:
|
||||
|
||||
1. Recreating the build configuration in which the test failed, and
|
||||
|
||||
2. Re-running the actual test.
|
||||
|
||||
To recreate a build configuration, use the testrunner.tcl **script** command
|
||||
to create a build script. A build script is a bash script on Linux or OSX, or
|
||||
a dos \*.bat file on windows. For example:
|
||||
|
||||
```
|
||||
# Create a script that recreates build configuration "Device-One" on
|
||||
# Linux or OSX:
|
||||
tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh
|
||||
|
||||
# Create a script that recreates build configuration "Have-Not" on Windows:
|
||||
tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat
|
||||
```
|
||||
|
||||
The generated bash or \*.bat file script accepts a single argument - a makefile
|
||||
target to build. This may be used either to run a [make] command test directly,
|
||||
or else to build a testfixture (or testfixture.exe) binary with which to
|
||||
run a Tcl test script, as <a href=#binary_test_failures>described above</a>.
|
||||
|
||||
|
||||
|
||||
# 4. Controlling CPU Core Utilization
|
||||
|
||||
When running either binary or source code tests, testrunner.tcl reports the
|
||||
number of jobs it intends to use to stdout. e.g.
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl
|
||||
splitting work across 16 jobs
|
||||
... more output ...
|
||||
```
|
||||
|
||||
By default, testfixture.tcl attempts to set the number of jobs to the number
|
||||
of real cores on the machine. This can be overridden using the "--jobs" (or -j)
|
||||
switch:
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
|
||||
splitting work across 8 jobs
|
||||
... more output ...
|
||||
```
|
||||
|
||||
The number of jobs may also be changed while an instance of testrunner.tcl is
|
||||
running by exucuting the following command from the directory containing the
|
||||
testrunner.log and testrunner.db files:
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
15
manifest
15
manifest
@ -1,5 +1,5 @@
|
||||
C More\scode\slegibility\sand\sstyle\simprovements\sin\sthe\sJNI\spieces.\sStart\swork\son\sa\sjavadoc\sbuild.
|
||||
D 2023-08-24T18:43:25.053
|
||||
C Add\sdoc/testrunner.md,\sfor\sdocumenting\sthe\stestrunner.tcl\sscript.
|
||||
D 2023-08-24T19:08:50.024
|
||||
F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
|
||||
F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
|
||||
F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724
|
||||
@ -41,6 +41,7 @@ F doc/compile-for-windows.md c52f2903f1cb11b2308798feecca2e44701b037b78f467a538a
|
||||
F doc/json-enhancements.md e356fc834781f1f1aa22ee300027a270b2c960122468499bf347bb123ce1ea4f
|
||||
F doc/lemon.html 44a53a1d2b42d7751f7b2f478efb23c978e258d794bfd172442307a755b9fa44
|
||||
F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
|
||||
F doc/testrunner.md 2434864be2219d4f0b6ffc99d0a2172d531c4ca4345340776f67ad4edd90dc90
|
||||
F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
|
||||
F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
|
||||
F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a
|
||||
@ -1606,7 +1607,7 @@ F test/temptable2.test 76821347810ecc88203e6ef0dd6897b6036ac788e9dd3e6b04fd4d163
|
||||
F test/temptable3.test d11a0974e52b347e45ee54ef1923c91ed91e4637
|
||||
F test/temptrigger.test 38f0ca479b1822d3117069e014daabcaacefffcc
|
||||
F test/tester.tcl 68454ef88508c196d19e8694daa27bff7107a91857799eaa12f417188ae53ede
|
||||
F test/testrunner.tcl a9fee4df57276bc9e446961b160068c269da5902cc8ffc3e8852d77626b7594c
|
||||
F test/testrunner.tcl ccdfda84732cf8665bd8d3bfee79b80841e221459e5d00a632a3a5c758966e1f
|
||||
F test/testrunner_data.tcl c448693eb6fdbadb78cb26f6253d4f335666f9836f988afa575de960b666b19f
|
||||
F test/thread001.test a0985c117eab62c0c65526e9fa5d1360dd1cac5b03bde223902763274ce21899
|
||||
F test/thread002.test c24c83408e35ba5a952a3638b7ac03ccdf1ce4409289c54a050ac4c5f1de7502
|
||||
@ -2095,8 +2096,8 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
|
||||
F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
|
||||
F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
|
||||
F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
|
||||
P cf185bcd25629d882a030b8b87048179a120ab1f84aa1d68b279c499dbdf0dba
|
||||
R 409ebb8571a40aa1108ef1efd5f91432
|
||||
U stephan
|
||||
Z 85677113ed9046ae48bf181600ab6418
|
||||
P 62b404d62fd62f4d220838b59c9f38a71afa2d4a8c3af0a5c9495fa7020972cf
|
||||
R a33be9d18a7bfec02bf745dca74a3ae4
|
||||
U dan
|
||||
Z b231c489ac6e506e7affd58c17184d19
|
||||
# Remove this line to create a well-formed Fossil manifest.
|
||||
|
||||
@ -1 +1 @@
|
||||
62b404d62fd62f4d220838b59c9f38a71afa2d4a8c3af0a5c9495fa7020972cf
|
||||
9c69a28401c7273823f2c2b291fd417febeb278afb9ce085a4b944505ca13d23
|
||||
@ -651,11 +651,14 @@ proc r_get_next_job {iJob} {
|
||||
proc make_new_testset {} {
|
||||
global TRG
|
||||
|
||||
set tests [testset_patternlist $TRG(patternlist)]
|
||||
|
||||
set tests [list]
|
||||
if {$TRG(zipvfs)!=""} {
|
||||
source [file join $TRG(zipvfs) test zipvfs_testrunner.tcl]
|
||||
set tests [concat $tests [zipvfs_testrunner_testset]]
|
||||
lappend tests {*}[zipvfs_testrunner_testset]
|
||||
}
|
||||
|
||||
if {$tests=="" || $TRG(patternlist)!=""} {
|
||||
lappend tests {*}[testset_patternlist $TRG(patternlist)]
|
||||
}
|
||||
|
||||
r_write_db {
|
||||
@ -960,7 +963,7 @@ sqlite3 trdb $TRG(dbname)
|
||||
trdb timeout $TRG(timeout)
|
||||
set tm [lindex [time { make_new_testset }] 0]
|
||||
if {$TRG(nJob)>1} {
|
||||
puts "splitting work across $TRG(nJob) cores"
|
||||
puts "splitting work across $TRG(nJob) jobs"
|
||||
}
|
||||
puts "built testset in [expr $tm/1000]ms.."
|
||||
run_testset
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user