mirrors/qemu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: reorganize testing.rst

Browse Source 
Clean up the heading levels to use === --- ~~~ ^^^ '''.  Reorganize the
outline for the Avocado part, and always include headings for the
class names.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This commit is contained in:
Paolo Bonzini 2021-09-07 17:21:58 +02:00
parent 9fce360176
commit 16e79e1b01
1 changed files with 76 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
docs/devel/testing.rst
View File

@ -1,11 +1,10 @@
===============
Testing in QEMU
===============
This document describes the testing infrastructure in QEMU.
Testing with "make check"
=========================
-------------------------
The "make check" testing family includes most of the C based tests in QEMU. For
a quick help, run ``make check-help`` from the source tree.
@ -24,7 +23,7 @@ expect the executables to exist and will fail with obscure messages if they
cannot find them.
Unit tests
----------
~~~~~~~~~~
Unit tests, which can be invoked with ``make check-unit``, are simple C tests
that typically link to individual QEMU object files and exercise them by
@ -67,7 +66,7 @@ and copy the actual command line which executes the unit test, then run
it from the command line.
QTest
-----
~~~~~
QTest is a device emulation testing framework. It can be very useful to test
device models; it could also control certain aspects of QEMU (such as virtual
@ -81,7 +80,7 @@ QTest cases can be executed with
make check-qtest
QAPI schema tests
-----------------
~~~~~~~~~~~~~~~~~
The QAPI schema tests validate the QAPI parser used by QMP, by feeding
predefined input to the parser and comparing the result with the reference
@ -108,14 +107,14 @@ parser (either fixing a bug or extending/modifying the syntax). To do this:
``qapi-schema += foo.json``
check-block
-----------
~~~~~~~~~~~
``make check-block`` runs a subset of the block layer iotests (the tests that
are in the "auto" group).
See the "QEMU iotests" section below for more information.
QEMU iotests
============
------------
QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
framework widely used to test block layer related features. It is higher level
@ -152,7 +151,7 @@ More options are supported by the ``./check`` script, run ``./check -h`` for
help.
Writing a new test case
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~
Consider writing a tests case when you are making any changes to the block
layer. An iotest case is usually the choice for that. There are already many
@ -206,7 +205,8 @@ test failure. If using such devices are explicitly desired, consider adding
``locking=off`` option to disable image locking.
Debugging a test case
-----------------------
~~~~~~~~~~~~~~~~~~~~~
The following options to the ``check`` script can be useful when debugging
a failing test:
@ -235,7 +235,7 @@ a failing test:
``$TEST_DIR/qemu-machine-<random_string>``.
Test case groups
----------------
~~~~~~~~~~~~~~~~
"Tests may belong to one or more test groups, which are defined in the form
of a comment in the test source file. By convention, test groups are listed
@ -285,10 +285,10 @@ Note that the following group names have a special meaning:
.. _container-ref:
Container based tests
=====================
---------------------
Introduction
------------
~~~~~~~~~~~~
The container testing framework in QEMU utilizes public images to
build and test QEMU in predefined and widely accessible Linux
@ -303,7 +303,7 @@ The container images are also used to augment the generation of tests
for testing TCG. See :ref:`checktcg-ref` for more details.
Docker Prerequisites
--------------------
~~~~~~~~~~~~~~~~~~~~
Install "docker" with the system package manager and start the Docker service
on your development machine, then make sure you have the privilege to run
@ -334,7 +334,7 @@ exploit the whole host with Docker bind mounting or other privileged
operations. So only do it on development machines.
Podman Prerequisites
--------------------
~~~~~~~~~~~~~~~~~~~~
Install "podman" with the system package manager.
@ -346,7 +346,7 @@ Install "podman" with the system package manager.
The last command should print an empty table, to verify the system is ready.
Quickstart
----------
~~~~~~~~~~
From source tree, type ``make docker-help`` to see the help. Testing
can be started without configuring or building QEMU (``configure`` and
@ -362,7 +362,7 @@ is downloaded and initialized automatically), in which the ``test-build`` job
is executed.
Registry
--------
~~~~~~~~
The QEMU project has a container registry hosted by GitLab at
``registry.gitlab.com/qemu-project/qemu`` which will automatically be
@ -376,7 +376,7 @@ locally by using the ``NOCACHE`` build option:
make docker-image-debian10 NOCACHE=1
Images
------
~~~~~~
Along with many other images, the ``centos8`` image is defined in a Dockerfile
in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
@ -391,7 +391,7 @@ mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
for example, to make qemu-user powered cross build containers work.
Tests
-----
~~~~~
Different tests are added to cover various configurations to build and test
QEMU. Docker tests are the executables under ``tests/docker`` named
@ -402,7 +402,7 @@ source and build it.
The full list of tests is printed in the ``make docker-help`` help.
Debugging a Docker test failure
-------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When CI tasks, maintainers or yourself report a Docker test failure, follow the
below steps to debug it:
@ -419,7 +419,7 @@ below steps to debug it:
the prompt for debug.
Options
-------
~~~~~~~
Various options can be used to affect how Docker tests are done. The full
list is in the ``make docker`` help text. The frequently used ones are:
@ -433,7 +433,7 @@ list is in the ``make docker`` help text. The frequently used ones are:
failure" section.
Thread Sanitizer
================
----------------
Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
building and testing with this tool.
@ -443,7 +443,7 @@ For more information on TSan:
https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
Thread Sanitizer in Docker
---------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~
TSan is currently supported in the ubuntu2004 docker.
The test-tsan test will build using TSan and then run make check.
@ -458,7 +458,7 @@ We recommend using DEBUG=1 to allow launching the test from inside the docker,
and to allow review of the warnings generated by TSan.
Building and Testing with TSan
------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is possible to build and test with TSan, with a few additional steps.
These steps are normally done automatically in the docker.
@ -497,7 +497,7 @@ This allows for running the test and then checking the warnings afterwards.
If you want TSan to stop and exit with error on warnings, use exitcode=66.
TSan Suppressions
-----------------
~~~~~~~~~~~~~~~~~
Keep in mind that for any data race warning, although there might be a data race
detected by TSan, there might be no actual bug here. TSan provides several
different mechanisms for suppressing warnings. In general it is recommended
@ -523,7 +523,7 @@ More information on the file format can be found here under "Blacklist Format":
https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
TSan Annotations
----------------
~~~~~~~~~~~~~~~~
include/qemu/tsan.h defines annotations. See this file for more descriptions
of the annotations themselves. Annotations can be used to suppress
TSan warnings or give TSan more information so that it can detect proper
@ -540,14 +540,14 @@ The full set of annotations can be found here:
https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
VM testing
==========
----------
This test suite contains scripts that bootstrap various guest images that have
necessary packages to build QEMU. The basic usage is documented in ``Makefile``
help which is displayed with ``make vm-help``.
Quickstart
----------
~~~~~~~~~~
Run ``make vm-help`` to list available make targets. Invoke a specific make
command to run build test in an image. For example, ``make vm-build-freebsd``
@ -562,7 +562,7 @@ concerned about attackers taking control of the guest and potentially
exploiting a QEMU security bug to compromise the host.
QEMU binaries
-------------
~~~~~~~~~~~~~
By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there
isn't one, or if it is older than 2.10, the test won't work. In this case,
@ -571,20 +571,20 @@ provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
Likewise the path to qemu-img can be set in QEMU_IMG environment variable.
Make jobs
---------
~~~~~~~~~
The ``-j$X`` option in the make command line is not propagated into the VM,
specify ``J=$X`` to control the make jobs in the guest.
Debugging
---------
~~~~~~~~~
Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
debugging and verbose output. If this is not enough, see the next section.
``V=1`` will be propagated down into the make jobs in the guest.
Manual invocation
-----------------
~~~~~~~~~~~~~~~~~
Each guest script is an executable script with the same command line options.
For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
@ -608,7 +608,7 @@ For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
$ ./netbsd --interactive --image /var/tmp/netbsd.img sh
Adding new guests
-----------------
~~~~~~~~~~~~~~~~~
Please look at existing guest scripts for how to add new guests.
@ -641,7 +641,7 @@ the script's ``main()``.
recommended.
Image fuzzer testing
====================
--------------------
An image fuzzer was added to exercise format drivers. Currently only qcow2 is
supported. To start the fuzzer, run
@ -654,7 +654,7 @@ Alternatively, some command different from "qemu-img info" can be tested, by
changing the ``-c`` option.
Acceptance tests using the Avocado Framework
============================================
--------------------------------------------
The ``tests/acceptance`` directory hosts functional tests, also known
as acceptance level tests. They're usually higher level tests, and
@ -693,7 +693,7 @@ Tests based on ``avocado_qemu.Test`` can easily:
- http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
Running tests
-------------
~~~~~~~~~~~~~
You can run the acceptance tests simply by executing:
@ -791,7 +791,7 @@ of Avocado or ``make check-acceptance``, and can also be queried using:
tests/venv/bin/avocado list tests/acceptance
Manual Installation
-------------------
~~~~~~~~~~~~~~~~~~~
To manually install Avocado and its dependencies, run:
@ -804,7 +804,7 @@ Alternatively, follow the instructions on this link:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
Overview
--------
~~~~~~~~
The ``tests/acceptance/avocado_qemu`` directory provides the
``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
@ -840,7 +840,7 @@ in the current directory, tagged as "quick", run:
avocado run -t quick .
The ``avocado_qemu.Test`` base test class
-----------------------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``avocado_qemu.Test`` class has a number of characteristics that
are worth being mentioned right away.
@ -890,7 +890,7 @@ At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
shutdown.
The ``avocado_qemu.LinuxTest`` base test class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``avocado_qemu.LinuxTest`` is further specialization of the
``avocado_qemu.Test`` class, so it contains all the characteristics of
@ -933,7 +933,7 @@ execution of a QEMU binary, giving its users:
a more succinct and intuitive way
QEMU binary selection
~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^
The QEMU binary used for the ``self.vm`` QEMUMachine instance will
primarily depend on the value of the ``qemu_bin`` parameter. If it's
@ -954,20 +954,23 @@ The resulting ``qemu_bin`` value will be preserved in the
``avocado_qemu.Test`` as an attribute with the same name.
Attribute reference
-------------------
~~~~~~~~~~~~~~~~~~~
Test
^^^^
Besides the attributes and methods that are part of the base
``avocado.Test`` class, the following attributes are available on any
``avocado_qemu.Test`` instance.
vm
~~
''
A QEMUMachine instance, initially configured according to the given
``qemu_bin`` parameter.
arch
~~~~
''''
The architecture can be used on different levels of the stack, e.g. by
the framework or by the test itself. At the framework level, it will
@ -984,7 +987,7 @@ name. If one is not given explicitly, it will either be set to
``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
cpu
~~~
'''
The cpu model that will be set to all QEMUMachine instances created
by the test.
@ -995,7 +998,7 @@ name. If one is not given explicitly, it will either be set to
``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
machine
~~~~~~~
'''''''
The machine type that will be set to all QEMUMachine instances created
by the test.
@ -1006,20 +1009,20 @@ name. If one is not given explicitly, it will either be set to
``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
qemu_bin
~~~~~~~~
''''''''
The preserved value of the ``qemu_bin`` parameter or the result of the
dynamic probe for a QEMU binary in the current working directory or
source tree.
LinuxTest
~~~~~~~~~
^^^^^^^^^
Besides the attributes present on the ``avocado_qemu.Test`` base
class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
distro
......
''''''
The name of the Linux distribution used as the guest image for the
test. The name should match the **Provider** column on the list
@ -1028,7 +1031,7 @@ of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_version
..............
''''''''''''''
The version of the Linux distribution as the guest image for the
test. The name should match the **Version** column on the list
@ -1037,7 +1040,7 @@ of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_checksum
...............
'''''''''''''''
The sha256 hash of the guest image file used for the test.
@ -1046,7 +1049,7 @@ same name), no validation on the integrity of the image will be
performed.
Parameter reference
-------------------
~~~~~~~~~~~~~~~~~~~
To understand how Avocado parameters are accessed by tests, and how
they can be passed to tests, please refer to::
@ -1060,8 +1063,11 @@ like the following:
PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
Test
^^^^
arch
~~~~
''''
The architecture that will influence the selection of a QEMU binary
(when one is not explicitly given).
@ -1074,31 +1080,30 @@ This parameter has a direct relation with the ``arch`` attribute. If
not given, it will default to None.
cpu
~~~
'''
The cpu model that will be set to all QEMUMachine instances created
by the test.
machine
~~~~~~~
'''''''
The machine type that will be set to all QEMUMachine instances created
by the test.
qemu_bin
~~~~~~~~
''''''''
The exact QEMU binary to be used on QEMUMachine.
LinuxTest
~~~~~~~~~
^^^^^^^^^
Besides the parameters present on the ``avocado_qemu.Test`` base
class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
distro
......
''''''
The name of the Linux distribution used as the guest image for the
test. The name should match the **Provider** column on the list
@ -1107,7 +1112,7 @@ of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_version
..............
''''''''''''''
The version of the Linux distribution as the guest image for the
test. The name should match the **Version** column on the list
@ -1116,7 +1121,7 @@ of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_checksum
...............
'''''''''''''''
The sha256 hash of the guest image file used for the test.
@ -1124,7 +1129,8 @@ If this value is not set in the code or by this parameter no
validation on the integrity of the image will be performed.
Skipping tests
--------------
~~~~~~~~~~~~~~
The Avocado framework provides Python decorators which allow for easily skip
tests running under certain conditions. For example, on the lack of a binary
on the test system or when the running environment is a CI system. For further
@ -1139,7 +1145,7 @@ environment variables became a kind of standard way to enable/disable tests.
Here is a list of the most used variables:
AVOCADO_ALLOW_LARGE_STORAGE
~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tests which are going to fetch or produce assets considered *large* are not
going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
the environment.
@ -1148,7 +1154,7 @@ The definition of *large* is a bit arbitrary here, but it usually means an
asset which occupies at least 1GB of size on disk when uncompressed.
AVOCADO_ALLOW_UNTRUSTED_CODE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are tests which will boot a kernel image or firmware that can be
considered not safe to run on the developer's workstation, thus they are
skipped by default. The definition of *not safe* is also arbitrary but
@ -1159,7 +1165,7 @@ You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
order to allow tests which make use of those kind of assets.
AVOCADO_TIMEOUT_EXPECTED
~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^
The Avocado framework has a timeout mechanism which interrupts tests to avoid the
test suite of getting stuck. The timeout value can be set via test parameter or
property defined in the test class, for further details::
@ -1173,7 +1179,7 @@ compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
has been used to determine whether those tests should run or not.
GITLAB_CI
~~~~~~~~~
^^^^^^^^^
A number of tests are flagged to not run on the GitLab CI. Usually because
they proved to the flaky or there are constraints on the CI environment which
would make them fail. If you encounter a similar situation then use that
@ -1186,7 +1192,7 @@ variable as shown on the code snippet below to skip the test:
do_something()
Uninstalling Avocado
--------------------
~~~~~~~~~~~~~~~~~~~~
If you've followed the manual installation instructions above, you can
easily uninstall Avocado. Start by listing the packages you have
@ -1204,7 +1210,7 @@ Avocado is installed will be cleaned up as part of ``make check-clean``.
.. _checktcg-ref:
Testing with "make check-tcg"
=============================
-----------------------------
The check-tcg tests are intended for simple smoke tests of both
linux-user and softmmu TCG functionality. However to build test
@ -1237,7 +1243,7 @@ itself.
See :ref:`container-ref` for more details.
Running subset of tests
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~
You can build the tests for one architecture::
@ -1251,7 +1257,7 @@ Adding ``V=1`` to the invocation will show the details of how to
invoke QEMU for the test which is useful for debugging tests.
TCG test dependencies
---------------------
~~~~~~~~~~~~~~~~~~~~~
The TCG tests are deliberately very light on dependencies and are
either totally bare with minimal gcc lib support (for softmmu tests)
@ -1285,7 +1291,7 @@ to run to exercise QEMU's linux-user code::
https://linux-test-project.github.io/
GCC gcov support
================
----------------
``gcov`` is a GCC tool to analyze the testing coverage by
instrumenting the tested code. To use it, configure QEMU with