e8e4298fea
This reverts commitseea2d14117("Makefile: remove $(TESTS_PYTHON)", 2023-05-26) and9c6692db55("tests: Use configure-provided pyvenv for tests", 2023-05-18). Right now, there is a conflict between wanting a ">=" constraint when using a distro-provided package and wanting a "==" constraint when installing Avocado from PyPI; this would provide the best of both worlds in terms of resiliency for both distros that have required packages and distros that don't. The conflict is visible also for meson, where we would like to install the latest 0.63.x version but also accept a distro 1.1.x version. But it is worse for avocado, for two reasons: 1) we cannot use an "==" constraint to install avocado if the venv includes a system avocado. The distro will package plugins that have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' 2) we cannot use ">=" either if the venv does _not_ include a system avocado, because that would result in the installation of v101.0 which is the one we've just reverted. So the idea is to encode the dependencies as an (acceptable, locked) tuple, like this hypothetical TOML that would be committed inside python/ and used by mkvenv.py: [meson] meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" } [docs] # 6.0 drops support for Python 3.7 sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" } sphinx_rtd_theme = { minimum = "0.5" } [avocado] avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" } Once this is implemented, it would also be possible to install avocado in pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora and CentOS Stream (the only distros where it's available). But until this is implemented, keep avocado in a separate venv. There is still the benefit of using a single python for meson custom_targets and for sphinx. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
145 lines
6.8 KiB
ReStructuredText
145 lines
6.8 KiB
ReStructuredText
=============================================================================
|
|
ACPI/SMBIOS avocado tests using biosbits
|
|
=============================================================================
|
|
|
|
Biosbits is a software written by Josh Triplett that can be downloaded
|
|
from https://biosbits.org/. The github codebase can be found
|
|
`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that executes
|
|
the bios components such as acpi and smbios tables directly through acpica
|
|
bios interpreter (a freely available C based library written by Intel,
|
|
downloadable from https://acpica.org/ and is included with biosbits) without an
|
|
operating system getting involved in between.
|
|
There are several advantages to directly testing the bios in a real physical
|
|
machine or VM as opposed to indirectly discovering bios issues through the
|
|
operating system. For one thing, the OSes tend to hide bios problems from the
|
|
end user. The other is that we have more control of what we wanted to test
|
|
and how by directly using acpica interpreter on top of the bios on a running
|
|
system. More details on the inspiration for developing biosbits and its real
|
|
life uses can be found in [#a]_ and [#b]_.
|
|
For QEMU, we maintain a fork of bios bits in gitlab along with all the
|
|
dependent submodules here: https://gitlab.com/qemu-project/biosbits-bits
|
|
This fork contains numerous fixes, a newer acpica and changes specific to
|
|
running this avocado QEMU tests using bits. The author of this document
|
|
is the sole maintainer of the QEMU fork of bios bits repo.
|
|
|
|
Under the directory ``tests/avocado/``, ``acpi-bits.py`` is a QEMU avocado
|
|
test that drives all this.
|
|
|
|
A brief description of the various test files follows.
|
|
|
|
Under ``tests/avocado/`` as the root we have:
|
|
|
|
::
|
|
|
|
├── acpi-bits
|
|
│ ├── bits-config
|
|
│ │ └── bits-cfg.txt
|
|
│ ├── bits-tests
|
|
│ ├── smbios.py2
|
|
│ ├── testacpi.py2
|
|
│ └── testcpuid.py2
|
|
├── acpi-bits.py
|
|
|
|
* ``tests/avocado``:
|
|
|
|
``acpi-bits.py``:
|
|
This is the main python avocado test script that generates a
|
|
biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
|
|
test failures. This is the script one would be interested in if they wanted
|
|
to add or change some component of the log parsing, add a new command line
|
|
to alter how QEMU is spawned etc. Test writers typically would not need to
|
|
modify this script unless they wanted to enhance or change the log parsing
|
|
for their tests. In order to enable debugging, you can set **V=1**
|
|
environment variable. This enables verbose mode for the test and also dumps
|
|
the entire log from bios bits and more information in case failure happens.
|
|
You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
|
|
verbose logs and also retain the temporary work directory the test used for
|
|
you to inspect and run the specific commands manually.
|
|
|
|
In order to run this test, please perform the following steps from the QEMU
|
|
build directory:
|
|
::
|
|
|
|
$ make check-venv (needed only the first time to create the venv)
|
|
$ ./tests/venv/bin/avocado run -t acpi tests/avocado
|
|
|
|
The above will run all acpi avocado tests including this one.
|
|
In order to run the individual tests, perform the following:
|
|
::
|
|
|
|
$ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
|
|
|
|
The above will produce output in tap format. You can omit "--tap -" in the
|
|
end and it will produce output like the following:
|
|
::
|
|
|
|
$ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
|
|
Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
|
|
JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef
|
|
JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
|
|
(1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s)
|
|
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
|
|
JOB TIME : 39.22 s
|
|
|
|
You can inspect the log file for more information about the run or in order
|
|
to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs
|
|
would be found in the test log.
|
|
|
|
* ``tests/avocado/acpi-bits/bits-config``:
|
|
|
|
This location contains biosbits configuration files that determine how the
|
|
software runs the tests.
|
|
|
|
``bits-config.txt``:
|
|
This is the biosbits config file that determines what tests
|
|
or actions are performed by bits. The description of the config options are
|
|
provided in the file itself.
|
|
|
|
* ``tests/avocado/acpi-bits/bits-tests``:
|
|
|
|
This directory contains biosbits python based tests that are run from within
|
|
the biosbits environment in the spawned VM. New additions of test cases can
|
|
be made in the appropriate test file. For example, new acpi tests can go
|
|
into testacpi.py2 and one would call testsuite.add_test() to register the new
|
|
test so that it gets executed as a part of the ACPI tests.
|
|
It might be occasionally necessary to disable some subtests or add a new
|
|
test that belongs to a test suite not already present in this directory. To
|
|
do this, please clone the bits source from
|
|
https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
|
|
Note that this is the "qemu-bits" branch and not the "bits" branch of the
|
|
repository. "qemu-bits" is the branch where we have made all the QEMU
|
|
specific enhancements and we must use the source from this branch only.
|
|
Copy the test suite/script that needs modification (addition of new tests
|
|
or disabling them) from python directory into this directory. For
|
|
example, in order to change cpuid related tests, copy the following
|
|
file into this directory and rename it with .py2 extension:
|
|
https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
|
|
Then make your additions and changes here. Therefore, the steps are:
|
|
|
|
(a) Copy unmodified test script to this directory from bits source.
|
|
(b) Add a SPDX license header.
|
|
(c) Perform modifications to the test.
|
|
|
|
Commits (a), (b) and (c) should go under separate commits so that the original
|
|
test script and the changes we have made are separated and clear.
|
|
|
|
The test framework will then use your modified test script to run the test.
|
|
No further changes would be needed. Please check the logs to make sure that
|
|
appropriate changes have taken effect.
|
|
|
|
The tests have an extension .py2 in order to indicate that:
|
|
|
|
(a) They are python2.7 based scripts and not python 3 scripts.
|
|
(b) They are run from within the bios bits VM and is not subjected to QEMU
|
|
build/test python script maintenance and dependency resolutions.
|
|
(c) They need not be loaded by avocado framework when running tests.
|
|
|
|
|
|
Author: Ani Sinha <anisinha@redhat.com>
|
|
|
|
References:
|
|
-----------
|
|
.. [#a] https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
|
|
.. [#b] https://www.youtube.com/watch?v=36QIepyUuhg
|
|
|