Commit Graph

503 Commits

Author SHA1 Message Date
Paolo Bonzini
e8e4298fea tests: Use separate virtual environment for avocado
This reverts commits eea2d14117 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db55 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).

Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.

The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:

1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado.  The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:

   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.

So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:

  [meson]
  meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }

  [docs]
  # 6.0 drops support for Python 3.7
  sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
  sphinx_rtd_theme = { minimum = "0.5" }

  [avocado]
  avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }

Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available).  But until
this is implemented, keep avocado in a separate venv.  There is still the
benefit of using a single python for meson custom_targets and for sphinx.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06 16:29:53 +02:00
Paolo Bonzini
eaf245becd Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"
This reverts commit ec5ffa0056.

Bumping avocado to version 101 has two issues.  First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages.  Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module).  As a first
step, this patch reverts the introduction of a too-new Avocado.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06 10:31:31 +02:00
Paolo Bonzini
eea2d14117 Makefile: remove $(TESTS_PYTHON)
It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-26 10:17:32 +02:00
John Snow
9c6692db55 tests: Use configure-provided pyvenv for tests
This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-20-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Kautuk Consul
ec5ffa0056 tests/requirements.txt: bump up avocado-framework version to 101.0
Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>
2023-04-27 14:57:22 +01:00
Paolo Bonzini
b898bf2846 configure: move tests/tcg/Makefile.prereqs to root build directory
It will not be specific to tests/tcg anymore, since it will be possible to
build firmware using container-based cross compilers too.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-26-alex.bennee@linaro.org>
2022-10-06 11:53:40 +01:00
Paolo Bonzini
6ffe81370a tests/tcg: add distclean rule
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-22-alex.bennee@linaro.org>
2022-10-06 11:53:40 +01:00
Paolo Bonzini
d674342e2e tests/tcg: remove -f from Makefile invocation
Instead of linking tests/tcg/Makefile.target into the build tree, name
the symbolic link "Makefile" and create it in every target subdirectory.
This makes it possible to just invoke "make" in tests/tcg subdirectories.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-21-alex.bennee@linaro.org>
2022-10-06 11:53:40 +01:00
Paolo Bonzini
a3e28f81fd tests: simplify Makefile invocation for tests/tcg
Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation
for tests/tcg.  For DOCKER_SCRIPT, resolve the path to docker.py in configure;
for TARGET, move it to config-$(TARGET).mak and use a symbolic link to break
the cycle.

The symbolic link is still needed because tests/tcg includes dummy config files
for targets that are not buildable.  Once that is cleaned up, the symbolic link
will go away too.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-20-alex.bennee@linaro.org>
2022-10-06 11:53:40 +01:00
John Snow
0e7647aa8a tests: install "qemu" namespace package into venv
This patch adds the "qemu" namespace package to the $build/tests/venv
directory. It does so in "editable" mode, which means that changes to
the source python directory will actively be reflected by the venv.

This patch also then removes any sys.path hacking from the avocado test
scripts directly. By doing this, the environment of where to find these
packages is managed entirely by the virtual environment and not by the
scripts themselves.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-7-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
John Snow
5a6f1199cb tests: add quiet-venv-pip macro
Factor out the "test venv pip" macro; rewrite the "check-venv" rule to
be a little more compact. Replace the "PIP" pseudo-command output with
"VENVPIP" to make it 1% more clear that we are talking about using pip
to install something into a venv.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-6-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
John Snow
a547c82701 tests: silence pip upgrade warnings during venv creation
Turn off the nag warning coaxing us to upgrade pip. It's not really that
interesting to see in CI logs, and as long as nothing is broken --
nothing is broken.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-5-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
John Snow
aa19dd33ce tests: use python3 as the python executable name
Use "python3" instead of "python" as per PEP0394:
https://peps.python.org/pep-0394/

This should always be defined (in a venv, at least!), matching the
preferred python shebang of "#!/usr/bin/env python3".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-4-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
John Snow
caa0f7cbaf tests: add "TESTS_PYTHON" variable to Makefile
This is a convenience feature: $(PYTHON) points to the Python executable
we were instructed to use by the configure script. We use that Python to
create a virtual environment with the "check-venv" target in
tests/Makefile.include.

$(TESTS_PYTHON) points to the Python executable belonging to the virtual
environment tied to the build. This Python executable is a symlink to
the binary used to create the venv, which will be the version provided
at configure time.

Using $(TESTS_PYTHON) therefore uses the $(PYTHON) executable, but with
paths modified to use packages installed to the venv.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-3-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
Dario Faggioli
d426ff938b tests/Makefile.include: Fix 'make check-help' output
Since commit 3d2f73ef75 ("build: use "meson test" as the test harness"),
check-report.tap is no more, and we have check-report.junit.xml.

Update the output of 'make check-help', which was still listing
'check-report.tap', accordingly.

Fixes: 3d2f73ef75
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Message-Id: <165366545439.6869.11633009118019728798.stgit@work>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
Paolo Bonzini
cd362defbb tests/tcg: merge configure.sh back into main configure script
tests/tcg/configure.sh has a complicated story.

In the beginning its code ran as part of the creation of config-target.mak
files, and that is where it placed the information on the target compiler.
However, probing for the buildability of TCG tests required multiple
inclusions of config-target.mak in the _main_ Makefile (not in
Makefile.target, which took care of building the QEMU executables in
the pre-Meson era), which polluted the namespace.

Thus, it was moved to a separate directory.  It created small config-*.mak
files in $(BUILD_DIR)/tests/tcg.  Those were also included multiple
times, but at least they were small and manageable; this was also an
important step in disentangling the TCG tests from Makefile.target.

Since then, Meson has allowed the configure script to go on a diet.
A few compilation tests survive (mostly for sanitizers) but these days
it mostly takes care of command line parsing, looking for tools, and
setting up the environment for Meson to do its stuff.

It's time to extend configure with the capability to build for more
than just one target: not just tests, but also firmware.  As a first
step, integrate all the logic to find cross compilers in the configure
script, and move tests/tcg/configure.sh back there (though as a
separate loop, not integrated in the one that generates target
configurations for Meson).

tests/tcg is actually very close to being buildable as a standalone
project, so I actually expect the compiler tests to move back to
tests/tcg, as a "configure" script of sorts which would run at Make
time after the docker images are built.  The GCC tree has a similar idea
of doing only bare-bones tree-wide configuration and leaving the rest
for Make time.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Paolo Bonzini
95808875be build: do a full build before running TCG tests
TCG tests need both QEMU and firmware to be built, so do "ninja all" before
trying to run them.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Paolo Bonzini
eebf199c09 tests/tcg: invoke Makefile.target directly from QEMU's makefile
Build the "docker.py cc" invocation directly in tests/tcg/configure.sh, and
remove the Makefile.qemu wrapper around Makefile.target.  The config-*.mak
files now include the actual variables used when building the tests, rather
than the CROSS_* variables that Makefile.qemu used to "translate".

This is a first step towards generalizing the cross-compilation infrastructure
so that it can be used for firmware as well.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-15-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-18-alex.bennee@linaro.org>
2022-04-20 16:04:20 +01:00
Paolo Bonzini
5377a10001 tests/tcg: list test targets in Makefile.prereqs
Omit the rules altogether for targets that do not have a compiler.
Makefile.qemu now is only invoked if the tests are actually built/run.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-14-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-17-alex.bennee@linaro.org>
2022-04-20 16:04:20 +01:00
Paolo Bonzini
f2d5935145 tests/tcg: prepare Makefile.prereqs at configure time
List the dependencies of the build-tcg-tests-* and run-tcg-tests-*
targets in a Makefile fragment, without going through Makefile.prereqs's
"parsing" of config-*.mak.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-13-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-16-alex.bennee@linaro.org>
2022-04-20 16:04:20 +01:00
Thomas Huth
425d6aba99 tests/Makefile.include: Let "make clean" remove the TCG tests, too
"make clean" should clear all binaries that have been built, but so
far it left the TCG tests still in place. Let's make sure that they
are now removed, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220301085900.1443232-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2022-03-23 10:25:22 +00:00
Thomas Huth
e287072b40 tests: Do not treat the iotests as separate meson test target anymore
If there is a failing iotest, the output is currently not logged to
the console anymore. To get this working again, we need to run the
meson test runner with "--print-errorlogs" (and without "--verbose"
due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ).
We could update the "meson test" call in tests/Makefile.include,
but actually it's nicer and easier if we simply do not treat the
iotests as separate test target anymore and integrate them along
with the other test suites. This has the disadvantage of not getting
the detailed progress indication there anymore, but since that was
only working right in single-threaded "make -j1" mode anyway, it's
not a huge loss right now.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220310075048.2303495-1-thuth@redhat.com>
Tested-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2022-03-22 09:40:54 +01:00
Paolo Bonzini
20cf5cb487 configure, meson: move guest-agent, tools to meson
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-02-21 10:35:53 +01:00
Alex Bennée
135e6a09a9 tests/Makefile.include: clean-up old code
This is no longer needed since a2ce7dbd91 ("meson: convert tests/qtest
to meson", 2020-08-21)

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220204204335.1689602-2-alex.bennee@linaro.org>
2022-02-09 12:08:41 +00:00
Paolo Bonzini
98487b9035 build: make check-block a meson test
"meson test" can be asked to run tests verbosely; this makes it usable
also for qemu-iotests's own harness, and it lets "make check-block"
reuse mtest2make.py's infrastructure to find and build test dependencies.

Adjust check-block.sh to use the standard exit code that reports a test
as skipped.  Alternatively, in the future we could make it produce TAP
output, which is consistent with all other "make check" tests.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-01-28 11:13:33 +01:00
Philippe Mathieu-Daudé
6d4c8af321 tests: Fix typo in check-help output
Fix typo in 'make check-help' output.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220111175528.22294-1-f4bug@amsat.org>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2022-01-18 12:32:16 +01:00
Willian Rampazzo
bbbd9b6ec6 tests/acceptance: rename tests acceptance to tests avocado
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-11-08 17:00:22 +01:00
Willian Rampazzo
333d7036ef tests/acceptance: introduce new check-avocado target
This introduces a new `make` target, `check-avocado`, and adds a
deprecation message about the `check-acceptance` target. This is
a preparation for renaming the `tests/acceptance` folder to
`tests/avocado`.

The plan is to remove the call to the `check-avocado` target one
or two months after the release and leave the warning to force
people to move to the new `check-avocado` target.

Later, the `check-acceptance` target can be removed. The intent
is to avoid a direct impact during the current soft freeze.

Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20211105155354.154864-2-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-11-08 17:00:22 +01:00
Paolo Bonzini
097a5f19a3 build: fix "make check" without earlier "make"
"make check", if not preceded by "make", will not build the tools
needed by qemu-iotests.  This happens because qemu-iotests, aka
"make check-block", is not yet part of meson.build.

While at it, remove the reference to the now-dead QEMU_IOTESTS_HELPERS-y
variable.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-10-13 10:47:22 +02:00
Paolo Bonzini
8009806ec8 tests: add missing dependency for check-block
qemu-iotests run qemu-storage-daemon, make sure it is up to date.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-10-13 10:46:53 +02:00
John Snow
c163c723ef python, iotests: remove socket_scm_helper
It's not used anymore, now.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
Cleber Rosa
181e1ab2ad Acceptance Tests: improve check-acceptance description
The "check-acceptance" make rule won't necessarily run *all* available
tests, because it employs a filter based on the currently configured
targets.  This change in the description of the rule makes that
behavior extra clear.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210924185506.2542588-3-crosa@redhat.com>
2021-09-27 19:06:47 +02:00
Willian Rampazzo
94c714620b tests/Makefile: add AVOCADO_TESTS option to make check-acceptance
Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TESTS environment variable.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-4-willianr@redhat.com>
2021-09-27 19:06:47 +02:00
Willian Rampazzo
23022794de tests/Makefile: allow control over tags during check-acceptance
Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.

This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.

This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.

e.g.:

AVOCADO_TAGS="foo bar baz" make check-acceptance

Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-2-willianr@redhat.com>
2021-09-27 19:06:47 +02:00
Cleber Rosa
414e9ae345 Acceptance tests: do not try to reuse packages from the system
The premise behind the original behavior is that it would save people
from downloading Avocado (and other dependencies) if already installed
on the system.  To be honest, I think it's extremely rare that the
same versions described as dependencies will be available on most
systems.  But, the biggest motivations here are that:

 1) Hacking on QEMU in the same system used to develop Avocado leads
    to confusion with regards to the exact bits that are being used;

 2) Not reusing Python packages from system wide installations gives
    extra assurance that the same behavior will be seen from tests run
    on different machines;

With regards to downloads, pip already caches the downloaded wheels
and tarballs under ~/.cache/pip, so there should not be more than
one download even if the venv is destroyed and recreated.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210415215141.1865467-3-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13 16:19:41 -04:00
Alex Bennée
812b31d3f9 configs: rename default-configs to configs and reorganise
In preparation for offering variation to our build configurations lets
move everything and rename it to default. Common included base configs
are also renamed.

During the cleanup the stale usb.mak and pci.mak references were
removed from MAINTAINERS.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210707131744.26027-5-alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-07-09 18:21:34 +02:00
Paolo Bonzini
8e19c0098c tests: remove "make check-speed" in favor of "make bench"
"make check-speed" has been broken since the removal of ninja2make
last October.  It was just a backwards-compatibility alias for
"make bench-speed", which in turn is in principle a subset of
"make bench".  Advertise the latter and drop "make check-speed"
completely since no one has noticed.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210310164612.285362-1-pbonzini@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-12 15:46:30 +01:00
Cleber Rosa
235c15fac5 Acceptance Tests: restore filtering of tests by target arch
Previously, tests were being filtered by the matching target
architectures to be built.  The benefit, compared to the current
situation, is a more concise test job that won't show tests canceled
because a matching QEMU binary was not found (those tests won't even
be attempted).

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-3-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09 11:26:31 +01:00
Cleber Rosa
93fca0f241 Acceptance Tests: restore downloading of VM images
The "get-vm-images" target defined in tests/Makefile.include is a
prerequisite for "check-acceptance", so that those files get
downloaded before the Avocado job even starts.

It looks like on c401c058a1 a TARGETS variable was introduced with a
different content than it was previously coming from the main
Makefile.  From that point on, the "get-vm-images" succeed without
doing anything because there was no matching architecture to download.

This restores the download of images (that match targets to be built)
before the job starts, eliminating downloads and their associated
failures during the tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-2-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09 11:26:31 +01:00
Alex Bennée
c401c058a1 tests/Makefile.include: don't use TARGET_DIRS for check-tcg
TARGET_DIRS reflects what we wanted to configure which in the normal
case is all our targets. However once meson has pared-down our target
list due to missing features we need to check the final list of
ninja-targets. This prevents check-tcg barfing on a --disable-tcg
build.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-14-alex.bennee@linaro.org>
2021-02-08 10:55:20 +00:00
Alex Bennée
47e3424ac9 scripts/mtest2make.py: export all-%s-targets variable and use it
There are some places where the conditional makefile support is the
simplest solution. Now we don't expose CONFIG_TCG as a variable create
a new one that can be checked for the check-help output.

As check-tcg is a PHONY target we re-use check-softfloat to gate that
as well.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>
2021-02-08 10:55:20 +00:00
Vladimir Sementsov-Ogievskiy
4cea90be62 tests/Makefile.include: export PYTHON for check-block.sh
check-block.sh called by make check-block rely on PYTHON variable being
set.

Fixes: f203080bbd
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210129051346.56109-1-vsementsov@virtuozzo.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-29 12:32:36 +01:00
Paolo Bonzini
09e93326e4 build: replace ninjatool with ninja
Now that the build is done entirely by Meson, there is no need
to keep the Makefile conversion.  Instead, we can ask Ninja about
the targets it exposes and forward them.

The main advantages are, from smallest to largest:

- reducing the possible namespace pollution within the Makefile

- removal of a relatively large Python program

- faster build because parsing Makefile.ninja is slower than
parsing build.ninja; and faster build after Meson runs because
we do not have to generate Makefile.ninja.

- tracking of command lines, which provides more accurate rebuilds

In addition the change removes the requirement for GNU make 3.82, which
was annoying on Mac, and avoids bugs on Windows due to ninjatool not
knowing how to convert Windows escapes to POSIX escapes.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-10-17 10:45:51 -04:00
Paolo Bonzini
88da4b043b tests/Makefile.include: unbreak non-tcg builds
Remove from check-block the requirement that all TARGET_DIRS are built.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-10-17 10:45:49 -04:00
Paolo Bonzini
8c17b8403c tests: tcg: do not use implicit rules
Use pattern rules to clarify which targets are going to match the
rule and to provide clearer error messages.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-10-04 18:36:23 +02:00
Alex Bennée
c17a386b6a plugins: move the more involved plugins to contrib
We have an exploding complexity problem in the testing so lets just
move the more involved plugins into contrib. tests/plugins still exist
for the basic plugins that exercise the API. We restore the old
pre-meson style Makefile for contrib as it also doubles as a guide for
out-of-tree plugin builds.

While we are at it add some examples to the documentation and a
specific plugins build target.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200909112742.25730-11-alex.bennee@linaro.org>
2020-09-10 10:47:03 +01:00
Pavel Dovgalyuk
94b4ec24b9 tests: bump avocado version
Reverse debugging test uses gdb remote client of avocado framework.
This client was fixed since the currently used version 76.
Therefore this patch bumps the version to 81 and fixes command
line version compatibility issue.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <159903462803.28509.16851113546106095750.stgit@pasha-ThinkPad-X280>
Message-Id: <20200908202352.298506-2-philmd@redhat.com>
Message-Id: <20200909112742.25730-9-alex.bennee@linaro.org>
2020-09-10 10:46:46 +01:00
Paolo Bonzini
296bf3d12b Makefile: remove dead variables and includes
Makefile.objs, the .d files and various CONFIG_* symbols are not
used anymore by the Make side of the build; they are only processed
by Meson.  We can delete them.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-09-08 11:43:16 +02:00
Marc-André Lureau
4fbcdcd742 meson: convert migration/initrd-stress
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-17-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-09-08 11:43:16 +02:00
Paolo Bonzini
9ed7247a59 meson: convert the speed tests
Use meson benchmark() for them, adjust mtest2make.py for that.
A new target "make bench" can be used to run all benchmarks.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-14-marcandre.lureau@redhat.com>
[Rewrite mtest2make part. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-09-08 11:43:16 +02:00