"meson test" can be asked to run tests verbosely; this makes it usable
also for qemu-iotests's own harness, and it lets "make check-block"
reuse mtest2make.py's infrastructure to find and build test dependencies.
Adjust check-block.sh to use the standard exit code that reports a test
as skipped. Alternatively, in the future we could make it produce TAP
output, which is consistent with all other "make check" tests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.
This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.
[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html
Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This introduces a new `make` target, `check-avocado`, and adds a
deprecation message about the `check-acceptance` target. This is
a preparation for renaming the `tests/acceptance` folder to
`tests/avocado`.
The plan is to remove the call to the `check-avocado` target one
or two months after the release and leave the warning to force
people to move to the new `check-avocado` target.
Later, the `check-acceptance` target can be removed. The intent
is to avoid a direct impact during the current soft freeze.
Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20211105155354.154864-2-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
"make check", if not preceded by "make", will not build the tools
needed by qemu-iotests. This happens because qemu-iotests, aka
"make check-block", is not yet part of meson.build.
While at it, remove the reference to the now-dead QEMU_IOTESTS_HELPERS-y
variable.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It's not used anymore, now.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The "check-acceptance" make rule won't necessarily run *all* available
tests, because it employs a filter based on the currently configured
targets. This change in the description of the rule makes that
behavior extra clear.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210924185506.2542588-3-crosa@redhat.com>
Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TESTS environment variable.
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-4-willianr@redhat.com>
Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.
This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.
This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.
e.g.:
AVOCADO_TAGS="foo bar baz" make check-acceptance
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-2-willianr@redhat.com>
The premise behind the original behavior is that it would save people
from downloading Avocado (and other dependencies) if already installed
on the system. To be honest, I think it's extremely rare that the
same versions described as dependencies will be available on most
systems. But, the biggest motivations here are that:
1) Hacking on QEMU in the same system used to develop Avocado leads
to confusion with regards to the exact bits that are being used;
2) Not reusing Python packages from system wide installations gives
extra assurance that the same behavior will be seen from tests run
on different machines;
With regards to downloads, pip already caches the downloaded wheels
and tarballs under ~/.cache/pip, so there should not be more than
one download even if the venv is destroyed and recreated.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210415215141.1865467-3-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
In preparation for offering variation to our build configurations lets
move everything and rename it to default. Common included base configs
are also renamed.
During the cleanup the stale usb.mak and pci.mak references were
removed from MAINTAINERS.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210707131744.26027-5-alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
"make check-speed" has been broken since the removal of ninja2make
last October. It was just a backwards-compatibility alias for
"make bench-speed", which in turn is in principle a subset of
"make bench". Advertise the latter and drop "make check-speed"
completely since no one has noticed.
Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210310164612.285362-1-pbonzini@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Previously, tests were being filtered by the matching target
architectures to be built. The benefit, compared to the current
situation, is a more concise test job that won't show tests canceled
because a matching QEMU binary was not found (those tests won't even
be attempted).
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-3-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The "get-vm-images" target defined in tests/Makefile.include is a
prerequisite for "check-acceptance", so that those files get
downloaded before the Avocado job even starts.
It looks like on c401c058a1 a TARGETS variable was introduced with a
different content than it was previously coming from the main
Makefile. From that point on, the "get-vm-images" succeed without
doing anything because there was no matching architecture to download.
This restores the download of images (that match targets to be built)
before the job starts, eliminating downloads and their associated
failures during the tests.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-2-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
TARGET_DIRS reflects what we wanted to configure which in the normal
case is all our targets. However once meson has pared-down our target
list due to missing features we need to check the final list of
ninja-targets. This prevents check-tcg barfing on a --disable-tcg
build.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-14-alex.bennee@linaro.org>
There are some places where the conditional makefile support is the
simplest solution. Now we don't expose CONFIG_TCG as a variable create
a new one that can be checked for the check-help output.
As check-tcg is a PHONY target we re-use check-softfloat to gate that
as well.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>
check-block.sh called by make check-block rely on PYTHON variable being
set.
Fixes: f203080bbd
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210129051346.56109-1-vsementsov@virtuozzo.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Now that the build is done entirely by Meson, there is no need
to keep the Makefile conversion. Instead, we can ask Ninja about
the targets it exposes and forward them.
The main advantages are, from smallest to largest:
- reducing the possible namespace pollution within the Makefile
- removal of a relatively large Python program
- faster build because parsing Makefile.ninja is slower than
parsing build.ninja; and faster build after Meson runs because
we do not have to generate Makefile.ninja.
- tracking of command lines, which provides more accurate rebuilds
In addition the change removes the requirement for GNU make 3.82, which
was annoying on Mac, and avoids bugs on Windows due to ninjatool not
knowing how to convert Windows escapes to POSIX escapes.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Remove from check-block the requirement that all TARGET_DIRS are built.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use pattern rules to clarify which targets are going to match the
rule and to provide clearer error messages.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
We have an exploding complexity problem in the testing so lets just
move the more involved plugins into contrib. tests/plugins still exist
for the basic plugins that exercise the API. We restore the old
pre-meson style Makefile for contrib as it also doubles as a guide for
out-of-tree plugin builds.
While we are at it add some examples to the documentation and a
specific plugins build target.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200909112742.25730-11-alex.bennee@linaro.org>
Reverse debugging test uses gdb remote client of avocado framework.
This client was fixed since the currently used version 76.
Therefore this patch bumps the version to 81 and fixes command
line version compatibility issue.
Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <159903462803.28509.16851113546106095750.stgit@pasha-ThinkPad-X280>
Message-Id: <20200908202352.298506-2-philmd@redhat.com>
Message-Id: <20200909112742.25730-9-alex.bennee@linaro.org>
Makefile.objs, the .d files and various CONFIG_* symbols are not
used anymore by the Make side of the build; they are only processed
by Meson. We can delete them.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use meson benchmark() for them, adjust mtest2make.py for that.
A new target "make bench" can be used to run all benchmarks.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-14-marcandre.lureau@redhat.com>
[Rewrite mtest2make part. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It was covered already in commit d3ca592b3c ("meson: convert check-block")
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-9-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This is required by test-qht-par unit test.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-5-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Rename the variable to be more explicit. A further clean-up patch will
move the actual to dependency check to meson entirely.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-4-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- builds QAPI builtins types/visitor to fix a linking issue with
unresolved symbols in the static library.
- work around a meson limitation on generated file output directories.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
check-block has its own test harness, unlike every other test. If
we capture its output, as is in general nicer to do without V=1,
there will be no sign of progress. So for lack of a better option
just move the invocation of the test back to Makefile rules.
As a side effect, this will also fix "make check" in --disable-tools
builds, as they were trying to run qemu-iotests without having
made qemu-img before.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
test-image-locking.c uses the qemu_lock_fd_test() function which is
only available on POSIX-like systems.
Message-Id: <20200804170055.2851-4-thuth@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20200823111757.72002-4-thuth@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The binaries move to the root directory, e.g. qemu-system-i386 or
qemu-arm. This requires changes to qtests, CI, etc.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Rules to execute tests are generated by a simple Python program
that integrates into the existing "make check" mechanism. This
provides familiarity for developers, and also allows piecewise
conversion of the testsuite Makefiles to meson.
The generated rules are based on QEMU's existing test harness
Makefile and TAP parser.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- tests/vm support for aarch64 VMs
- tests/tcg better cross-compiler detection
- update docker tooling to support registries
- update docker support for xtensa
- gitlab build docker images and store in registry
- gitlab use docker images for builds
- a number of skipIf updates to support move
- linux-user MAP_FIXED_NOREPLACE fix
- qht-bench compiler tweaks
- configure fix for secret keyring
- tsan fiber annotation clean-up
- doc updates for mttcg/icount/gdbstub
- fix cirrus to use brew bash for iotests
- revert virtio-gpu breakage
- fix LC_ALL to avoid sorting changes in iotests
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAl8J0yoACgkQ+9DbCVqe
KkSzTAf/Vn+9TU8Qt7nZvl7W4tz7Sy5K8EJGwj2RXx6CWWWLiFbsXurIM8Krw5Vc
RmvUxwa359b+J0lQpfeNDHYm1nM8RZLFlkG0a5bl0I8sW0EcPjBRtwNaGKXh2p0u
u2RS2QAi6A9AvYT4ZREYlBM+o9WzbxCEQm4s8fr6WEJCQfxBnb5/bGiEjWR64e8C
j9Kvou+zAKfVizbQMtu+mwqjsoPtcS1b3vVcO7anhNuUsuaEKkS0dFWzWvw3lwJR
STIYnb8Y/eJ1yKr0hPH2qtWv3n6yhlYvYmpUCH6AwshGMUoeFEzR2VoWS6yZPGG6
na6XA3UW5R9AxIDfkCJ5ueeo8t9xMQ==
=HRWa
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-and-misc-110720-2' into staging
Testing and misc build updates:
- tests/vm support for aarch64 VMs
- tests/tcg better cross-compiler detection
- update docker tooling to support registries
- update docker support for xtensa
- gitlab build docker images and store in registry
- gitlab use docker images for builds
- a number of skipIf updates to support move
- linux-user MAP_FIXED_NOREPLACE fix
- qht-bench compiler tweaks
- configure fix for secret keyring
- tsan fiber annotation clean-up
- doc updates for mttcg/icount/gdbstub
- fix cirrus to use brew bash for iotests
- revert virtio-gpu breakage
- fix LC_ALL to avoid sorting changes in iotests
# gpg: Signature made Sat 11 Jul 2020 15:56:42 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-testing-and-misc-110720-2: (50 commits)
iotests: Set LC_ALL=C for sort
Revert "vga: build virtio-gpu as module"
tests: fix "make check-qtest" for modular builds
.cirrus.yml: add bash to the brew packages
tests/docker: update toolchain set in debian-xtensa-cross
tests/docker: fall back more gracefully when pull fails
docs: Add to gdbstub documentation the PhyMemMode
docs/devel: add some notes on tcg-icount for developers
docs/devel: convert and update MTTCG design document
tests/qht-bench: Adjust threshold computation
tests/qht-bench: Adjust testing rate by -1
travis.yml: Test also the other targets on s390x
shippable: pull images from registry instead of building
testing: add check-build target
containers.yml: build with docker.py tooling
gitlab: limit re-builds of the containers
tests: improve performance of device-introspect-test
gitlab: add avocado asset caching
gitlab: enable check-tcg for linux-user tests
linux-user/elfload: use MAP_FIXED_NOREPLACE in pgb_reserved_va
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
If we want to continue to split build and check phase it seems like a
good idea to allow building of the tests during our multi-threaded
build phase.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200701135652.1366-40-alex.bennee@linaro.org>
As part of migrating things from Travis to GitLab add the acceptance
tests. To do this:
- rename system1 to system-ubuntu-main
- rename system2 to system-fedora-misc
- split into build/check/acceptance
- remove -j from check stages
- use artifacts to save build stage
- add post acceptance template and use
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-31-alex.bennee@linaro.org>
If a test is unsuccessful, the result is "not ok", which does not match
the regex because it includes a space.
This regex matches both "ok" and "not ok".
Signed-off-by: Havard Skinnemoen <hskinnemoen@google.com>
Message-Id: <20200628213046.2028271-1-hskinnemoen@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
