Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.
Bump up the avocado-framework version to 101.0.
Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>
It will not be specific to tests/tcg anymore, since it will be possible to
build firmware using container-based cross compilers too.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-26-alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-22-alex.bennee@linaro.org>
Instead of linking tests/tcg/Makefile.target into the build tree, name
the symbolic link "Makefile" and create it in every target subdirectory.
This makes it possible to just invoke "make" in tests/tcg subdirectories.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-21-alex.bennee@linaro.org>
Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation
for tests/tcg. For DOCKER_SCRIPT, resolve the path to docker.py in configure;
for TARGET, move it to config-$(TARGET).mak and use a symbolic link to break
the cycle.
The symbolic link is still needed because tests/tcg includes dummy config files
for targets that are not buildable. Once that is cleaned up, the symbolic link
will go away too.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-20-alex.bennee@linaro.org>
This patch adds the "qemu" namespace package to the $build/tests/venv
directory. It does so in "editable" mode, which means that changes to
the source python directory will actively be reflected by the venv.
This patch also then removes any sys.path hacking from the avocado test
scripts directly. By doing this, the environment of where to find these
packages is managed entirely by the virtual environment and not by the
scripts themselves.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-7-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Factor out the "test venv pip" macro; rewrite the "check-venv" rule to
be a little more compact. Replace the "PIP" pseudo-command output with
"VENVPIP" to make it 1% more clear that we are talking about using pip
to install something into a venv.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-6-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Turn off the nag warning coaxing us to upgrade pip. It's not really that
interesting to see in CI logs, and as long as nothing is broken --
nothing is broken.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-5-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use "python3" instead of "python" as per PEP0394:
https://peps.python.org/pep-0394/
This should always be defined (in a venv, at least!), matching the
preferred python shebang of "#!/usr/bin/env python3".
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-4-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This is a convenience feature: $(PYTHON) points to the Python executable
we were instructed to use by the configure script. We use that Python to
create a virtual environment with the "check-venv" target in
tests/Makefile.include.
$(TESTS_PYTHON) points to the Python executable belonging to the virtual
environment tied to the build. This Python executable is a symlink to
the binary used to create the venv, which will be the version provided
at configure time.
Using $(TESTS_PYTHON) therefore uses the $(PYTHON) executable, but with
paths modified to use packages installed to the venv.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-3-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Since commit 3d2f73ef75 ("build: use "meson test" as the test harness"),
check-report.tap is no more, and we have check-report.junit.xml.
Update the output of 'make check-help', which was still listing
'check-report.tap', accordingly.
Fixes: 3d2f73ef75
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Message-Id: <165366545439.6869.11633009118019728798.stgit@work>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
tests/tcg/configure.sh has a complicated story.
In the beginning its code ran as part of the creation of config-target.mak
files, and that is where it placed the information on the target compiler.
However, probing for the buildability of TCG tests required multiple
inclusions of config-target.mak in the _main_ Makefile (not in
Makefile.target, which took care of building the QEMU executables in
the pre-Meson era), which polluted the namespace.
Thus, it was moved to a separate directory. It created small config-*.mak
files in $(BUILD_DIR)/tests/tcg. Those were also included multiple
times, but at least they were small and manageable; this was also an
important step in disentangling the TCG tests from Makefile.target.
Since then, Meson has allowed the configure script to go on a diet.
A few compilation tests survive (mostly for sanitizers) but these days
it mostly takes care of command line parsing, looking for tools, and
setting up the environment for Meson to do its stuff.
It's time to extend configure with the capability to build for more
than just one target: not just tests, but also firmware. As a first
step, integrate all the logic to find cross compilers in the configure
script, and move tests/tcg/configure.sh back there (though as a
separate loop, not integrated in the one that generates target
configurations for Meson).
tests/tcg is actually very close to being buildable as a standalone
project, so I actually expect the compiler tests to move back to
tests/tcg, as a "configure" script of sorts which would run at Make
time after the docker images are built. The GCC tree has a similar idea
of doing only bare-bones tree-wide configuration and leaving the rest
for Make time.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org>
TCG tests need both QEMU and firmware to be built, so do "ninja all" before
trying to run them.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org>
Build the "docker.py cc" invocation directly in tests/tcg/configure.sh, and
remove the Makefile.qemu wrapper around Makefile.target. The config-*.mak
files now include the actual variables used when building the tests, rather
than the CROSS_* variables that Makefile.qemu used to "translate".
This is a first step towards generalizing the cross-compilation infrastructure
so that it can be used for firmware as well.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-15-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-18-alex.bennee@linaro.org>
Omit the rules altogether for targets that do not have a compiler.
Makefile.qemu now is only invoked if the tests are actually built/run.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-14-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-17-alex.bennee@linaro.org>
List the dependencies of the build-tcg-tests-* and run-tcg-tests-*
targets in a Makefile fragment, without going through Makefile.prereqs's
"parsing" of config-*.mak.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220401141326.1244422-13-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220419091020.3008144-16-alex.bennee@linaro.org>
"make clean" should clear all binaries that have been built, but so
far it left the TCG tests still in place. Let's make sure that they
are now removed, too.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220301085900.1443232-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
If there is a failing iotest, the output is currently not logged to
the console anymore. To get this working again, we need to run the
meson test runner with "--print-errorlogs" (and without "--verbose"
due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ).
We could update the "meson test" call in tests/Makefile.include,
but actually it's nicer and easier if we simply do not treat the
iotests as separate test target anymore and integrate them along
with the other test suites. This has the disadvantage of not getting
the detailed progress indication there anymore, but since that was
only working right in single-threaded "make -j1" mode anyway, it's
not a huge loss right now.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220310075048.2303495-1-thuth@redhat.com>
Tested-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
This is no longer needed since a2ce7dbd91 ("meson: convert tests/qtest
to meson", 2020-08-21)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220204204335.1689602-2-alex.bennee@linaro.org>
"meson test" can be asked to run tests verbosely; this makes it usable
also for qemu-iotests's own harness, and it lets "make check-block"
reuse mtest2make.py's infrastructure to find and build test dependencies.
Adjust check-block.sh to use the standard exit code that reports a test
as skipped. Alternatively, in the future we could make it produce TAP
output, which is consistent with all other "make check" tests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.
This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.
[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html
Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This introduces a new `make` target, `check-avocado`, and adds a
deprecation message about the `check-acceptance` target. This is
a preparation for renaming the `tests/acceptance` folder to
`tests/avocado`.
The plan is to remove the call to the `check-avocado` target one
or two months after the release and leave the warning to force
people to move to the new `check-avocado` target.
Later, the `check-acceptance` target can be removed. The intent
is to avoid a direct impact during the current soft freeze.
Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20211105155354.154864-2-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
"make check", if not preceded by "make", will not build the tools
needed by qemu-iotests. This happens because qemu-iotests, aka
"make check-block", is not yet part of meson.build.
While at it, remove the reference to the now-dead QEMU_IOTESTS_HELPERS-y
variable.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It's not used anymore, now.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The "check-acceptance" make rule won't necessarily run *all* available
tests, because it employs a filter based on the currently configured
targets. This change in the description of the rule makes that
behavior extra clear.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210924185506.2542588-3-crosa@redhat.com>
Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TESTS environment variable.
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-4-willianr@redhat.com>
Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.
This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.
This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.
e.g.:
AVOCADO_TAGS="foo bar baz" make check-acceptance
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210923161141.232208-2-willianr@redhat.com>
The premise behind the original behavior is that it would save people
from downloading Avocado (and other dependencies) if already installed
on the system. To be honest, I think it's extremely rare that the
same versions described as dependencies will be available on most
systems. But, the biggest motivations here are that:
1) Hacking on QEMU in the same system used to develop Avocado leads
to confusion with regards to the exact bits that are being used;
2) Not reusing Python packages from system wide installations gives
extra assurance that the same behavior will be seen from tests run
on different machines;
With regards to downloads, pip already caches the downloaded wheels
and tarballs under ~/.cache/pip, so there should not be more than
one download even if the venv is destroyed and recreated.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210415215141.1865467-3-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
In preparation for offering variation to our build configurations lets
move everything and rename it to default. Common included base configs
are also renamed.
During the cleanup the stale usb.mak and pci.mak references were
removed from MAINTAINERS.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210707131744.26027-5-alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
"make check-speed" has been broken since the removal of ninja2make
last October. It was just a backwards-compatibility alias for
"make bench-speed", which in turn is in principle a subset of
"make bench". Advertise the latter and drop "make check-speed"
completely since no one has noticed.
Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210310164612.285362-1-pbonzini@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Previously, tests were being filtered by the matching target
architectures to be built. The benefit, compared to the current
situation, is a more concise test job that won't show tests canceled
because a matching QEMU binary was not found (those tests won't even
be attempted).
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-3-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The "get-vm-images" target defined in tests/Makefile.include is a
prerequisite for "check-acceptance", so that those files get
downloaded before the Avocado job even starts.
It looks like on c401c058a1 a TARGETS variable was introduced with a
different content than it was previously coming from the main
Makefile. From that point on, the "get-vm-images" succeed without
doing anything because there was no matching architecture to download.
This restores the download of images (that match targets to be built)
before the job starts, eliminating downloads and their associated
failures during the tests.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210225232122.1254879-2-crosa@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
TARGET_DIRS reflects what we wanted to configure which in the normal
case is all our targets. However once meson has pared-down our target
list due to missing features we need to check the final list of
ninja-targets. This prevents check-tcg barfing on a --disable-tcg
build.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-14-alex.bennee@linaro.org>
There are some places where the conditional makefile support is the
simplest solution. Now we don't expose CONFIG_TCG as a variable create
a new one that can be checked for the check-help output.
As check-tcg is a PHONY target we re-use check-softfloat to gate that
as well.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>
check-block.sh called by make check-block rely on PYTHON variable being
set.
Fixes: f203080bbd
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210129051346.56109-1-vsementsov@virtuozzo.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Now that the build is done entirely by Meson, there is no need
to keep the Makefile conversion. Instead, we can ask Ninja about
the targets it exposes and forward them.
The main advantages are, from smallest to largest:
- reducing the possible namespace pollution within the Makefile
- removal of a relatively large Python program
- faster build because parsing Makefile.ninja is slower than
parsing build.ninja; and faster build after Meson runs because
we do not have to generate Makefile.ninja.
- tracking of command lines, which provides more accurate rebuilds
In addition the change removes the requirement for GNU make 3.82, which
was annoying on Mac, and avoids bugs on Windows due to ninjatool not
knowing how to convert Windows escapes to POSIX escapes.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Remove from check-block the requirement that all TARGET_DIRS are built.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use pattern rules to clarify which targets are going to match the
rule and to provide clearer error messages.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
We have an exploding complexity problem in the testing so lets just
move the more involved plugins into contrib. tests/plugins still exist
for the basic plugins that exercise the API. We restore the old
pre-meson style Makefile for contrib as it also doubles as a guide for
out-of-tree plugin builds.
While we are at it add some examples to the documentation and a
specific plugins build target.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200909112742.25730-11-alex.bennee@linaro.org>
Reverse debugging test uses gdb remote client of avocado framework.
This client was fixed since the currently used version 76.
Therefore this patch bumps the version to 81 and fixes command
line version compatibility issue.
Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <159903462803.28509.16851113546106095750.stgit@pasha-ThinkPad-X280>
Message-Id: <20200908202352.298506-2-philmd@redhat.com>
Message-Id: <20200909112742.25730-9-alex.bennee@linaro.org>
Makefile.objs, the .d files and various CONFIG_* symbols are not
used anymore by the Make side of the build; they are only processed
by Meson. We can delete them.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use meson benchmark() for them, adjust mtest2make.py for that.
A new target "make bench" can be used to run all benchmarks.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-14-marcandre.lureau@redhat.com>
[Rewrite mtest2make part. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It was covered already in commit d3ca592b3c ("meson: convert check-block")
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-9-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
