ppc has no avocado tests for the KVM backend. Add a KVM boot_linux.py
test for pseries.
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
The ppc64 and s390x tests were first marked skipIf GITLAB_CI by commit
c0c8687ef0 ("tests/avocado: disable BootLinuxPPC64 test in CI"), and
commit 0f26d94ec9 ("tests/acceptance: skip s390x_ccw_vrtio_tcg on
GitLab") due to being very heavy-weight for gitlab CI.
Commit 9b45cc9931 ("docs/devel: rationalise unstable gitlab tests under
FLAKY_TESTS") changed this to being flaky but it isn't really, it just
had a long runtime.
So take the SPEED=slow variable from qtests and introduce it to avocado,
and make these tests require it.
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
One problem with flaky tests is they often only fail under CI
conditions which makes it hard to debug. We add an optional allow_fail
job so developers can trigger the only the flaky tests in the CI
environment if they are debugging.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-8-alex.bennee@linaro.org>
It doesn't make sense to have two classes of flaky tests. While it may
take the constrained environment of CI to trigger failures easily it
doesn't mean they don't occasionally happen on developer machines. As
CI is the gating factor to passing there is no point developers
running the tests locally anyway unless they are trying to fix things.
While we are at it update the language in the docs to discourage the
QEMU_TEST_FLAKY_TESTS becoming a permanent solution.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
This test is exceptionally heavyweight (nearly 330s) compared to the
two (both endians) TuxRun baseline tests which complete in under 160s.
The coverage is slightly reduced but a more directed test could make
up the difference.
tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:
Overall coverage rate:
lines......: 9.6% (44110 of 458817 lines)
functions..: 16.5% (6767 of 41054 functions)
branches...: 6.0% (13395 of 222634 branches)
tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:
Overall coverage rate:
lines......: 11.6% (53408 of 458817 lines)
functions..: 18.7% (7691 of 41054 functions)
branches...: 7.9% (17692 of 224218 branches)
So lets skip for GITLAB_CI and save a few CI minutes.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230228190653.1602033-25-alex.bennee@linaro.org>
The two TCG tests for GICv2 and GICv3 are very heavy weight distros
that take a long time to boot up, especially for an --enable-debug
build. The total code coverage they give is:
Overall coverage rate:
lines......: 11.2% (59584 of 530123 lines)
functions..: 15.0% (7436 of 49443 functions)
branches...: 6.3% (19273 of 303933 branches)
We already get pretty close to that with the machine_aarch64_virt
tests which only does one full boot (~120s vs ~600s) of alpine. We
expand the kernel+initrd boot (~8s) to test both GICs and also add an
RNG device and a block device to generate a few IRQs and exercise the
storage layer. With that we get to a coverage of:
Overall coverage rate:
lines......: 11.0% (58121 of 530123 lines)
functions..: 14.9% (7343 of 49443 functions)
branches...: 6.0% (18269 of 303933 branches)
which I feel is close enough given the massive time saving. If we want
to target any more sub-systems we can use lighter weight more directed
tests.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230203181632.2919715-1-alex.bennee@linaro.org
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
We now have a much lighter weight test in machine_aarch64_virt which
tests the full boot chain in less time. Rename the tests while we are
at it to make it clear it is a Fedora cloud image.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20221117172532.538149-11-alex.bennee@linaro.org>
On my machine, a debug build of QEMU takes about 260 seconds to
complete this test, so with the current timeout value of 180 seconds
it always times out. Double the timeout value to 360 so the test
definitely has enough time to complete.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20221110142901.3832318-1-peter.maydell@linaro.org>
Message-Id: <20221117172532.538149-9-alex.bennee@linaro.org>
The two tests
tests/avocado/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv2
tests/avocado/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3
take quite a long time to run, and the current timeout of 240s
is not enough for the tests to complete on slow machines:
we've seen these tests time out in the gitlab CI in the
'avocado-system-alpine' CI job, for instance. The timeout
is also insufficient for running the test with a debug build
of QEMU: on my machine the tests take over 10 minutes to run
in that config.
Push the timeout up to 720s so that the test definitely has
enough time to complete.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
These are timing out on gitlab.
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20221027183637.2772968-7-alex.bennee@linaro.org>
We don't want to rely on the soon to be reduced default time. These
tests are still slow for something we want to run in CI though.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220914155950.804707-6-alex.bennee@linaro.org>
We don't want to rely on the soon to be reduced default time. These
tests are still slow for something we want to run in CI though.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220914155950.804707-5-alex.bennee@linaro.org>
We don't want to rely on the soon to be reduced default time. These
tests are still slow for something we want to run in CI though.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220914155950.804707-4-alex.bennee@linaro.org>
There is a Linux kernel bug present until v5.12 that prevents
booting with FEAT_LPA2 enabled. As a workaround for TCG, allow
the feature to be disabled from -cpu max.
Since this kernel bug is present in the Fedora 31 image that
we test in avocado, disable lpa2 on the command-line.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.
This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.
[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html
Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
