One problem with flaky tests is they often only fail under CI
conditions which makes it hard to debug. We add an optional allow_fail
job so developers can trigger the only the flaky tests in the CI
environment if they are debugging.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-8-alex.bennee@linaro.org>
It doesn't make sense to have two classes of flaky tests. While it may
take the constrained environment of CI to trigger failures easily it
doesn't mean they don't occasionally happen on developer machines. As
CI is the gating factor to passing there is no point developers
running the tests locally anyway unless they are trying to fix things.
While we are at it update the language in the docs to discourage the
QEMU_TEST_FLAKY_TESTS becoming a permanent solution.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
Add smoke tests to ensure that we'll not break the 'max' CPU type when
adding new frozen/ratified RISC-V extensions.
Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230912132423.268494-12-dbarboza@ventanamicro.com>
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Commit 18a536f1f8 ("accel/tcg: Always require can_do_io") fixed
the GitLab issue #1884: we can now re-enable those tests.
This reverts commit f959c3d87c.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231003063808.66564-1-philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Commit 0d58c66068 ("softmmu: Use async_run_on_cpu in tcg_commit")
introduced a regression which is only triggered by the MIPS Malta
machine. Since those tests are gatting and disturb the CI workflow,
disable them until https://gitlab.com/qemu-project/qemu/-/issues/1866
is fixed.
$ make check-avocado \
AVOCADO_TAGS='arch:mipsel arch:mips64el' \
AVOCADO_ALLOW_UNTRUSTED_CODE=1 \
AVOCADO_TIMEOUT_EXPECTED=1
AVOCADO tests/avocado
(04/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_4k: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (90.39 s)
(05/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_16k_up: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (90.29 s)
(06/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_64k_dbg: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (92.53 s)
(11/24) tests/avocado/machine_mips_malta.py:MaltaMachineFramebuffer.test_mips_malta_i6400_framebuffer_logo_1core: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (25.78 s)
RESULTS : PASS 8 | ERROR 0 | FAIL 0 | SKIP 7 | WARN 2 | INTERRUPT 5 | CANCEL 2
JOB TIME : 525.60 s ^^^^^^^^^^^
Reported-by: Thomas Huth <thuth@redhat.com>
Suggested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230913135339.9128-1-philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230914155422.426639-10-alex.bennee@linaro.org>
The ppc64 tuxrun tests are currently failing if "slirp" has been
disabled in the binary since they are using "-netdev user" now.
We have to skip the test if this network backend is missing.
Fixes: 6ee3624236 ("improve code coverage for ppc64")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230606192802.666000-1-thuth@redhat.com>
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Commit c0c8687ef0 disabled the
boot_linux.py test-case due to which the code coverage for ppc
decreased by around 2%. As per the discussion on
https://lore.kernel.org/qemu-devel/87sfdpqcy4.fsf@linaro.org/ it
was mentioned that the baseline test for ppc64 could be modified
to make up this 2% code coverage. This patch attempts to achieve
this 2% code coverage by adding various device command line
arguments (to ./qemu-system-ppc64) in the tuxrun_baselines.py
test-case.
The code coverage report with boot_linux.py, without it and finally
with these tuxrun_baselines.py changes is as follows:
With boot_linux.py
------------------
lines......: 13.8% (58006 of 420997 lines)
functions..: 20.7% (7675 of 36993 functions)
branches...: 9.2% (22146 of 240611 branches)
Without boot_linux.py (without this patch changes)
--------------------------------------------------
lines......: 11.9% (50174 of 420997 lines)
functions..: 18.8% (6947 of 36993 functions)
branches...: 7.4% (17580 of 239017 branches)
Without boot_linux.py (with this patch changes)
-----------------------------------------------
lines......: 13.8% (58287 of 420997 lines)
functions..: 20.7% (7640 of 36993 functions)
branches...: 8.4% (20223 of 240611 branches)
Rebased on Alex Benee's testing/next branch:
https://gitlab.com/stsquad/qemu/-/tree/testing/next
Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Reported-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230424041830.1275636-1-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-13-alex.bennee@linaro.org>
The tuxboot images now have a stable snapshot URL so we can enable the
checksums and remove the avocado warnings. We will have to update as
old snapshots retire but that won't be too frequent.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230424092249.58552-3-alex.bennee@linaro.org>
Now that the previous commit ("hw/mips/gt64xxx_pci: Don't
endian-swap GT_PCI0_CFGADDR") fixed the issue accessing
the GT64120 PCI config-address register on big-endian
targets, we can enable this TuxRun test.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230223220404.63630-1-philmd@linaro.org>
We are abusing the avocado tags which are intended to provide test
selection metadata to provide parameters to our test. This works OK up
until the point you need to have ,'s in the field as this is the tag
separator character which is the case for a number of the drive
parameters. Fix this by making drive a parameter to the common helper
function.
Fixes: 267fe57c23 (tests: add tuxrun baseline test to avocado)
Reviewed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230315174331.2959-11-alex.bennee@linaro.org>
The TuxRun project (www.tuxrun.org) uses QEMU to run tests on a wide
variety of kernel configurations on wide range of our emulated
platforms. They publish a known good set of images at:
https://storage.tuxboot.com/
to help with bisecting regressions in either the kernel, firmware or
QEMU itself. The tests are pretty lightweight as they contain just a
kernel with a minimal rootfs which boots a lot faster than most of the
distros. In time they might be persuaded to version their known good
baselines and we can then enable proper checksums.
For a couple of tests we currently skip:
- mips64, a regression against previous stable release
- sh4, very unstable with intermittent oops
Total run time: 340s (default) -> 890s (debug)
Overall coverage rate (tested targets + disabled tests):
lines......: 16.1% (126894 of 789848 lines)
functions..: 20.6% (15954 of 77489 functions)
branches...: 9.3% (40727 of 439365 branches)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Anders Roxell <anders.roxell@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230228190653.1602033-11-alex.bennee@linaro.org>
