While there isn't any easy way to make the inline counts thread safe
we can ensure the callback based ones are. While we are at it we can
reduce introduce a new option ("idle") to dump a report of the current
bb and insn count each time a vCPU enters the idle state.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Robert Foley <robert.foley@linaro.org>
Cc: Dave Bort <dbort@dbort.com>
Message-Id: <20200713200415.26214-8-alex.bennee@linaro.org>
Not all compilers support the -Wpsabi (clang-9 in my case). To handle
this gracefully we pare back the shared build machinery so the
Makefile is relatively "standalone". We still take advantage of
config-host.mak as configure has done a bunch of probing for us but
that is it.
Fixes: bac8d222a
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200713200415.26214-6-alex.bennee@linaro.org>
This worked on a system that was already bootstrapped because the
stage 2 images already existed even if they wouldn't be used. What we
should have pulled down was the FROM line containers first because
building on gitlab doesn't have the advantage of using our build
system to build the pre-requisite bits.
We still pull the image we want to build just in case we can use the
cached data.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
The libssh problem only exists in Ubuntu 18.04 - we can enable it
in 20.04 again.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713185237.9419-1-thuth@redhat.com>
Message-Id: <20200713200415.26214-3-alex.bennee@linaro.org>
- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAl8NsgMRHGt3b2xmQHJl
ZGhhdC5jb20ACgkQfwmycsiPL9Z0tA//eqauxD7cTEpwrtLNrRtpiBtMG64BBpxz
QfkURzB38bMVahHlwq3Gt7Zcov8V4V7vxK66h688Z/fhw3vmqIeVe8+P6+Y5s9FL
jil8lewHuLTa6xELeugoV7SZXH8AAh1W2fQmiR7EPiOmpSE0wf7C5IShVlX8A04E
r0n09+61qGjRIe1hNTwTtldqQEfx6UGnxQWcQb81JUPA1lZhX3cnPg/j94Bofr+m
v/DbVTfsmUtTMjc0PdU7n4DKTWu8OS5B/X0unF21rTtO//cYBrhAeY3ax2jbFBWi
CIZK8HLI5m9/HFyltql1LOsd+B5TtfnXMfSdvDh2jaVUlto7wTeTnWU1fv4wxUB5
hk7XgJo/y203ebFNHpTmW8tvLfGTP8uqCVfOEFxzjy+JHGrarlbWkwL2LMOFFAZ2
s2WcwlfqiYGFTG4+OFdhPf9qPWKSqMr+jTdZJTse64/c6+YXWHk+pP9lfYEUOgSi
OYwdQUY9uiZ1K13q5Tif2TbFvs+c118xdTgVhAV7VtfPnWc3c647dX7iaq8Szknc
IT93670Iqf/PzEj+L7XUbbLIIsAcmxD0sr7QAQEt7bfiYIDRIQLiVPyzXplETFg2
SEkvtqBovm84ct7pWQzqA6lFvr3oIFDNquR40XFGozHNnlBeNi5s7pXQnqUBLElr
wDDuEi+z5QM=
=DB0q
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads
# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream:
block: Avoid stale pointer dereference in blk_get_aio_context()
qemu-img: Deprecate use of -b without -F
block: Add support to warn on backing file change without format
iotests: Specify explicit backing format where sensible
qcow2: Deprecate use of qemu-img amend to change backing file
block: Error if backing file fails during creation without -u
qcow: Tolerate backing_fmt=
vmdk: Add trivial backing_fmt support
sheepdog: Add trivial backing_fmt support
block: Finish deprecation of 'qemu-img convert -n -o'
qemu-img: Flush stdout before before potential stderr messages
file-posix: Mitigate file fragmentation with extent size hints
iotests/059: Filter out disk size with more standard filter
qemu-img map: Don't limit block status request size
iotests: Simplify _filter_img_create() a bit
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Creating an image that requires format probing of the backing image is
potentially unsafe (we've had several CVEs over the years based on
probes leaking information to the guest on a subsequent boot, although
these days tools like libvirt are aware of the issue enough to prevent
the worst effects). For example, if our probing algorithm ever
changes, or if other tools like libvirt determine a different probe
result than we do, then subsequent use of that backing file under a
different format will present corrupted data to the guest.
Fortunately, the worst effects occur only when the backing image is
originally raw, and we at least prevent commit into a probed raw
backing file that would change its probed type.
Still, it is worth starting a deprecation clock so that future
qemu-img can refuse to create backing chains that would rely on
probing, to encourage clients to avoid unsafe practices. Most
warnings are intentionally emitted from bdrv_img_create() in the block
layer, but qemu-img convert uses bdrv_create() which cannot emit its
own warning without causing spurious warnings on other code paths. In
the end, all command-line image creation or backing file rewriting now
performs a check.
Furthermore, if we probe a backing file as non-raw, then it is safe to
explicitly record that result (rather than relying on future probes);
only where we probe a raw image do we care about further warnings to
the user when using such an image (for example, commits into a
probed-raw backing file are prevented), to help them improve their
tooling. But whether or not we make the probe results explicit, we
still warn the user to remind them to upgrade their workflow to supply
-F always.
iotest 114 specifically wants to create an unsafe image for later
amendment rather than defaulting to our new default of recording a
probed format, so it needs an update. While touching it, expand it to
cover all of the various warnings enabled by this patch. iotest 301
also shows a change to qcow messages.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-11-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well. In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.
There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.
Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Back in commit 6e6e55f5 (Jul 2017, v2.10), we tweaked the code to warn
if the backing file could not be opened but the user gave a size,
unless the user also passes the -u option to bypass the open of the
backing file. As one common reason for failure to open the backing
file is when there is mismatch in the requested backing format in
relation to what the backing file actually contains, we actually want
to open the backing file and ensure that it has the right format in as
many cases as possible. iotest 301 for qcow demonstrates how
detecting explicit format mismatch is useful to prevent the creation
of an image that would probe differently than the user requested. Now
is the time to finally turn the warning an error, as promised.
Note that the original warning was added prior to our documentation of
an official deprecation policy (eb22aeca, also Jul 2017), and because
the warning didn't mention the word "deprecated", we never actually
remembered to document it as such. But the warning has been around
long enough that I don't see prolonging it another two releases.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-7-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qcow has no space in the metadata to store a backing format, and there
are existing qcow images backed both by raw or by other formats
(usually qcow) images, reliant on probing to tell the difference. On
the bright side, because we probe every time, raw files are marked as
probed and we thus forbid a commit action into the backing file where
guest-controlled contents could change the result of the probe next
time around (the iotest added here proves that).
Still, allowing the user to specify the backing format during
creation, even if we can't record it, is a good thing. This patch
blindly allows any value that resolves to a known driver, even if the
user's request is a mismatch from what probing finds; then the next
patch will further enhance things to verify that the user's request
matches what we actually probe. With this and the next patch in
place, we will finally be ready to deprecate the creation of images
where a backing format was not explicitly specified by the user.
Note that this is only for QemuOpts usage; there is no change to the
QAPI to allow a format through -blockdev.
Add a new iotest 301 just for qcow, to demonstrate the latest
behavior, and to make it easier to show the improvements made in the
next patch.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-6-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
It's been two releases since we started warning; time to make the
combination an error as promised. There was no iotest coverage, so
add some.
While touching the documentation, tweak another section heading for
consistent style.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-3-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
During 'qemu-img create ... 2>&1', if --quiet is not in force, we can
end up with buffered I/O in stdout that was produced before failure,
but which appears in output after failure. This is confusing; the fix
is to flush stdout prior to attempting anything that might produce an
error message. Several iotests demonstrate the resulting ordering
change now that the merged outputs now reflect chronology. (An even
better fix would be to avoid printf from within block.c altogether,
but that's much more invasive...)
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-2-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The actual disk space used by an image can vary between filesystems and
depending on other settings like an extent size hint. Replace the one
call of "$QEMU_IMG info" and the associated one-off sed filter with the
more standard "_img_info" and the standard filter from common.filter.
Apart from turning "vmdk" into "IMGFMT" and changing the placeholder for
cid fields, this only removes the "disk size" line.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Not only is it a bit stupid to try to filter multi-line "Formatting"
output (because we only need it for a single test, which can easily be
amended to no longer need it), it is also problematic when there can be
output after a "Formatting" line that we do not want to filter as if it
were part of it.
So rename _filter_img_create to _do_filter_img_create, let it filter
only a single line, and let _filter_img_create loop over all input
lines, calling _do_filter_img_create only on those that match
/^Formatting/ (basically, what _filter_img_create_in_qmp did already).
(And fix 020 to work with that.)
Reported-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200709110205.310942-1-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200701105331.121670-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Make it simpler to debug when qemu-io fails due to wrong arguments or
environment.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200701105331.121670-3-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
The only user (iotest 205) of QemuIoInteractive provides -f argument,
so it's a bit inefficient to use qemu_io_args, which contains -f too.
And we are going to add one more test, which wants to specify -f by
hand. Let's use qemu_io_args_no_fmt.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200701105331.121670-2-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
There should be a space between "forking" and "for".
Message-Id: <20200709083719.22221-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Signed-off-by: Thomas Huth <thuth@redhat.com>
In 45222b9a90, I fixed a broken check for rcu_enable_atfork introduced
in d6919e4cb6. I added a call to rcu_enable_atfork after the
call to qemu_init in fuzz.c, but forgot to include the corresponding
header, breaking --enable-fuzzing --enable-werror builds.
Fixes: 45222b9a90 ("fuzz: fix broken qtest check at rcu_disable_atfork")
Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <20200708200104.21978-3-alxndr@bu.edu>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This test never required "chardev/char-mux.h", remove it.
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200423202112.644-3-philmd@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
With a reconnect socket, qemu_char_open() will start a background
thread. It should keep a reference on the chardev.
Fixes invalid read:
READ of size 8 at 0x6040000ac858 thread T7
#0 0x5555598d37b8 in unix_connect_saddr /home/elmarco/src/qq/util/qemu-sockets.c:954
#1 0x5555598d4751 in socket_connect /home/elmarco/src/qq/util/qemu-sockets.c:1109
#2 0x555559707c34 in qio_channel_socket_connect_sync /home/elmarco/src/qq/io/channel-socket.c:145
#3 0x5555596adebb in tcp_chr_connect_client_task /home/elmarco/src/qq/chardev/char-socket.c:1104
#4 0x555559723d55 in qio_task_thread_worker /home/elmarco/src/qq/io/task.c:123
#5 0x5555598a6731 in qemu_thread_start /home/elmarco/src/qq/util/qemu-thread-posix.c:519
#6 0x7ffff40d4431 in start_thread (/lib64/libpthread.so.0+0x9431)
#7 0x7ffff40029d2 in __clone (/lib64/libc.so.6+0x1019d2)
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200420112012.567284-1-marcandre.lureau@redhat.com>
When the disconnect event is triggered in the connecting stage,
the tcp_chr_disconnect_locked may be called twice.
The first call:
#0 qemu_chr_socket_restart_timer (chr=0x55555582ee90) at chardev/char-socket.c:120
#1 0x000055555558e38c in tcp_chr_disconnect_locked (chr=<optimized out>) at chardev/char-socket.c:490
#2 0x000055555558e3cd in tcp_chr_disconnect (chr=0x55555582ee90) at chardev/char-socket.c:497
#3 0x000055555558ea32 in tcp_chr_new_client (chr=chr@entry=0x55555582ee90, sioc=sioc@entry=0x55555582f0b0) at chardev/char-socket.c:892
#4 0x000055555558eeb8 in qemu_chr_socket_connected (task=0x55555582f300, opaque=<optimized out>) at chardev/char-socket.c:1090
#5 0x0000555555574352 in qio_task_complete (task=task@entry=0x55555582f300) at io/task.c:196
#6 0x00005555555745f4 in qio_task_thread_result (opaque=0x55555582f300) at io/task.c:111
#7 qio_task_wait_thread (task=0x55555582f300) at io/task.c:190
#8 0x000055555558f17e in tcp_chr_wait_connected (chr=0x55555582ee90, errp=0x555555802a08 <error_abort>) at chardev/char-socket.c:1013
#9 0x0000555555567cbd in char_socket_client_reconnect_test (opaque=0x5555557fe020 <client8unix>) at tests/test-char.c:1152
The second call:
#0 0x00007ffff5ac3277 in raise () from /lib64/libc.so.6
#1 0x00007ffff5ac4968 in abort () from /lib64/libc.so.6
#2 0x00007ffff5abc096 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007ffff5abc142 in __assert_fail () from /lib64/libc.so.6
#4 0x000055555558d10a in qemu_chr_socket_restart_timer (chr=0x55555582ee90) at chardev/char-socket.c:125
#5 0x000055555558df0c in tcp_chr_disconnect_locked (chr=<optimized out>) at chardev/char-socket.c:490
#6 0x000055555558df4d in tcp_chr_disconnect (chr=0x55555582ee90) at chardev/char-socket.c:497
#7 0x000055555558e5b2 in tcp_chr_new_client (chr=chr@entry=0x55555582ee90, sioc=sioc@entry=0x55555582f0b0) at chardev/char-socket.c:892
#8 0x000055555558e93a in tcp_chr_connect_client_sync (chr=chr@entry=0x55555582ee90, errp=errp@entry=0x7fffffffd178) at chardev/char-socket.c:944
#9 0x000055555558ec78 in tcp_chr_wait_connected (chr=0x55555582ee90, errp=0x555555802a08 <error_abort>) at chardev/char-socket.c:1035
#10 0x000055555556804b in char_socket_client_test (opaque=0x5555557fe020 <client8unix>) at tests/test-char.c:1023
Run test/test-char to reproduce this issue.
test-char: chardev/char-socket.c:125: qemu_chr_socket_restart_timer: Assertion `!s->reconnect_timer' failed.
Signed-off-by: Li Feng <fengli@smartx.com>
Acked-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200522025554.41063-1-fengli@smartx.com>
- tests/vm support for aarch64 VMs
- tests/tcg better cross-compiler detection
- update docker tooling to support registries
- update docker support for xtensa
- gitlab build docker images and store in registry
- gitlab use docker images for builds
- a number of skipIf updates to support move
- linux-user MAP_FIXED_NOREPLACE fix
- qht-bench compiler tweaks
- configure fix for secret keyring
- tsan fiber annotation clean-up
- doc updates for mttcg/icount/gdbstub
- fix cirrus to use brew bash for iotests
- revert virtio-gpu breakage
- fix LC_ALL to avoid sorting changes in iotests
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAl8J0yoACgkQ+9DbCVqe
KkSzTAf/Vn+9TU8Qt7nZvl7W4tz7Sy5K8EJGwj2RXx6CWWWLiFbsXurIM8Krw5Vc
RmvUxwa359b+J0lQpfeNDHYm1nM8RZLFlkG0a5bl0I8sW0EcPjBRtwNaGKXh2p0u
u2RS2QAi6A9AvYT4ZREYlBM+o9WzbxCEQm4s8fr6WEJCQfxBnb5/bGiEjWR64e8C
j9Kvou+zAKfVizbQMtu+mwqjsoPtcS1b3vVcO7anhNuUsuaEKkS0dFWzWvw3lwJR
STIYnb8Y/eJ1yKr0hPH2qtWv3n6yhlYvYmpUCH6AwshGMUoeFEzR2VoWS6yZPGG6
na6XA3UW5R9AxIDfkCJ5ueeo8t9xMQ==
=HRWa
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-and-misc-110720-2' into staging
Testing and misc build updates:
- tests/vm support for aarch64 VMs
- tests/tcg better cross-compiler detection
- update docker tooling to support registries
- update docker support for xtensa
- gitlab build docker images and store in registry
- gitlab use docker images for builds
- a number of skipIf updates to support move
- linux-user MAP_FIXED_NOREPLACE fix
- qht-bench compiler tweaks
- configure fix for secret keyring
- tsan fiber annotation clean-up
- doc updates for mttcg/icount/gdbstub
- fix cirrus to use brew bash for iotests
- revert virtio-gpu breakage
- fix LC_ALL to avoid sorting changes in iotests
# gpg: Signature made Sat 11 Jul 2020 15:56:42 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-testing-and-misc-110720-2: (50 commits)
iotests: Set LC_ALL=C for sort
Revert "vga: build virtio-gpu as module"
tests: fix "make check-qtest" for modular builds
.cirrus.yml: add bash to the brew packages
tests/docker: update toolchain set in debian-xtensa-cross
tests/docker: fall back more gracefully when pull fails
docs: Add to gdbstub documentation the PhyMemMode
docs/devel: add some notes on tcg-icount for developers
docs/devel: convert and update MTTCG design document
tests/qht-bench: Adjust threshold computation
tests/qht-bench: Adjust testing rate by -1
travis.yml: Test also the other targets on s390x
shippable: pull images from registry instead of building
testing: add check-build target
containers.yml: build with docker.py tooling
gitlab: limit re-builds of the containers
tests: improve performance of device-introspect-test
gitlab: add avocado asset caching
gitlab: enable check-tcg for linux-user tests
linux-user/elfload: use MAP_FIXED_NOREPLACE in pgb_reserved_va
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Michael started to work on the AVR port few years ago [*] and kept
improving the code over various series.
List of people who help him (in chronological order):
- Richard Henderson
- Sarah Harris and Edward Robbins
- Philippe Mathieu-Daudé and Aleksandar Markovic
- Pavel Dovgalyuk
- Thomas Huth
[*] The oldest contribution I could find on the list is from 2016:
https://lists.nongnu.org/archive/html/qemu-devel/2016-06/msg02985.html
Tests included:
$ avocado --show=app run -t arch:avr tests/acceptance/
Fetching asset from tests/acceptance/machine_avr6.py:AVR6Machine.test_freertos
(1/1) tests/acceptance/machine_avr6.py:AVR6Machine.test_freertos: PASS (2.13 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 2.35 s
$ make check-qtest-avr
TEST check-qtest-avr: tests/qtest/boot-serial-test
TEST check-qtest-avr: tests/qtest/cdrom-test
TEST check-qtest-avr: tests/qtest/device-introspect-test
TEST check-qtest-avr: tests/qtest/machine-none-test
TEST check-qtest-avr: tests/qtest/qmp-test
TEST check-qtest-avr: tests/qtest/qmp-cmd-test
TEST check-qtest-avr: tests/qtest/qom-test
TEST check-qtest-avr: tests/qtest/test-hmp
TEST check-qtest-avr: tests/qtest/qos-test
CI results:
. https://cirrus-ci.com/build/5697049146425344
. https://gitlab.com/philmd/qemu/-/pipelines/165328058
. https://travis-ci.org/github/philmd/qemu/builds/705817933
. https://app.shippable.com/github/philmd/qemu/runs/822/summary/console
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE+qvnXhKRciHc/Wuy4+MsLN6twN4FAl8JgE8ACgkQ4+MsLN6t
wN5KMhAA1IivMK9uD9x8vOK3H1fSIju6ufJz7mlynDXG/LV0dhms1t8n/AlPE/Gk
TLhBUBwwdsejVDrTu6GGukoybsKLyULdt8MIK5z6bd++TwY3MlUkdOJeteviBUuP
0hcagNR/Dyg1WLAq/VH4KGHfzisprfDM7sXTvjE3raKBSpqIwO5tfUn4kVm/LWB9
sQNsVbtyKqnM3UW+QYGAN3eGAOM2SWx8pUZqV+UrDDEDoFJIiip7jxoN1t7PJaQp
O6t2/omLzbOMrpwqmNAIfrsMjovRylrd8nDGlX/OF5SrEbwXi3qvdJBtEOdBPasp
owXbu2Uwo4VUu5x7kzAiTlflBBSOmDpILbYVn5jGLKMZmOjLTPVbrzMKGZqx5GXD
gWmmX2aD8ejl2XGmKM+gC1smQJ6/aMTILoYXq97hIKi8pMH7AB2a8Tmzseiqx/E/
Lz1DrrnIW5vwFQAPnhdJCU1GF3B9VUcHG3w0sjvgGKDfpe8tLEgkmISi7CUbbA9/
rJs2P24haqfdFXWLQU2sO1ygTR1vLNy5/ZbU1nyrAPpjWnGeX6GXVaWGQo83BaDd
rIfnx0upNYFdaO5Vi4cbHUFGe1fVuR3C/l1xiDbmnzx3yMgys3036Equ2h3fSPqO
4k79bee9ByAUl1YE0X8T4gasCRS6RaSWXZYz7lfdc9h5hGny8LM=
=gFE5
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/philmd-gitlab/tags/avr-port-20200711' into staging
8bit AVR port from Michael Rolnik.
Michael started to work on the AVR port few years ago [*] and kept
improving the code over various series.
List of people who help him (in chronological order):
- Richard Henderson
- Sarah Harris and Edward Robbins
- Philippe Mathieu-Daudé and Aleksandar Markovic
- Pavel Dovgalyuk
- Thomas Huth
[*] The oldest contribution I could find on the list is from 2016:
https://lists.nongnu.org/archive/html/qemu-devel/2016-06/msg02985.html
Tests included:
$ avocado --show=app run -t arch:avr tests/acceptance/
Fetching asset from tests/acceptance/machine_avr6.py:AVR6Machine.test_freertos
(1/1) tests/acceptance/machine_avr6.py:AVR6Machine.test_freertos: PASS (2.13 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 2.35 s
$ make check-qtest-avr
TEST check-qtest-avr: tests/qtest/boot-serial-test
TEST check-qtest-avr: tests/qtest/cdrom-test
TEST check-qtest-avr: tests/qtest/device-introspect-test
TEST check-qtest-avr: tests/qtest/machine-none-test
TEST check-qtest-avr: tests/qtest/qmp-test
TEST check-qtest-avr: tests/qtest/qmp-cmd-test
TEST check-qtest-avr: tests/qtest/qom-test
TEST check-qtest-avr: tests/qtest/test-hmp
TEST check-qtest-avr: tests/qtest/qos-test
CI results:
. https://cirrus-ci.com/build/5697049146425344
. https://gitlab.com/philmd/qemu/-/pipelines/165328058
. https://travis-ci.org/github/philmd/qemu/builds/705817933
. https://app.shippable.com/github/philmd/qemu/runs/822/summary/console
# gpg: Signature made Sat 11 Jul 2020 10:03:11 BST
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
* remotes/philmd-gitlab/tags/avr-port-20200711: (32 commits)
target/avr/disas: Fix store instructions display order
target/avr/cpu: Fix $PC displayed address
target/avr/cpu: Drop tlb_flush() in avr_cpu_reset()
target/avr: Add section into QEMU documentation
tests/acceptance: Test the Arduino MEGA2560 board
tests/boot-serial: Test some Arduino boards (AVR based)
hw/avr: Add limited support for some Arduino boards
hw/avr: Add some ATmega microcontrollers
hw/avr: Add support for loading ELF/raw binaries
hw/misc: avr: Add limited support for power reduction device
hw/timer: avr: Add limited support for 16-bit timer peripheral
hw/char: avr: Add limited support for USART peripheral
tests/machine-none: Add AVR support
target/avr: Register AVR support with the rest of QEMU
target/avr: Add support for disassembling via option '-d in_asm'
target/avr: Initialize TCG register variables
target/avr: Add instruction translation - CPU main translation function
target/avr: Add instruction translation - MCU Control Instructions
target/avr: Add instruction translation - Bit and Bit-test Instructions
target/avr: Add instruction translation - Data Transfer Instructions
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Otherwise the result is basically unpredictable.
(Note that the precise environment variable to control sorting order is
LC_COLLATE, but LC_ALL overrides LC_COLLATE, and we do not want the
sorting order to be messed up if LC_ALL is set in the environment.)
Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200710163253.381630-3-mreitz@redhat.com>
Switch to the prebuilt xtensa toolchains release 2020.07.
Drop csp toolchain as the csp core is not a part of QEMU.
Add de233_fpu and dsp3400 toolchains to enable DFPU and FPU2000 tests.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: fix path in configure.sh]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200708082347.27318-1-jcmvbkbc@gmail.com>
Message-Id: <20200709141327.14631-13-alex.bennee@linaro.org>
I only spotted this in the small window between my testing with my
registry while waiting for the gitlab PR to go in. As we pre-pull the
registry image we know if that fails there isn't any point attempting
to use the cache. Fall back to the way we used to do it at that point.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200709141327.14631-12-alex.bennee@linaro.org>
In 06c4cc3660, we split the multiplication in two parts to avoid
a clang warning. But because double still rounds to 53 bits, this
does not provide additional precision beyond multiplication by
nextafter(0x1p64, 0), the largest representable value smaller
than 2**64.
However, since we have eliminated 1.0, mutiplying by 2**64 produces
a better distribution of input values to the output values.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200626200950.1015121-3-richard.henderson@linaro.org>
Since the seed must be non-zero, subtracting 1 means puts the
rate in 0..UINT64_MAX-1, which allows the 0 and UINT64_MAX
thresholds to corrspond to 0% (never) and 100% (always).
Suggested-by: Emilio G. Cota <cota@braap.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200626200950.1015121-2-richard.henderson@linaro.org>
If we want to continue to split build and check phase it seems like a
good idea to allow building of the tests during our multi-threaded
build phase.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200701135652.1366-40-alex.bennee@linaro.org>
Total execution time with "-m slow" and x86_64 QEMU, drops from 3
minutes 15 seconds, down to 54 seconds.
Individual tests drop from 17-20 seconds, down to 3-4 seconds.
The cost of this change is that any QOM bugs resulting in the test
failure will not be directly associated with the device that caused
the failure. The test case is not frequently identifying such bugs
though, and the cause is likely easily visible in the patch series
that causes the failure. So overall the shorter running time is
considered the more important factor.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
[thuth: Add the tree check to test_device_intro_none() and
test_device_intro_abstract(), too, just to be sure...]
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200710060719.22386-1-thuth@redhat.com>
We happily use all the cross images for both cross-building QEMU as
well as building the linux-user tests. However calling docker from
within docker seems not to work. As we can build in Debian anyway why
not include an image that has all the compilers available for
non-docker invocation.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-33-alex.bennee@linaro.org>
We were missing a bunch of compilers which we could use if they were
locally installed. The defaults are based on Debian as they seem to be
the best distro for well distributed cross-build compilers.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200701135652.1366-32-alex.bennee@linaro.org>
As part of migrating things from Travis to GitLab add the acceptance
tests. To do this:
- rename system1 to system-ubuntu-main
- rename system2 to system-fedora-misc
- split into build/check/acceptance
- remove -j from check stages
- use artifacts to save build stage
- add post acceptance template and use
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-31-alex.bennee@linaro.org>
This fails on GitLab but not when run locally on the same container
image. It's very confusing.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200701135652.1366-30-alex.bennee@linaro.org>
For some reason these tests fail all the time on GitLab. I can
re-create the hang around 3% of the time locally but it doesn't seem
to be MTTCG related. For now skipIf on GITLAB_CI.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
Message-Id: <20200701135652.1366-29-alex.bennee@linaro.org>
The old path doesn't exist but the rx-virt.dtb file has the same
checksum so lets use that.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200701135652.1366-28-alex.bennee@linaro.org>
Currently the test takes more the 900 seconds on GitLab and then times
out. Running on Travis seems to be OK.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200701135652.1366-27-alex.bennee@linaro.org>
We need additional python packages to run check-acceptance. Add them
to the docker images we will be using later.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200701135652.1366-26-alex.bennee@linaro.org>
This allows us to point the tools towards a registry from which they
can grab pre-built layers instead of doing everything from scratch
each time. To enable this we need to be using the DOCKER_BUILDKIT
engine.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200701135652.1366-25-alex.bennee@linaro.org>
We've been misusing the tag naming scheme for some time by overloading
the post : section with the image type. Really it should be saved for
the revision of that particular build. Move the details to the other
side so we have:
qemu/image-name
with the implied :latest version added by the tooling.
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-18-alex.bennee@linaro.org>
Reported-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-17-alex.bennee@linaro.org>
We currently limit TCG guests to -smp 1 but now we have added some
aarch64 guests we can do better when running on x86_64 hardware.
Raise the limit for TCG guests when it is safe to do so.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-16-alex.bennee@linaro.org>
optparse has been deprecated since version 3.2 and argparse is the
blessed replacement. Take the opportunity to enhance our help output
showing defaults when called.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Robert Foley <robert.foley@linaro.org>
Message-Id: <20200701135652.1366-15-alex.bennee@linaro.org>
This adds support to basevm.py so that we always
drain the console chars. This makes use of
support added in an earlier commit that allows
QEMUMachine to use the ConsoleSocket.
This is a workaround we found was needed since
there is a known issue where QEMU will hang waiting
for console characters to be consumed.
We also added the option of logging the console to a file.
LOG_CONSOLE=1 will now log the output to a file.
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200601211421.1277-10-robert.foley@linaro.org>
Message-Id: <20200701135652.1366-14-alex.bennee@linaro.org>
This change converts existing scripts to using for example self.ROOT_PASS,
to self._config['root_pass'].
We made similar changes for GUEST_USER, and GUEST_PASS.
This allows us also to remove the change in basevm.py,
which adds __getattr__ for backwards compatibility.
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200601211421.1277-8-robert.foley@linaro.org>
Message-Id: <20200701135652.1366-12-alex.bennee@linaro.org>
