Commit Graph

71265 Commits

Author SHA1 Message Date
Max Reitz
5a840549ee iotests: Convert to preallocated encrypted qcow2
Add a test case for converting an empty image (which only returns zeroes
when read) to a preallocated encrypted qcow2 image.
qcow2_has_zero_init() should return 0 then, thus forcing qemu-img
convert to create zero clusters.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Acked-by: Stefano Garzarella <sgarzare@redhat.com>
Tested-by: Stefano Garzarella <sgarzare@redhat.com>
Message-id: 20190724171239.8764-10-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
9956688a8f vhdx: Fix .bdrv_has_zero_init()
Fixed VHDX images cannot guarantee to be zero-initialized.  If the image
has the "fixed" subformat, forward the call to the underlying storage
node.

Reported-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-9-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
0a28bf2826 vdi: Fix .bdrv_has_zero_init()
Static VDI images cannot guarantee to be zero-initialized.  If the image
has been statically allocated, forward the call to the underlying
storage node.

Reported-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Weil <sw@weilnetz.de>
Acked-by: Stefano Garzarella <sgarzare@redhat.com>
Tested-by: Stefano Garzarella <sgarzare@redhat.com>
Message-id: 20190724171239.8764-8-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
38841dcd27 qcow2: Fix .bdrv_has_zero_init()
If a qcow2 file is preallocated, it can no longer guarantee that it
initially appears as filled with zeroes.

So implement .bdrv_has_zero_init() by checking whether the file is
preallocated; if so, forward the call to the underlying storage node,
except for when it is encrypted: Encrypted preallocated images always
return effectively random data, so .bdrv_has_zero_init() must always
return 0 for them.

.bdrv_has_zero_init_truncate() can remain bdrv_has_zero_init_1(),
because it presupposes PREALLOC_MODE_OFF.

Reported-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-7-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
b647d69adc block: Use bdrv_has_zero_init_truncate()
vhdx and parallels call bdrv_has_zero_init() when they do not really
care about an image's post-create state but only about what happens when
you grow an image.  That is a bit ugly, and also overly safe when
growing preallocated images without preallocating the new areas.

Let them use bdrv_has_zero_init_truncate() instead.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-6-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
[mreitz: Added commit message]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
1dcaf52760 block: Implement .bdrv_has_zero_init_truncate()
We need to implement .bdrv_has_zero_init_truncate() for every block
driver that supports truncation and has a .bdrv_has_zero_init()
implementation.

Implement it the same way each driver implements .bdrv_has_zero_init().
This is at least not any more unsafe than what we had before.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-5-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
ceaca56fee block: Add bdrv_has_zero_init_truncate()
No .bdrv_has_zero_init() implementation returns 1 if growing the file
would add non-zero areas (at least with PREALLOC_MODE_OFF), so using it
in lieu of this new function was always safe.

But on the other hand, it is possible that growing an image that is not
zero-initialized would still add a zero-initialized area, like when
using nonpreallocating truncation on a preallocated image.  For callers
that care only about truncation, not about creation with potential
preallocation, this new function is useful.

Alternatively, we could have added a PreallocMode parameter to
bdrv_has_zero_init().  But the only user would have been qemu-img
convert, which does not have a plain PreallocMode value right now -- it
would have to parse the creation option to obtain it.  Therefore, the
simpler solution is to let bdrv_has_zero_init() inquire the
preallocation status and add the new bdrv_has_zero_init_truncate() that
presupposes PREALLOC_MODE_OFF.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-4-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
cdf3bc934a mirror: Fix bdrv_has_zero_init() use
bdrv_has_zero_init() only has meaning for newly created images or image
areas.  If the mirror job itself did not create the image, it cannot
rely on bdrv_has_zero_init()'s result to carry any meaning.

This is the case for drive-mirror with mode=existing and always for
blockdev-mirror.

Note that we only have to zero-initialize the target with sync=full,
because other modes actually do not promise that the target will contain
the same data as the source after the job -- sync=top only promises to
copy anything allocated in the top layer, and sync=none will only copy
new I/O.  (Which is how mirror has always handled it.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-3-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Max Reitz
4d7c487eac qemu-img: Fix bdrv_has_zero_init() use in convert
bdrv_has_zero_init() only has meaning for newly created images or image
areas.  If qemu-img convert did not create the image itself, it cannot
rely on bdrv_has_zero_init()'s result to carry any meaning.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-2-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Maxim Levitsky
672de729a1 LUKS: support preallocation
preallocation=off and preallocation=metadata
both allocate luks header only, and preallocation=falloc/full
is passed to underlying file.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1534951

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-id: 20190716161901.1430-1-mlevitsk@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-08-19 17:13:26 +02:00
Richard Henderson
59a3a1c0c2 target/riscv: Remove redundant declaration pragmas
These are now generated by decodetree itself.

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Acked-by: Palmer Dabbelt <palmer@sifive.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 08:13:14 -07:00
Richard Henderson
c692079597 decodetree: Suppress redundant declaration warnings
We can tell that a decodetree input file is "secondary" when it
uses an argument set marked "!extern".  This indicates that at
least one of the insn translation functions will have already
been declared by the "primary" input file, but given only the
secondary we cannot tell which.

Avoid redundant declaration warnings by suppressing them with pragmas.

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 08:13:14 -07:00
Richard Henderson
94597b6146 decodetree: Allow !function with no input bits
Call this form a "parameter", returning a value extracted
from the DisasContext.

Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 08:03:41 -07:00
Peter Maydell
50d69ee0d8 Softfloat updates
- minor refactoring of constants
   - drop LIT64 macro
   - re-organise header inclusion
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAl1agzUACgkQ+9DbCVqe
 KkTg3gf/VuIspVQc7v4hzrOVhXSzt0IKEnbvXu1DQd9zcOltfNPRbT1dvOmWmRWL
 +qKEWATUneM/nmCPtQE/XYjKbQJ8/w0QmvcD17IPDTcbVXLrPSbb6CK5rgs5W27K
 kQ0NMIi47iJnGxmcixVntHbkcPHWmhMgj6FYk0XNdJQnJvgyDo/wL7Bct+GSUp+/
 7EMQ5QrQ6qK1vVHo+zAPEB8vXyP+tQxMyCKk7arwezeI7wWJePcqMcGNgq6R7VYU
 4cRwMr/eaE4QxfpEVvweNGK8L4fSm//KrJFUu3VaugmDxR/T0eRtH3mGT4rPi6YS
 6MCdkuxqsJRCEgX5svQ6qVmuggu8hQ==
 =7Ke7
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-softfloat-headers-190819-1' into staging

Softfloat updates

  - minor refactoring of constants
  - drop LIT64 macro
  - re-organise header inclusion

# gpg: Signature made Mon 19 Aug 2019 12:08:37 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-softfloat-headers-190819-1:
  targets (various): use softfloat-helpers.h where we can
  target/riscv: rationalise softfloat includes
  target/mips: rationalise softfloat includes
  fpu: rename softfloat-specialize.h -> .inc.c
  fpu: make softfloat-macros "self-contained"
  fpu: move inline helpers into a separate header
  fpu: remove the LIT64 macro
  target/m68k: replace LIT64 with UINT64_C macros
  fpu: replace LIT64 with UINT64_C macros
  fpu: use min/max values from stdint.h for integral overflow
  fpu: convert float[16/32/64]_squash_denormal to new modern style
  fpu: replace LIT64 usage with UINT64_C for specialize constants

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-08-19 15:58:01 +01:00
Peter Maydell
3fbd3405d2 - Run the iotest during "make check"
-----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAl1XvtURHHRodXRoQHJl
 ZGhhdC5jb20ACgkQLtnXdP5wLbUqqRAAqfjPRPtXSZxaxl63O5EADugLY72s+04i
 Zc5MC4Ivwj0x1WA6JIG77kz6xmjIOizRIkF1jkyEkG+AgIrjw3rdYhCD4Iav0n/v
 nqltkOaf1FzdQYCHUTn0WUYn7Df2bSkjSTPhnqbCaGq5WjXzgzi9jhCFlZpo374J
 9yWk74nt3QlOUjLw6+cm0HxEf9IlRQdPwJNYJwsrYHgspJwcwAYJ7xaL34huoxkJ
 10fA9q6QK1bh67nZpAJOte3wQ8r35cUT4ZaIiyO0MFMrEiLp4/1gKYpkZwWq0+iV
 25rWVjogzRjp+LejMAltY9MmUekCl5ZzVVuhdt2jGPbNanzdHxHydYFUELP6WmrM
 zAyYQkDvG7JiY2F0M9rKJZOnpO2pYF2hxc/nqD04qF3HD0zG1eoIg056UlKcc5b/
 kIgR2srlj0aHWhGJ2/DV3w5ZowjJGKBAYYxQdEJmiuLpGdOimSoWXbjacDEd4mUf
 DevXv6k9cAvexZxU4cOUzpip4U3MGC0rJ1BNIgTs6eIeKq3geROTpuHjJFSBHZiP
 H/MqtoT4xTXJYomc9MiZG90fG9KyywlEF6e0GjVIcadJEmFIbJ+DfrAknVRjuaij
 ThMSIuvMEpXFhyghlApURePNi8W3FIIYHISw0JE/u7+4/7L/iYDToYeM/o563+8O
 zbj0n9fSewI=
 =oVZx
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2019-08-17' into staging

- Run the iotest during "make check"

# gpg: Signature made Sat 17 Aug 2019 09:46:13 BST
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* remotes/huth-gitlab/tags/pull-request-2019-08-17:
  gitlab-ci: Remove qcow2 tests that are handled by "make check" already
  tests: Run the iotests during "make check" again
  block: fix NetBSD qemu-iotests failure

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-08-19 14:14:09 +01:00
Peter Maydell
9f3a972e22 Pull request
Stable notes: patches one and two can be considered
               for the next -stable release.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+ber27ys35W+dsvQfe+BBqr8OQ4FAl1XOSYACgkQfe+BBqr8
 OQ5WYQ//W3H3UsXHXk1GNM2O10rVUWnFwF6GHnK14csGPBe/bzPokQEdbJtLHaMF
 6oyoANQ994LPBHODthm662IUMs1mZGQ0HlHaeNF8MFlwofznJKJT1aXXmvqHy8VC
 lU8HiYMpdDhjc0hn3zc3iGhdkUmm2axvViZb91rqtFrsZRkzfxhuowjr0RtOQsbD
 mp40FGmb/KLkT41gVHK2E/SR7H5MqzKXrUanbmcs1J8MuWz8JmldNr2jk+nbpZ8Q
 dQU+sHBOFwfPEAEZaLhYiYC9c4Edrkqa1KyIf6vF7rVZQRlq77brdLw5WTFHRWWq
 2n76jNG10M6kxUc4PjJcwQZDY8nweFZX+HBABODrAHBmvqV2Y1bSwpk6j2aYKj37
 YPV8jAUWDAkX/CNjnsUIzTrmwA81hdZwQ4Be6vGWtf5tr/Jz+JmStSH+pZhksRa4
 R9+Fs4h6YpcjvsZ9925GzV9zOo9Gr8H+h8LNHZ8CGs5wgxXm7XBwZG8q02Rh2jkC
 v/1P1VrhEuNCpmwEX9TDsPL+cHW0PiiusckLli5pF8IeS5c9iwtlJ6TTO4Lip/6n
 fX6/ZBPues0YlqfMOwLnT2/dHd5hrkBG751QMqfLq55+M+qFU9k2wjiK2+xvehAU
 bJ3/ZLgxAJU1hBlWA0S+0mKKQScgI5FopbSi7WRuBxHjTmMwkz8=
 =0oP2
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/jnsnow/tags/ide-pull-request' into staging

Pull request

Stable notes: patches one and two can be considered
              for the next -stable release.

# gpg: Signature made Sat 17 Aug 2019 00:15:50 BST
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/ide-pull-request:
  hw/ide/atapi: Use the ldst API
  Revert "ide/ahci: Check for -ECANCELED in aio callbacks"
  dma-helpers: ensure AIO callback is invoked after cancellation

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-08-19 12:32:15 +01:00
Alex Bennée
5f8ab0004e targets (various): use softfloat-helpers.h where we can
Generally the cpu and non-FP helper files just want to manipulate the
softfloat flags. For this they can just use the -helpers.h include
which brings in a minimal number of inline helpers.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
135b03cb9d target/riscv: rationalise softfloat includes
We should avoid including the whole of softfloat headers in cpu.h and
explicitly include it only where we will be calling softfloat
functions. We can use the -types.h and -helpers.h in cpu.h for the few
bits that are global.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Palmer Dabbelt <palmer@sifive.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
502700d067 target/mips: rationalise softfloat includes
We should avoid including the whole of softfloat headers in cpu.h and
explicitly include it only where we will be calling softfloat
functions. We can use the -types.h in cpu.h for the few bits that are
global. We also move the restore_snan_bit_mode into internal.h and
include -helpers.h there.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
00f43279a3 fpu: rename softfloat-specialize.h -> .inc.c
This is not a normal header and should only be included in the main
softfloat.c file to bring in the various target specific
specialisations. Indeed as it contains non-inlined C functions it is
not even a legal header. Rename it to match our included C convention.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
5937fb63a9 fpu: make softfloat-macros "self-contained"
The macros use the "flags" type and to be consistent if anyone just
needs the macros we should bring in the header we need. There is an
outstanding TODO to audit the use of "flags" and replace with bool at
which point this include could be dropped.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
e34c47ea3f fpu: move inline helpers into a separate header
There are a bunch of users of the inline helpers who do not need
access to the entire softfloat API. Move those inline helpers into a
new header file which can be included without bringing in the rest of
the world.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-19 12:07:13 +01:00
Alex Bennée
472038ccf5 fpu: remove the LIT64 macro
Now the rest of the code has been cleaned up we can remove this.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 12:07:13 +01:00
Alex Bennée
e23263004d target/m68k: replace LIT64 with UINT64_C macros
In our quest to eliminate the home rolled LIT64 macro we fixup usage
inside for m68k's many constants.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 12:07:13 +01:00
Alex Bennée
e932112420 fpu: replace LIT64 with UINT64_C macros
In our quest to eliminate the home rolled LIT64 macro we fixup usage
inside the softfloat code. While we are at it we remove some of the
extraneous spaces to closer fit the house style.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 12:07:13 +01:00
Alex Bennée
2c217da0fc fpu: use min/max values from stdint.h for integral overflow
Remove some more use of LIT64 while making the meaning more clear. We
also avoid the need of casts as the results by definition fit into the
return type.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 12:07:13 +01:00
Alex Bennée
e6b405fe00 fpu: convert float[16/32/64]_squash_denormal to new modern style
This also allows us to remove the extractFloat16exp/frac helpers. We
avoid using the floatXX_pack_raw functions as they are slight overkill
for masking out all but the top bit of the number. The generated code
is almost exactly the same as makes no difference to the
pre-conversion code.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-08-19 12:07:13 +01:00
Alex Bennée
f7e81a9457 fpu: replace LIT64 usage with UINT64_C for specialize constants
We have a wrapper that does the right thing from stdint.h so lets use
it for our constants in softfloat-specialize.h

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-19 12:07:13 +01:00
Peter Maydell
1f37316238 Pull request
Rebase notes:
 
 011/36:[0003] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
 016/36:[----] [-C] 'iotests: Add virtio-scsi device helper'
 017/36:[0002] [FC] 'iotests: add test 257 for bitmap-mode backups'
 030/36:[0011] [FC] 'block/backup: teach TOP to never copy unallocated regions'
 032/36:[0018] [FC] 'iotests/257: test traditional sync modes'
 
 11: A new hbitmap call was added late in 4.1, changed to
     bdrv_dirty_bitmap_next_zero.
 16: Context-only (self.has_quit is new context in 040)
 17: Removed 'auto' to follow upstream trends in iotest fashion
 30: Handled explicitly on-list with R-B from Max.
 32: Fix capitalization in test, as mentioned on-list.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+ber27ys35W+dsvQfe+BBqr8OQ4FAl1XOE0ACgkQfe+BBqr8
 OQ5vZw//RyNt7SDRyB2X1eQnD1LGNunH7ZgucCNccFf0QlbfEYAQWCobylcLw5jk
 WH3GAbo41YgN5yhqTYKc17bXGTdmrQjE5INmpL1X1Eq85HqOGSxK94lQ3EBodbAS
 ktAriK2b9xW/DZ4OvP6pQ5U6PkQZk/GyzvcgteFjqbR6gGCpGGPr97nRjfn7aNzn
 7UuEl30CrdEZyVEkxIxTWnRWOSGXv3zPupkTzPb4PQ3nOEHX3iq+rmEkp9mE46dW
 VJucWC+SXGfI5+Lpsofa7+oJ2Cqa6spFRYk/3gLmahvm5bDT0CYG5XKCTSFkFedO
 lsibS/LkdecHDsTYlU5eIIhcD5l7hhPd+A4dlSQyNMqqHY90wPsaTgAy9czURjIO
 H37DJfAld1z8AE5ekXisdTIOypa7pmnS+4WyB73ZynJrCyxNRjI2heNavnrkPDMG
 IZQT64wGmaaleQZdOo30rAqp8Lsh7XTag0RRijlw8OdngYQyMzyp94uU0hm4grMu
 6UJD0w81jxacyRZ/rfUT4BLKwzly4vfS1LlVoNaQk5fgkuJeKbfYJnVgL7ZNfzvR
 +DTg1NHOudHkgUaW+LtV2F74D5k1xPYSpT+1WVDP6n8fzXEMyQPLb6RZcWhAazka
 8vAiRSLygoXdDzhQHqvSIqweuZPXOrpRjU1Iiva3r8wcxjKGj7M=
 =4WRP
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

Rebase notes:

011/36:[0003] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
016/36:[----] [-C] 'iotests: Add virtio-scsi device helper'
017/36:[0002] [FC] 'iotests: add test 257 for bitmap-mode backups'
030/36:[0011] [FC] 'block/backup: teach TOP to never copy unallocated regions'
032/36:[0018] [FC] 'iotests/257: test traditional sync modes'

11: A new hbitmap call was added late in 4.1, changed to
    bdrv_dirty_bitmap_next_zero.
16: Context-only (self.has_quit is new context in 040)
17: Removed 'auto' to follow upstream trends in iotest fashion
30: Handled explicitly on-list with R-B from Max.
32: Fix capitalization in test, as mentioned on-list.

# gpg: Signature made Sat 17 Aug 2019 00:12:13 BST
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request: (36 commits)
  tests/test-hbitmap: test next_zero and _next_dirty_area after truncate
  block/backup: refactor write_flags
  block/backup: deal with zero detection
  qapi: add dirty-bitmaps to query-named-block-nodes result
  iotests/257: test traditional sync modes
  block/backup: support bitmap sync modes for non-bitmap backups
  block/backup: teach TOP to never copy unallocated regions
  block/backup: add backup_is_cluster_allocated
  block/backup: centralize copy_bitmap initialization
  block/backup: improve sync=bitmap work estimates
  iotests/257: test API failures
  block/backup: hoist bitmap check into QMP interface
  iotests/257: Refactor backup helpers
  iotests/257: add EmulatedBitmap class
  iotests/257: add Pattern class
  iotests: test bitmap moving inside 254
  qapi: implement block-dirty-bitmap-remove transaction action
  blockdev: reduce aio_context locked sections in bitmap add/remove
  block/backup: loosen restriction on readonly bitmaps
  iotests: add test 257 for bitmap-mode backups
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-08-19 10:55:03 +01:00
Thomas Huth
72e031f3b8 gitlab-ci: Remove qcow2 tests that are handled by "make check" already
Since most iotests are now run during "make check" already, we do not
need to test them explicitly from the gitlab-ci.yml script anymore.
And while we're at it, add some of the new non-auto tests >= 246 instead.

Message-Id: <20190717111947.30356-5-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2019-08-17 09:06:17 +02:00
Thomas Huth
bdd95e4784 tests: Run the iotests during "make check" again
People often forget to run the iotests before submitting patches or pull
requests - this is likely due to the fact that we do not run the tests
during our mandatory "make check" tests yet. Now that we've got a proper
"auto" group of iotests that should be fine to run in every environment,
we can enable the iotests during "make check" again by running the "auto"
tests by default from the check-block.sh script.

Some cases still need to be checked first, though: iotests need bash and
GNU sed (otherwise they fail), and if gprof is enabled, it spoils the
output of some test cases causing them to fail. So if we detect that one
of the required programs is missing or that gprof is enabled, we still
have to skip the iotests to avoid failures.

And finally, since we are using check-block.sh now again, this patch also
removes the qemu-iotests-quick.sh script since we do not need that anymore
(and having two shell wrapper scripts around the block tests seems rather
confusing than helpful).

Message-Id: <20190717111947.30356-4-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
[AJB: -makecheck to check-block.sh, move check-block to start and gate it]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-08-17 09:03:35 +02:00
Paolo Bonzini
f6fc1e30cf block: fix NetBSD qemu-iotests failure
Opening a block device on NetBSD has an additional step compared to other OSes,
corresponding to raw_normalize_devicepath.  The error message in that function
is slightly different from that in raw_open_common and this was causing spurious
failures in qemu-iotests.  However, in general it is not important to know what
exact step was failing, for example in the qemu-iotests case the error message
contains the fairly unequivocal "No such file or directory" text from strerror.
We can thus fix the failures by standardizing on a single error message for
both raw_open_common and raw_normalize_devicepath; in fact, we can even
use error_setg_file_open to make sure the error message is the same as in
the rest of QEMU.

Message-Id: <20190725095920.28419-1-pbonzini@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2019-08-17 09:02:59 +02:00
Philippe Mathieu-Daudé
614ab7d127 hw/ide/atapi: Use the ldst API
The big-endian load/store functions are already provided
by "qemu/bswap.h".
Avoid code duplication, use the generic API.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190808130454.9930-1-philmd@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 19:14:04 -04:00
John Snow
8ec41c4265 Revert "ide/ahci: Check for -ECANCELED in aio callbacks"
This reverts commit 0d910cfeaf.

It's not correct to just ignore an error code in a callback; we need to
handle that error and possible report failure to the guest so that they
don't wait indefinitely for an operation that will now never finish.

This ought to help cases reported by Nutanix where iSCSI returns a
legitimate -ECANCELED for certain operations which should be propagated
normally.

Reported-by: Shaju Abraham <shaju.abraham@nutanix.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190729223605.7163-1-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 19:14:04 -04:00
Paolo Bonzini
539343c0a4 dma-helpers: ensure AIO callback is invoked after cancellation
dma_aio_cancel unschedules the BH if there is one, which corresponds
to the reschedule_dma case of dma_blk_cb.  This can stall the DMA
permanently, because dma_complete will never get invoked and therefore
nobody will ever invoke the original AIO callback in dbs->common.cb.

Fix this by invoking the callback (which is ensured to happen after
a bdrv_aio_cancel_async, or done manually in the dbs->bh case), and
add assertions to check that the DMA state machine is indeed waiting
for dma_complete or reschedule_dma, but never both.

Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20190729213416.1972-1-pbonzini@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 19:14:04 -04:00
Vladimir Sementsov-Ogievskiy
a5f8a60b3e tests/test-hbitmap: test next_zero and _next_dirty_area after truncate
Test that hbitmap_next_zero and hbitmap_next_dirty_area can find things
after old bitmap end.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190805164652.42409-1-vsementsov@virtuozzo.com
Tested-by: John Snow <jsnow@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 18:29:43 -04:00
Vladimir Sementsov-Ogievskiy
a1ed82b443 block/backup: refactor write_flags
write flags are constant, let's store it in BackupBlockJob instead of
recalculating. It also makes two boolean fields to be unused, so,
drop them.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190730163251.755248-4-vsementsov@virtuozzo.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 18:29:43 -04:00
Vladimir Sementsov-Ogievskiy
319bd5edb9 block/backup: deal with zero detection
We have detect_zeroes option, so at least for blockdev-backup user
should define it if zero-detection is needed. For drive-backup leave
detection enabled by default but do it through existing option instead
of open-coding.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190730163251.755248-2-vsementsov@virtuozzo.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 18:29:43 -04:00
Vladimir Sementsov-Ogievskiy
590a63d598 qapi: add dirty-bitmaps to query-named-block-nodes result
Let's add a possibility to query dirty-bitmaps not only on root nodes.
It is useful when dealing both with snapshots and incremental backups.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190717173937.18747-1-jsnow@redhat.com
[Added deprecation information. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
[Fixed spelling --js]
2019-08-16 18:29:43 -04:00
John Snow
bd5ceebf6d iotests/257: test traditional sync modes
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-12-jsnow@redhat.com
[Edit 'Bitmap' --> 'bitmap' in 257.out --js]
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 18:29:43 -04:00
John Snow
1a2b8b406b block/backup: support bitmap sync modes for non-bitmap backups
Accept bitmaps and sync policies for the other backup modes.
This allows us to do things like create a bitmap synced to a full backup
without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything,
  because the bitmap isn't used for input or output.
  It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this
  job never completes with success anyway, and the resulting artifact
  of 'always' is suspect: because we start with a full bitmap and only
  copy out segments that get written to, the final output bitmap will
  always be ... a fully set bitmap.

  Maybe there's contexts in which bitmaps make sense for sync=none,
  but not without more severe changes to the current job, and omitting
  it here doesn't prevent us from adding it later.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 18:29:43 -04:00
John Snow
7e30dd618e block/backup: teach TOP to never copy unallocated regions
Presently, If sync=TOP is selected, we mark the entire bitmap as dirty.
In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the
allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is
invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups
now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Taken together, the floor and ceiling move to meet each other.


Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190716000117.25219-10-jsnow@redhat.com
[Remove ret = -ECANCELED change. --js]
[Squash in conflict resolution based on Max's patch --js]
Message-id: c8b0ab36-79c8-0b4b-3193-4e12ed8c848b@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
dba8700f16 block/backup: add backup_is_cluster_allocated
Modify bdrv_is_unallocated_range to utilize the pnum return from
bdrv_is_allocated, and in the process change the semantics from
"is unallocated" to "is allocated."

Optionally returns a full number of clusters that share the same
allocation status.

This will be used to carefully toggle bits in the bitmap for sync=top
initialization in the following commits.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
141cdcdf84 block/backup: centralize copy_bitmap initialization
Just a few housekeeping changes that keeps the following commit easier
to read; perform the initial copy_bitmap initialization in one place.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
0fff1f1371 block/backup: improve sync=bitmap work estimates
When making backups based on bitmaps, the work estimate can be more
accurate. Update iotests to reflect the new strategy.

TOP work estimates are broken, but do not get worse with this commit.
That issue is addressed in the following commits instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
352092d382 iotests/257: test API failures
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
a6c9365ad4 block/backup: hoist bitmap check into QMP interface
This is nicer to do in the unified QMP interface that we have now,
because it lets us use the right terminology back at the user.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
0af2a09c6b iotests/257: Refactor backup helpers
This test needs support for non-bitmap backups and missing or
unspecified bitmap sync modes, so rewrite the helpers to be a little
more generic.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
32afa5a1d4 iotests/257: add EmulatedBitmap class
Represent a bitmap with an object that we can mark and clear bits in.
This makes it easier to manage partial writes when we don't write a
full group's worth of patterns before an error.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00
John Snow
b0a32bef7b iotests/257: add Pattern class
Just kidding, this is easier to manage with a full class instead of a
namedtuple.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190716000117.25219-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-08-16 16:28:03 -04:00