Commit Graph

1359 Commits

Author SHA1 Message Date
Max Reitz
876df72d75 iotests: Fix 233 for ports other than 10809
233 generally filters the port, but in two cases does not.  If some
other concurrently running application has already taken port 10809,
this will result in an output mismatch.  Fix this by applying the
filter in these two cases, too.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20190506160529.6955-1-mreitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-05-07 09:43:42 -05:00
Alberto Garcia
a2aa8b07cd iotests: Check that images are in read-only mode after block-commit
This tests the fix from the previous patch.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-30 15:29:00 +02:00
Max Reitz
f4619af0c1 qemu-img: Make create hint at protocol options
qemu-img create allows giving just a format and "-o help" to get a list
of the options supported by that format.  Users may not realize that the
protocol level may offer even more options, which they only get to see
by specifying a filename.

This patch adds a note to hint at that fact.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-30 15:29:00 +02:00
Max Reitz
0ef5a8e6ce iotests: Perform the correct test in 082
In the "amend" section of 082, we perform a single "convert" test
(namely "convert -o help").  That does not make sense, especially
because we have done exactly that "convert" test earlier in 082 already.

Replacing "convert" by "amend" yields an error, which is correct because
there is no point in "amend" having a default format.  The user has to
either specify the format, or give a file for qemu-img to probe.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-30 15:29:00 +02:00
Eric Blake
de38b5005e qemu-img: Saner printing of large file sizes
Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

  $ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nbd'
  image: nbd+unix://?socket=/tmp/nbdkitHSAzNz/socket
  file format: raw
  virtual size: -8388607T (9223372036854775296 bytes)
  disk size: unavailable

But there's no reason to have two separate implementations of integer
to human-readable abbreviation, where one has overflow and stops at
'T', while the other avoids overflow and goes all the way to 'E'. With
this patch, the output now claims 8EiB instead of -8388607T, which
really is the correct rounding of largest file size supported by qemu
(we could go 511 bytes larger if we used byte-accurate sizing instead
of rounding up to the next sector boundary, but that wouldn't change
the human-readable result).

Quite a few iotests need updates to expected output to match.

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Max Reitz <mreitz@redhat.com>
2019-04-30 15:29:00 +02:00
Thomas Huth
36b9986b08 tests/qemu-iotests: Fix output of qemu-io related tests
One of the recent commits changed the way qemu-io prints out its
errors and warnings - they are now prefixed with the program name.
We've got to adapt the iotests accordingly to prevent that they
are failing.

Fixes: 99e98d7c9f ("qemu-io: Use error_[gs]et_progname()")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-30 15:29:00 +02:00
Eric Blake
9749636b00 iotest: Fix 241 to run in generic directory
Filter the qemu-nbd server output to get rid of a direct reference
to my build directory.

Fixes: e9dce9cb
Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-12 18:03:01 +02:00
Max Reitz
23e1d05411 iotests: Let 245 pass on tmpfs
tmpfs does not support O_DIRECT.  Detect this case, and skip flipping
@direct if the filesystem does not support it.

Fixes: bf3e50f623
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-12 18:03:01 +02:00
Thomas Huth
f18957b854 tests/qemu-iotests/235: Allow fallback to tcg
iotest 235 currently only works with KVM - this is bad for systems where
it is not available, e.g. CI pipelines. The test also works when using
"tcg" as accelerator, so we can simply add that to the list of accelerators,
too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:56 +02:00
Alberto Garcia
d20ba603f2 block: test block-stream with a base node that is used by block-commit
The base node of a block-stream operation indicates the first image
from the backing chain starting from which no data is copied to the
top node.

The block-stream job allows others to use that base image, so a second
block-stream job could be writing to it at the same time. An important
restriction is that the base image must not disappear while the stream
job is ongoing. stream_start() freezes the backing chain from top to
base with that purpose but it does it too late in the code so there is
a race condition there.

This bug was fixed in the previous commit, and this patch contains an
iotest for this scenario.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Thomas Huth
38e694fcc9 tests/qemu-iotests: Remove redundant COPYING file
The file tests/qemu-iotests/COPYING is the same text as in the
COPYING file in the main directory. So as far as I can see, we don't
need the duplicate here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Thomas Huth
e0a59749ef iotests: Fix test 200 on s390x without virtio-pci
virtio-pci is optional on s390x, e.g. in downstream RHEL builds, it
is disabled. On s390x, virtio-ccw should be used instead. Other tests
like 051 or 240 already use virtio-scsi-ccw instead of virtio-scsi-pci
on s390x, so let's do the same here and always use virtio-scsi-ccw on
s390x.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Eric Blake
b0245d6478 nbd/server: Advertise actual minimum block size
Both NBD_CMD_BLOCK_STATUS and structured NBD_CMD_READ will split their
reply according to bdrv_block_status() boundaries. If the block device
has a request_alignment smaller than 512, but we advertise a block
alignment of 512 to the client, then this can result in the server
reply violating client expectations by reporting a smaller region of
the export than what the client is permitted to address (although this
is less of an issue for qemu 4.0 clients, given recent client patches
to overlook our non-compliance at EOF).  Since it's always better to
be strict in what we send, it is worth advertising the actual minimum
block limit rather than blindly rounding it up to 512.

Note that this patch is not foolproof - it is still possible to
provoke non-compliant server behavior using:

$ qemu-nbd --image-opts driver=blkdebug,align=512,image.driver=file,image.filename=/path/to/non-aligned-file

That is arguably a bug in the blkdebug driver (it should never pass
back block status smaller than its alignment, even if it has to make
multiple bdrv_get_status calls and determine the
least-common-denominator status among the group to return). It may
also be possible to observe issues with a backing layer with smaller
alignment than the active layer, although so far I have been unable to
write a reliable iotest for that scenario (but again, an issue like
that could be argued to be a bug in the block layer, or something
where we need a flag to bdrv_block_status() to state whether the
result must be aligned to the current layer's limits or can be
subdivided for accuracy when chasing backing files).

Anyways, as blkdebug is not normally used, and as this patch makes our
server more interoperable with qemu 3.1 clients, it is worth applying
now, even while we still work on a larger patch series for the 4.1
timeframe to have byte-accurate file lengths.

Note that the iotests output changes - for 223 and 233, we can see the
server's better granularity advertisement; and for 241, the three test
cases have the following effects:
- natural alignment: the server's smaller alignment is now advertised,
and the hole reported at EOF is now the right result; we've gotten rid
of the server's non-compliance
- forced server alignment: the server still advertises 512 bytes, but
still sends a mid-sector hole. This is still a server compliance bug,
which needs to be fixed in the block layer in a later patch; output
does not change because the client is already being tolerant of the
non-compliance
- forced client alignment: the server's smaller alignment means that
the client now sees the server's status change mid-sector without any
protocol violations, but the fact that the map shows an unaligned
mid-sector hole is evidence of the block layer problems with aligned
block status, to be fixed in a later patch

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-7-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: rebase to enhanced iotest 241 coverage]
2019-04-01 08:52:28 -05:00
Eric Blake
a62a85ef5c nbd/client: Report offsets in bdrv_block_status
It is desirable for 'qemu-img map' to have the same output for a file
whether it is served over file or nbd protocols. However, ever since
we implemented block status for NBD (2.12), the NBD protocol forgot to
inform the block layer that as the final layer in the chain, the
offset is valid; without an offset, the human-readable form of
qemu-img map gives up with the unhelpful:

$ nbdkit -U - data data="1" size=512 --run 'qemu-img map $nbd'
Offset          Length          Mapped to       File
qemu-img: File contains external, encrypted or compressed clusters.

The --output=json form always works, because it is reporting the
lower-level bdrv_block_status results directly rather than trying to
filter out sparse ranges for human consumption - but now it also
shows the offset member.

With this patch, the human output changes to:

Offset          Length          Mapped to       File
0               0x200           0               nbd+unix://?socket=/tmp/nbdkitOxeoLa/socket

This change is observable to several iotests.

Fixes: 78a33ab5
Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-4-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 20:52:29 -05:00
Eric Blake
e9dce9cb6e iotests: Add 241 to test NBD on unaligned images
Add a test for the NBD client workaround in the previous patch.  It's
not really feasible for an iotest to assume a specific tracing engine,
so we can't really probe trace_nbd_parse_blockstatus_compliance to see
if the server was fixed vs. whether the client just worked around the
server (other than by rearranging order between code patches and this
test). But having a successful exchange sure beats the previous state
of an error message. Since format probing can change alignment, we can
use that as an easy way to test several configurations.

Not tested yet, but worth adding to this test in future patches: an
NBD server that can advertise a non-sector-aligned size (such as
nbdkit) causes qemu as the NBD client to misbehave when it rounds the
size up and accesses beyond the advertised size. Qemu as NBD server
never advertises a non-sector-aligned size (since bdrv_getlength()
currently rounds up to sector boundaries); until qemu can act as such
a server, testing that flaw will have to rely on external binaries.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-2-eblake@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: add forced-512 alignment, and nbdkit reproducer comment]
2019-03-30 20:50:58 -05:00
Vladimir Sementsov-Ogievskiy
a66c4b83c9 iotests: add 248: test resume mirror after auto pause on ENOSPC
Test that mirror job actually resume on resume command after being
automatically paused on ENOSPC error.

It's a follow-up test for 8d9648cbf3
    "blockjob: fix user pause in block_job_error_action"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: John Snow <jsnow@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-26 11:37:51 +01:00
Lukáš Doktor
59fba0aaee qemu-iotests: Treat custom TEST_DIR in 051
When custom TEST_DIR is specified the output includes it without leading
'/':

    $ TEST_DIR=/var/tmp ./check -file -qcow2 051
    ....
-drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file":
{"driver": "file", "filename": "TEST_DIR/t.qcow2"}}, "driver": "qcow2",
"file": {"driver": "file", "filename": SNAPSHOT_PATH}} (qcow2,
read-only)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file":
{"driver": "file", "filename": "TEST_DIR/t.qcow2"}}, "driver": "qcow2",
"file": {"driver": "file", "filename": "TEST_DIR/vl.ziHfeP"}} (qcow2,
read-only)

Let's remove it from the sed regexp.

Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-19 15:51:31 +01:00
Kevin Wolf
27e42789b7 qemu-iotests: Fix 232 for non-qcow2
232 is marked as generic, but commit 12efe428c9 added code that assumes
qcow2. What the new test really needs is backing files and support for
updating the backing file link (.bdrv_change_backing_file).

Split the non-generic code into a new test case 247 and make it work
with qed, too.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-19 15:49:29 +01:00
Sergio Lopez
9cd97956cf iotests: 153: Wait for an answer to QMP commands
There are various actions in this test that must be executed
sequentially, as the result of it depends on the state triggered by the
previous one.

If the last argument of _send_qemu_cmd() is an empty string, it just
sends the QMP commands without waiting for an answer. While unlikely, it
may happen that the next action in the test gets invoked before QEMU
processes the QMP request.

This issue seems to be easier to reproduce on servers with limited
resources or highly loaded.

With this change, we wait for an answer on all _send_qemu_cmd() calls.

Signed-off-by: Sergio Lopez <slp@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-19 15:49:29 +01:00
Peter Maydell
523a2a42c3 Pull request
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+ber27ys35W+dsvQfe+BBqr8OQ4FAlyIFSwACgkQfe+BBqr8
 OQ7wghAAm16eCEr57oTO7QXR3y8uVFsKqXBn9cNH6nbrFp2PUQSglwMDKBls1Z5m
 olF23X/JaqSlSmkL9BBuzDZ6Up+kkHKuxPq4/5RKXfiDI0pr3R0eqts0COAlaN9q
 Bew3ipj99m8gzMi2093AW4+Ob0N3658fuDTGLe1M1Uoy7CEg1QJ7rVOBBEui7vIl
 RbZ8l/Zmb4ldNpB3lnE4Nh9ue8fy0RAj3Nai161nCnNeXNF/VzD3Ye8bojSBbnux
 PIMX6/RWmykX4feIf9QP8apDpxX4HkyuPq5EdwT9PD8PwdyXPAXZtsYUNCuNtQuk
 n5VKFVgFYgqUclBeVHmrMYPU4K4iCFQp4/Fua7wzPEC0iG05NiiDv91oVkEJCp3L
 ManHeuGfNLCcXaIntKZhuJl1cK8yMM3yDww6/pPTehrPjcyvKa0NOqhQBExektcD
 R6q7maJRzFaxSxdcs+Zzuog9zESvH1mlJxQCKzeYhAP0kkxInyTELE/Vbx37xuqR
 RFfZYyVQ6x87Q/sxHx4EMiV97WUM8elZOQdSEC/okt5WUUNpgIu0WF9nSQ1VKZ8C
 CZmv5xh9ogfwvB/kOm6IVwNkLvVagJQcLwddORI5LLXLbSIUcuwVSuyMp/7iDtQ/
 hnHkGs2mIJ2JUYbSSNsSJNs6oTurn8eTFCeGoYKJgd9l4QxaThU=
 =ekU+
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

# gpg: Signature made Tue 12 Mar 2019 20:23:08 GMT
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request: (22 commits)
  tests/qemu-iotests: add bitmap resize test 246
  block/qcow2-bitmap: Allow resizes with persistent bitmaps
  block/qcow2-bitmap: Don't check size for IN_USE bitmap
  docs/interop/qcow2: Improve bitmap flag in_use specification
  bitmaps: Fix typo in function name
  block/dirty-bitmaps: implement inconsistent bit
  block/dirty-bitmaps: disallow busy bitmaps as merge source
  block/dirty-bitmaps: prohibit removing readonly bitmaps
  block/dirty-bitmaps: prohibit readonly bitmaps for backups
  block/dirty-bitmaps: add block_dirty_bitmap_check function
  block/dirty-bitmap: add inconsistent status
  block/dirty-bitmaps: add inconsistent bit
  iotests: add busy/recording bit test to 124
  blockdev: remove unused paio parameter documentation
  block/dirty-bitmaps: move comment block
  block/dirty-bitmaps: unify qmp_locked and user_locked calls
  block/dirty-bitmap: explicitly lock bitmaps with successors
  nbd: change error checking order for bitmaps
  block/dirty-bitmap: change semantics of enabled predicate
  block/dirty-bitmap: remove set/reset assertions against enabled bit
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

# Conflicts:
#	tests/qemu-iotests/group
2019-03-13 17:30:34 +00:00
Alberto Garcia
bf3e50f623 qemu-iotests: Test the x-blockdev-reopen QMP command
This patch adds several tests for the x-blockdev-reopen QMP command.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-12 20:30:14 +01:00
Kevin Wolf
23dece19da file-posix: Make auto-read-only dynamic
Until now, with auto-read-only=on we tried to open the file read-write
first and if that failed, read-only was tried. This is actually not good
enough for libvirt, which gives QEMU SELinux permissions for read-write
only as soon as it actually intends to write to the image. So we need to
be able to switch between read-only and read-write at runtime.

This patch makes auto-read-only dynamic, i.e. the file is opened
read-only as long as no user of the node has requested write
permissions, but it is automatically reopened read-write as soon as the
first writer is attached. Conversely, if the last writer goes away, the
file is reopened read-only again.

bs->read_only is no longer set for auto-read-only=on files even if the
file descriptor is opened read-only because it will be transparently
upgraded as soon as a writer is attached. This changes the output of
qemu-iotests 232.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-12 20:30:14 +01:00
Kevin Wolf
3085513778 file-posix: Fix bdrv_open_flags() for snapshot=on
Using a different read-only setting for bs->open_flags than for the
flags to the driver's open function is just inconsistent and a bad idea.
After this patch, the temporary snapshot keeps being opened read-only if
read-only=on,snapshot=on is passed.

If we wanted to change this behaviour to make only the orginal image
file read-only, but the temporary overlay read-write (as the comment in
the removed code suggests), that change would have to be made in
bdrv_temp_snapshot_options() (where the comment suggests otherwise).

Addressing this inconsistency before introducing dynamic auto-read-only
is important because otherwise we would immediately try to reopen the
temporary overlay even though the file is already unlinked.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-12 20:30:14 +01:00
Kevin Wolf
12efe428c9 qemu-iotests: commit to backing file with auto-read-only
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-03-12 20:30:14 +01:00
John Snow
e2ec4119dc tests/qemu-iotests: add bitmap resize test 246
Test that we can actually resize qcow2 images with persistent bitmaps
correctly. Throw some other goofy stuff at the test while we're at it,
like adding bitmaps of different granularities and at different times.

Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Eric Blake <eblake@redhat.com>
Message-id: 20190311185147.52309-5-vsementsov@virtuozzo.com
   [vsmentsov: drop \n from the end of test output,
      test output changed a bit: some bitmaps goes in other order
      int the output]
Signed-off-by: John Snow <jsnow@redhat.com>
2019-03-12 15:00:48 -04:00
John Snow
c61b198b63 iotests: add busy/recording bit test to 124
This adds a simple test that ensures the busy bit works for push backups,
as well as doubling as bonus test for incremental backups that get interrupted
by EIO errors.

Recording bit tests are already handled sufficiently by 236.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Eric Blake <eblake@redhat.com>
Message-id: 20190223000614.13894-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-03-12 12:05:48 -04:00
John Snow
4db6ceb0b5 block/dirty-bitmap: add recording and busy properties
The current API allows us to report a single status, which we've defined as:

Frozen: has a successor, treated as qmp_locked, may or may not be enabled.
Locked: no successor, qmp_locked. may or may not be enabled.
Disabled: Not frozen or locked, disabled.
Active: Not frozen, locked, or disabled.

The problem is that both "Frozen" and "Locked" mean nearly the same thing,
and that both of them do not intuit whether they are recording guest writes
or not.

This patch deprecates that status field and introduces two orthogonal
properties instead to replace it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190223000614.13894-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-03-12 12:05:48 -04:00
Peter Maydell
e2a18635a4 nbd patches for 2019-03-08
- support TLS client authorization in NBD servers
 - iotest 223 race fix
 -----BEGIN PGP SIGNATURE-----
 
 iQEcBAABCAAGBQJcgqh3AAoJEKeha0olJ0Nq/bgH/1TXo49gC9SMNcBBHd5vqc6/
 J+eXYQihmGLy7pNkfCBTB0QZz9d7V4tN/N1PAuIfzsHxcQJeyUBwcY7jin2SiTTM
 lfR9NWDY43OE+8tcPSXODyo3mge8g3d1X3vw8/QMX95TDrKQ8SMwAllegCFBKPZs
 T0+Jyfd8oA0NcQz4EPPUL5f2ptLo2slye2ZjbMBn/1WFrYkL+joUYJgyakYcZnY/
 mcvmXF2JLG2fPzFoU1yvF+oZn6J2fx5pw92P+SZ7lA+qRzlWfvrVyK9sNqCS+K5m
 qdfMeeL/SyUPsUvlcbDH7iSjxWkR/h7MtXRq83FHzupasMeXiQ9ieb3MFAtHnGM=
 =5pyZ
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2019-03-08' into staging

nbd patches for 2019-03-08

- support TLS client authorization in NBD servers
- iotest 223 race fix

# gpg: Signature made Fri 08 Mar 2019 17:37:59 GMT
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2019-03-08:
  iotests: Wait for qemu to end in 223
  nbd: fix outdated qapi docs syntax for tls-creds
  nbd: allow authorization with nbd-server-start QMP command
  qemu-nbd: add support for authorization of TLS clients

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-03-09 20:55:44 +00:00
Kevin Wolf
ac40260dc5 qemu-iotests: amend with external data file
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:46 +01:00
Kevin Wolf
76b90e23e7 qemu-iotests: General tests for qcow2 with external data file
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:46 +01:00
Kevin Wolf
c35896c5e8 qemu-iotests: Preallocation with external data file
Test that preallocating metadata results in a somewhat larger qcow2
file, but preallocating data only affects the disk usage of the data
file and the qcow2 file stays small.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:46 +01:00
Kevin Wolf
6c3944dc62 qcow2: Implement data-file-raw create option
Provide an option to force QEMU to always keep the external data file
consistent as a standalone read-only raw image.

At the moment, this means making sure that write_zeroes requests are
forwarded to the data file instead of just updating the metadata, and
checking that no backing file is used.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:46 +01:00
Kevin Wolf
9b890bdcb6 qcow2: Store data file name in the image
Rather than requiring that the external data file node is passed
explicitly when creating the qcow2 node, store the filename in the
designated header extension during .bdrv_create and read it from there
as a default during .bdrv_open.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:46 +01:00
Kevin Wolf
77e023ff79 qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()
qcow2_alloc_compressed_cluster_offset() used to return the cluster
offset for success and 0 for error. This doesn't only conflict with 0 as
a valid host offset, but also loses the error code.

Similar to the change made to qcow2_alloc_cluster_offset() for
uncompressed clusters in commit 148da7ea9d, make the function return
0/-errno and return the allocated cluster offset in a by-reference
parameter.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Kevin Wolf
93c2493646 qcow2: Basic definitions for external data files
This adds basic constants, struct fields and helper function for
external data file support to the implementation.

QCOW2_INCOMPAT_MASK and QCOW2_AUTOCLEAR_MASK are not updated yet so that
opening images with an external data file still fails (we don't handle
them correctly yet).

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Kevin Wolf
97f94cb4f8 qemu-iotests: Test qcow2 preallocation modes
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-03-08 12:26:45 +01:00
Philippe Mathieu-Daudé
bde36af1ab qemu-iotests: Ensure GNU sed is used
Various sed regexp from common.filter use sed GNU extensions.
Instead of spending time to write these regex to be POSIX compliant,
verify the GNU sed is available and use it.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Philippe Mathieu-Daudé
11a82d1429 qemu-iotests: Improve portability by searching bash in the $PATH
Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.

Patch created mechanically by running:

  $ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \
    | while read f; do \
      sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \
    done

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Andrey Shinkevich
d9df28e7b0 iotests: check whitelisted formats
Some test cases require specific formats. The method decorator
skip_if_unsupported() checks if requested formats are whitelisted.
The test #139 was selected for a sample output, after running
$ ./check -qcow2 131-140

137 3s ...
138 0s ...
139 2s ...
    [case not run] testBlkDebug (__main__.TestBlockdevDel): formats ['blkdebug'] are not whitelisted
    [case not run] testBlkVerify (__main__.TestBlockdevDel): formats ['blkverify'] are not whitelisted
    [case not run] testQuorum (__main__.TestBlockdevDel): formats ['quorum'] are not whitelisted
140 0s ...
Not run: 131 135 136
Some cases not run in: 139
Passed all 7 tests

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Andrey Shinkevich
57ed557f03 iotests: ask QEMU for supported formats
Supported formats listed by 'qemu' may differ from those listed by
'qemu-img' due to whitelists. Some test cases require specific formats
that may be used with qemu. They can be inquired directly by running
'qemu -drive format=help'. The response takes whitelists into account.
The method supported_formats() serves for that. The method decorator
skip_if_unsupported() checks if all requested formats are whitelisted.
If not, the test case will be skipped. That has been implemented in
the 'check' file in the way similar to the 'test notrun' mechanism.

Suggested-by: Roman Kagan <rkagan@virtuozzo.com>
Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Andrey Shinkevich
ce090f656c iotests: open notrun files in text mode
Replace the binary mode with the default text one when *.notrun
files are opened for skipped tests. That change is made for the
compatibility with Python 3 which returns error otherwise.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Stefan Hajnoczi
b74b1adef0 iotests: use iotests.VM in 238
Test 238 does not require the kvm accelerator.  Using the qtest
accelerator allows the test to run in both non-kvm and non-tcg
environments.

iotests.VM implicitly uses the qtest accelerator and is really the class
that this test should be using.  Switch to that instead of
qemu.QEMUMachine.

Suggested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-03-08 12:26:45 +01:00
Peter Maydell
6cb4f6db4f Python queue, 2019-02-22
Python:
 * introduce "python" directory with module namespace
 * log QEMU launch command line on qemu.QEMUMachine
 
 Acceptance Tests:
 * initrd 4GiB+ test
 * migration test
 * multi vm support in test class
 * bump Avocado version and drop "🥑 enable"
 -----BEGIN PGP SIGNATURE-----
 
 iQIcBAABCAAGBQJccE9jAAoJEGV+jTOl8gnzb1cP/j99kGbgfQJA4CftO9eRXdIm
 FKms4Z42n7KPus+/DphgfOXGYaHzPcqJQNguQYHuPlWaM3DWNU0rcFfAi/QdcZC1
 3iYMyQwiRubjnCMN0Ab4k+GhpCPW6fea6GTzyvqha4jNRhCIhx7v54GTDfxWESQp
 nqW40gAONGSG98DdFgubxg1YYqt7zlI9EVogGixe1gO9SVDkMEe7uH8tPCl9mt2m
 VjN7AeP/NTDmidiwu+2LwSpDC0UmpDAsFnxGI6rDcNx8NOnjSHkSHmtxNJ8j2uZz
 9P0ncGui+LfivdQh/yiBgrjTWXEXAx/oHKQCz7r8uJ8f60eYLFtjTHm//2G7lG48
 luLSnNKq/niM4k/vNhBQr0ByqoHHlpmqAjbmYqw7wdvImBbkXN2Gh9kjNs55S8VZ
 Z7wTceC0G7pyM3LCdFnikyCXKoRxLZ3AXQ3YXFN0PgX/IsyHVuBWBGPFkPkLwcRa
 JW3DEmwx/oeTg2MKp7iA3dGTUIarbsjp+R04erMznlLvE+NgmB8ENY8T+qZ6c+NM
 ZNyp1MH2nuTJsYxY3CkVKwPUqNSoaTLkMxvoZW5rKQdtvNinCYZpaeHuBchaHJed
 E63r0+1n9vAMH3PHDrypW5qjcjSDBOHS+8ajhr0jr2r+6grLQKYEP8q+PwubUaMq
 BsS5jOb8gLGC8ESfZxx/
 =dwff
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/cleber/tags/python-next-pull-request' into staging

Python queue, 2019-02-22

Python:
* introduce "python" directory with module namespace
* log QEMU launch command line on qemu.QEMUMachine

Acceptance Tests:
* initrd 4GiB+ test
* migration test
* multi vm support in test class
* bump Avocado version and drop "🥑 enable"

# gpg: Signature made Fri 22 Feb 2019 19:37:07 GMT
# gpg:                using RSA key 657E8D33A5F209F3
# gpg: Good signature from "Cleber Rosa <crosa@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3

* remotes/cleber/tags/python-next-pull-request:
  Acceptance tests: expect boot to extract 2GiB+ initrd with linux-v4.16
  Acceptance tests: use linux-3.6 and set vm memory to 4GiB
  tests.acceptance: adds simple migration test
  tests.acceptance: adds multi vm capability for acceptance tests
  scripts/qemu.py: log QEMU launch command line
  Introduce a Python module structure
  Acceptance tests: drop usage of "🥑 enable"

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-03-07 16:16:02 +00:00
Eric Blake
054be36054 iotests: Wait for qemu to end in 223
When iotest 223 was first written, it didn't matter if we waited for
the qemu process to clean up. But with the introduction of a later
qemu-nbd process trying to reuse the same file, there is a race where
even though the asynchronous qemu process has responded to "quit", it
has not yet had time to unlock the file and exit, resulting in:

-[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false},
-{ "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+qemu-nbd: Failed to blk_new_open 'tests/qemu-iotests/scratch/t.qcow2': Failed to get shared "write" lock
+Is another process using the image [tests/qemu-iotests/scratch/t.qcow2]?
+qemu-img: Could not open 'driver=nbd,server.type=unix,server.path=tests/qemu-iotests/scratch/qemu-nbd.sock,x-dirty-bitmap=qemu:dirty-bitmap:b': Failed to connect socket tests/qemu-iotests/scratch/qemu-nbd.sock: Connection refused
+./common.nbd: line 33: kill: (11122) - No such process

Fixes: ddd09448
Reported-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190305182908.13557-1-eblake@redhat.com>
Tested-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2019-03-06 11:05:27 -06:00
Daniel P. Berrange
b25e12daff qemu-nbd: add support for authorization of TLS clients
Currently any client which can complete the TLS handshake is able to use
the NBD server. The server admin can turn on the 'verify-peer' option
for the x509 creds to require the client to provide a x509 certificate.
This means the client will have to acquire a certificate from the CA
before they are permitted to use the NBD server. This is still a fairly
low bar to cross.

This adds a '--tls-authz OBJECT-ID' option to the qemu-nbd command which
takes the ID of a previously added 'QAuthZ' object instance. This will
be used to validate the client's x509 distinguished name. Clients
failing the authorization check will not be permitted to use the NBD
server.

For example to setup authorization that only allows connection from a client
whose x509 certificate distinguished name is

   CN=laptop.example.com,O=Example Org,L=London,ST=London,C=GB

escape the commas in the name and use:

  qemu-nbd --object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
                    endpoint=server,verify-peer=yes \
           --object 'authz-simple,id=auth0,identity=CN=laptop.example.com,,\
                     O=Example Org,,L=London,,ST=London,,C=GB' \
           --tls-creds tls0 \
           --tls-authz authz0 \
	   ....other qemu-nbd args...

NB: a real shell command line would not have leading whitespace after
the line continuation, it is just included here for clarity.

Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20190227162035.18543-2-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: split long line in --help text, tweak 233 to show that whitespace
after ,, in identity= portion is actually okay]
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-03-06 11:05:27 -06:00
Peter Maydell
093a2af7b6 nbd patches for 2019-02-25
- iotest failure fixes for tests related to NBD
 -----BEGIN PGP SIGNATURE-----
 
 iQEcBAABCAAGBQJcdW1XAAoJEKeha0olJ0NqNH8H/2DF/IVFGIx0mNlBjeu7pk8E
 /qDa4V19g6/Hyl/DTdqHi8afipewav+hNbhBrT9SCFtF+tON13HZqMoFJYcoQ+iU
 eBRwgpSPXhAMmVsoEYlYBATjq/+B8IpNEhj8uGd9VqhsnSYElgHnyIlSw+znw/KR
 t9/LTXKCRkabUgjNqzQcuV+8w8IeusjDocFIVaFZiTnUAgTXx/vIR9AGUtgwkJE0
 WFOUFTLYwy20h9J6FOzyhBG9hqM6KH9JYaKY7laJSzEanY5A6c8XrAfdI39o3oQb
 I0/GmDKos0yKAvdlXsULu/nStu8NYl+ET36Qi80Bc4Uo9XJeUx/IbioSFNnGjfY=
 =AIRl
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2019-02-25-v2' into staging

nbd patches for 2019-02-25

- iotest failure fixes for tests related to NBD

# gpg: Signature made Tue 26 Feb 2019 16:46:15 GMT
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2019-02-25-v2:
  iotests: avoid broken pipe with certtool
  iotests: ensure we print nbd server log on error
  iotests: handle TypeError for Python 3 in test 242

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-02-28 12:02:07 +00:00
Daniel P. Berrangé
3e6f45446b iotests: avoid broken pipe with certtool
When we run "certtool 2>&1 | head -1" the latter command is likely to
complete and exit before certtool has written everything it wants to
stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to
quit with broken pipe before it has finished writing the desired
output file to disk. This causes non-deterministic failures of the
iotest 233 because the certs are sometimes zero length files.
If certtool fails the "head -1" means we also lose any useful error
message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a
more flexible & reliable manner.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-3-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-02-26 10:45:37 -06:00
Daniel P. Berrangé
84f8b840a2 iotests: ensure we print nbd server log on error
If we abort the iotest early the server.log file might contain useful
information for diagnosing the problem. Ensure its contents are
displayed in this case.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-2-berrange@redhat.com>
[eblake: fix shell quoting]
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-02-26 10:45:37 -06:00
Andrey Shinkevich
42eb4591fe iotests: handle TypeError for Python 3 in test 242
The data type for bytes in Python 3 differs from the one in Python 2.
The type cast that is compatible with both versions was applied.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reported-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <1551197495-24425-1-git-send-email-andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-02-26 10:37:06 -06:00
Max Reitz
6a4e88e179 iotests: Skip 211 on insufficient memory
VDI keeps the whole bitmap in memory, and the maximum size (which is
tested here) is 2 GB.  This may not be available on all machines, and it
rarely is available when running a 32 bit build.

Fix this by making VM.run_job() return the error string if an error
occurred, and checking whether that contains "Could not allocate bmap"
in 211.  If so, the test is skipped.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190218180646.30282-1-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Stefan Hajnoczi
0482098608 iotests: add LUKS payload overhead to 178 qemu-img measure test
The previous patch includes the LUKS payload overhead into the qemu-img
measure calculation for qcow2.  Update qemu-iotests 178 to exercise this
new code path.

Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190218104525.23674-3-stefanha@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
8a57a4be83 iotests.py: s/_/-/g on keys in qmp_log()
This follows what qmp() does, so the output will correspond to the
actual QMP command.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190210145736.1486-11-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
e35792b6fd iotests: Let 045 be run concurrently
Adding a telnet monitor for no real purpose on a fixed port is not so
great.  Just use a null monitor instead.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-10-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
ac3589dc46 iotests: Filter SSH paths
8908b253c4 has implemented filtering of
remote paths for NFS, but forgot SSH.  This patch takes care of that.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-9-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
56a6e5d0ca iotests.py: Filter filename in any string value
filter_qmp_testfiles() currently filters the filename only for specific
keys.  However, there are more keys that take filenames (such as
block-commit's @top and @base, or ssh's @path), and it does not make
sense to list them all here.  "$TEST_DIR/$PID-" should have enough
entropy not to appear anywhere randomly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
011a576113 iotests.py: Add is_str()
On Python 2.x, strings are not always unicode strings.  This function
checks whether a given value is a plain string, or a unicode string (if
there is a difference).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-7-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
9ac10f2e2c iotests: Fix 207 to use QMP filters for qmp_log
Fixes: 08fcd6111e
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
c48221aa91 iotests: Fix 232 for LUKS
With IMGOPTSSYNTAX, $TEST_IMG is useless for this test (it only tests
the file-posix protocol driver).  Therefore, if $TEST_IMG_FILE is set,
use that instead.

Because this test requires the file protocol, $TEST_IMG_FILE will always
be set if $IMGOPTSSYNTAX is true.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
8f4ed6983a iotests: Remove superfluous rm from 232
This test creates no such file.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:28 +01:00
Max Reitz
10ba68d10c iotests: Fix 237 for Python 2.x
math.ceil() returns an integer on Python 3.x, but a float on Python 2.x.
range() always needs integers, so we need an explicit conversion on 2.x
(which does not hurt on 3.x).

It is not quite clear whether we want to support Python 2.x for any
prolonged time, but this may as well be fixed along with the other
issues some iotests have right now.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190210145736.1486-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
250c04f554 iotests: Re-add filename filters
A previous commit removed the default filters for qmp_log with the
intention to make them explicit; but this happened only for test 206.
There are more tests (for more exotic image formats than qcow2) which
require the filename filter, though.

Note that 237 is still broken for Python 2.x, which is fixed in the next
commit.

Fixes: f8ca8609d8
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-2-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
7b14f23149 iotests: Test json:{} filenames of internal BDSs
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-32-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
998b3a1e5a block: Purify .bdrv_refresh_filename()
Currently, BlockDriver.bdrv_refresh_filename() is supposed to both
refresh the filename (BDS.exact_filename) and set BDS.full_open_options.
Now that we have generic code in the central bdrv_refresh_filename() for
creating BDS.full_open_options, we can drop the latter part from all
BlockDriver.bdrv_refresh_filename() implementations.

This also means that we can drop all of the existing default code for
this from the global bdrv_refresh_filename() itself.

Furthermore, we now have to call BlockDriver.bdrv_refresh_filename()
after having set BDS.full_open_options, because the block driver's
implementation should now be allowed to depend on BDS.full_open_options
being set correctly.

Finally, with this patch we can drop the @options parameter from
BlockDriver.bdrv_refresh_filename(); also, add a comment on this
function's purpose in block/block_int.h while touching its interface.

This completely obsoletes blklogwrite's implementation of
.bdrv_refresh_filename().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-25-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
97e2f021f8 block: Generically refresh runtime options
Instead of having every block driver which implements
bdrv_refresh_filename() copy all of the strong runtime options over to
bs->full_open_options, implement this process generically in
bdrv_refresh_filename().

This patch only adds this new generic implementation, it does not remove
the old functionality. This is done in a follow-up patch.

With this patch, some superfluous information (that should never have
been there) may be removed from some JSON filenames, as can be seen in
the change to iotests 110's and 228's reference outputs.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-24-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
c0625e8092 iotests: Add quorum case to test 110
Test 110 tests relative backing filenames for complex BDS trees.  Now
that the originally supposedly failing test passes, let us add a new
failing test: Quorum can never work automatically (without detecting
whether all child nodes have the same base directory, but that would be
rather inconsistent behavior).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-21-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
8df686165b block: Use bdrv_dirname() for relative filenames
bdrv_get_full_backing_filename_from_filename() breaks down when it comes
to JSON filenames. Using bdrv_dirname() as the basis is better because
since we have BDS, we can descend through the BDS tree to the protocol
layer, which gives us a greater probability of finding a non-JSON name;
also, bdrv_dirname() is more correct as it allows block drivers to
override the generation of that directory name in a protocol-specific
way.

We still need to keep bdrv_get_full_backing_filename_from_filename(),
though, because it has valid callers which need it during image creation
when no BDS is available yet.

This makes a test case in qemu-iotest 110, which was supposed to fail,
work. That is actually good, but we need to change the reference output
(and the comment in 110) accordingly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-20-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:27 +01:00
Max Reitz
0f62cd8204 iotests: Add test for backing file overrides
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-9-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:26 +01:00
Max Reitz
ef7afd6399 iotests.py: Add node_info()
This function queries a node; since we cannot do that right now, it
executes query-named-block-nodes and returns the matching node's object.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190201192935.18394-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:26 +01:00
Max Reitz
f2ea0b2082 iotests.py: Add filter_imgfmt()
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-7-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:25 +01:00
Max Reitz
909936234c block: Respect backing bs in bdrv_refresh_filename
Basically, bdrv_refresh_filename() should respect all children of a
BlockDriverState. However, generally those children are driver-specific,
so this function cannot handle the general case. On the other hand,
there are only few drivers which use other children than @file and
@backing (that being vmdk, quorum, and blkverify).

Most block drivers only use @file and/or @backing (if they use any
children at all). Both can be implemented directly in
bdrv_refresh_filename.

The user overriding the file's filename is already handled, however, the
user overriding the backing file is not. If this is done, opening the
BDS with the plain filename of its file will not be correct, so we may
not set bs->exact_filename in that case.

iotest 051 contains test cases for overriding the backing file, and so
its output changes with this patch applied.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-02-25 15:11:25 +01:00
Cleber Rosa
8f8fd9edba Introduce a Python module structure
This is a simple move of Python code that wraps common QEMU
functionality, and are used by a number of different tests
and scripts.

By treating that code as a real Python module, we can more easily:
 * reuse code
 * have a proper place for the module's own unittests
 * apply a more consistent style
 * generate documentation

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20190206162901.19082-2-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2019-02-22 14:07:01 -05:00
Eric Blake
f67cf661f8 dirty-bitmap: Expose persistent flag to 'query-block'
Since qemu currently doesn't flush persistent bitmaps to disk until
shutdown (which might be MUCH later), it's useful if 'query-block'
at least shows WHICH bitmaps will (eventually) make it to persistent
storage.  Update affected iotests.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190204210512.27458-1-eblake@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2019-02-19 17:49:43 -05:00
Andrey Shinkevich
ddd113beed qcow2: list of bitmaps new test 242
A new test file 242 added to the qemu-iotests set. It checks
the format of qcow2 specific information for the new added
section that lists details of bitmaps.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <1549638368-530182-4-git-send-email-andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: pep8 compliance, avoid trailing blank line]
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-02-11 14:35:43 -06:00
John Snow
039be85c41 iotests/236: fix transaction kwarg order
It's not enough to order the kwargs for consistent QMP log output,
we must also sort any sub-dictionaries in lists that appear as values.

Reported-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:45 +01:00
Max Reitz
fff2388d5d iotests: Filter second BLOCK_JOB_ERROR from 229
Without this filter, this test sometimes fails.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:45 +01:00
Alberto Garcia
eb97813ff5 virtio-scsi: Forbid devices with different iothreads sharing a blockdev
This patch forbids attaching a disk to a SCSI device if its using a
different AioContext. Test case included.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:45 +01:00
Alberto Garcia
3ff35ba391 scsi-disk: Acquire the AioContext in scsi_*_realize()
This fixes a crash when attaching two disks with the same blockdev to
a SCSI device that is using iothreads. Test case included.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:45 +01:00
Alberto Garcia
a6f230c8d1 virtio-scsi: Move BlockBackend back to the main AioContext on unplug
This fixes a crash when attaching a disk to a SCSI device using
iothreads, then detaching it and reattaching it again. Test case
included.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:45 +01:00
Kevin Wolf
4a960ece17 vmdk: Reject excess extents in blockdev-create
Clarify that the number of extents provided in BlockdevCreateOptionsVmdk
must match the number of extents that will actually be used. Providing
more extents will result in an error now.

This requires adapting the test case to provide the right number of
extents.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
2019-02-01 13:46:44 +01:00
Kevin Wolf
1c4e7b640b iotests: Add VMDK tests for blockdev-create
Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:44 +01:00
Fam Zheng
bab4feb2fa iotests: Filter cid numbers in VMDK extent info
Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:44 +01:00
Max Reitz
9a378495c3 iotests: Make 234 stable
This test waits for a MIGRATION event with status=completed on the
source VM before querying the migration status on both source and
destination.  However, just because the source says migration has
completed does not mean the destination thinks the same.  Therefore, in
some cases, the destination VM may still report "active" instead of
"completed" when asked for its migration status.

Fix this by enabling migration events on both VMs and waiting until both
source and destination emit a status=completed MIGRATION event.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:44 +01:00
yuchenlin
76f1cf0a5e qemu-iotests: add test case for dmg
Recently, some bugs in dmg file have been fixed. To prevent reading dmg
is broken someday in the future, add a simple test which ensures the
conversion from dmg to raw should not hang or face any I/O error.

Signed-off-by: yuchenlin <npes87184@gmail.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:44 +01:00
Alberto Garcia
67b24427fe mirror: Block the source BlockDriverState in mirror_start_job()
The mirror_start_job() function used for the commit-active job blocks
the source, target and all intermediate nodes for the duration of the
job.

   target <- intermediate <- source

Since 4ef85a9c23 this function creates a dummy mirror_top_bs that
goes on top of the source node, and it is this dummy node that gets
blocked instead. The source node is never blocked or added to the
job's list of nodes.

   target <- intermediate <- source <- mirror_top

At the moment I don't think it is possible to exploit this problem
because any additional job on 'source' would either be forbidden for
other reasons or it would need to involve an additional node that is
blocked, causing an error.

This can be seen in the error messages, however, because they never
refer to the source node being blocked:

  $ qemu-img create -f qcow2 hd0.qcow2 1M
  $ qemu-img create -f qcow2 -b hd0.qcow2 hd1.qcow2
  $ qemu-io -c 'write 0 1M' hd0.qcow2
  $ $QEMU -drive if=none,file=hd1.qcow2,node-name=hd1
  { "execute": "qmp_capabilities" }
  { "execute": "block-commit", "arguments": {"device": "hd1", "speed": 256}}
  { "execute": "block-stream", "arguments": {"device": "hd1"}}
  { "error": {"class": "GenericError",
    "desc": "Node 'hd0' is busy: block device is in use by block job: commit"}}

After this patch the error message refers to 'hd1', as it should.

The expected output of iotest 141 also needs to be updated for the
same reason.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-02-01 13:46:44 +01:00
Max Reitz
908b30164b iotests: Allow 147 to be run concurrently
To do this, we need to allow creating the NBD server on various ports
instead of a single one (which may not even work if you run just one
instance, because something entirely else might be using that port).

So we just pick a random port in [32768, 32768 + 1024) and try to create
a server there.  If that fails, we just retry until something sticks.

For the IPv6 test, we need a different range, though (just above that
one).  This is because "localhost" resolves to both 127.0.0.1 and ::1.
This means that if you bind to it, it will bind to both, if possible, or
just one if the other is already in use.  Therefore, if the IPv6 test
has already taken [::1]:some_port and we then try to take
localhost:some_port, that will work -- only the second server will be
bound to 127.0.0.1:some_port alone and not [::1]:some_port in addition.
So we have two different servers on the same port, one for IPv4 and one
for IPv6.

But when we then try to connect to the server through
localhost:some_port, we will always end up at the IPv6 one (as long as
it is up), and this may not be the one we want.

Thus, we must make sure not to create an IPv6-only NBD server on the
same port as a normal "dual-stack" NBD server -- which is done by using
distinct port ranges, as explained above.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-01-31 00:44:55 +01:00
Max Reitz
dfadac9a37 iotests: Bind qemu-nbd to localhost in 147
By default, qemu-nbd binds to 0.0.0.0.  However, we then proceed to
connect to "localhost".  Usually, this works out fine; but if this test
is run concurrently, some other test function may have bound a different
server to ::1 (on the same port -- you can bind different serves to the
same port, as long as one is on IPv4 and the other on IPv6).

So running qemu-nbd works, it can bind to 0.0.0.0:NBD_PORT.  But
potentially a concurrent test has successfully taken [::1]:NBD_PORT.  In
this case, trying to connect to "localhost" will lead us to the IPv6
instance, where we do not want to end up.

Fix this by just binding to "localhost".  This will make qemu-nbd error
out immediately and not give us cryptic errors later.

(Also, it will allow us to just try a different port as of a future
patch.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-01-31 00:44:49 +01:00
Max Reitz
e1e6eccd49 iotests.py: Add qemu_nbd_pipe()
In some cases, we may want to deal with qemu-nbd errors (e.g. by
launching it in a different configuration until it no longer throws
any).  In that case, we do not want its output ending up in the test
output.

It may still be useful for handling the error, though, so add a new
function that works basically like qemu_nbd(), only that it returns the
qemu-nbd output instead of making it end up in the log.  In contrast to
qemu_img_pipe(), it does still return the exit code as well, though,
because that is even more important for error handling.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2019-01-31 00:44:29 +01:00
Stefan Hajnoczi
202277f43d iotests: add 238 for throttling tgm unregister iothread segfault
Hot-unplug a scsi-hd using an iothread.  The previous patch fixes a
segfault in this scenario.

This patch adds a regression test.

Suggested-by: Alberto Garcia <berto@igalia.com>
Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190114133257.30299-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-24 10:02:28 +00:00
Eric Blake
ddd09448fd iotests: Enhance 223, 233 to cover 'qemu-nbd --list'
Any good new feature deserves some regression testing :)
Coverage includes:
- 223: what happens when there are 0 or more than 1 export,
proof that we can see multiple contexts including qemu:dirty-bitmap
- 233: proof that we can list over TLS, and that mix-and-match of
plain/TLS listings will behave sanely

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20190117193658.16413-22-eblake@redhat.com>
2019-01-21 15:49:52 -06:00
Eric Blake
d08980511d iotests: Make 233 output more reliable
We have a race between the nbd server and the client both trying
to report errors at once which can make the test sometimes fail
if the output lines swap order under load.  Break the race by
collecting server messages into a file and then replaying that
at the end of the test.

We may yet want to fix the server to not output ANYTHING for a
client action except when -v was used (to avoid malicious clients
from being able to DoS a server by filling up its logs), but that
is saved for a future patch.

Signed-off-by: Eric Blake <eblake@redhat.com>
CC: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190117193658.16413-2-eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-01-21 15:49:51 -06:00
Eric Blake
636192c4b6 qemu-nbd: Add --bitmap=NAME option
Having to fire up qemu, then use QMP commands for nbd-server-start
and nbd-server-add, just to expose a persistent dirty bitmap, is
rather tedious.  Make it possible to expose a dirty bitmap using
just qemu-nbd (of course, for now this only works when qemu-nbd is
visiting a BDS formatted as qcow2).

Of course, any good feature also needs unit testing, so expand
iotest 223 to cover it.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190111194720.15671-9-eblake@redhat.com>
2019-01-14 10:09:46 -06:00
Eric Blake
5fcbeb0681 nbd: Allow bitmap export during QMP nbd-server-add
With the experimental x-nbd-server-add-bitmap command, there was
a window of time where an NBD client could see the export but not
the associated dirty bitmap, which can cause a client that planned
on using the dirty bitmap to be forced to treat the entire image
as dirty as a safety fallback.  Furthermore, if the QMP client
successfully exports a disk but then fails to add the bitmap, it
has to take on the burden of removing the export.  Since we don't
allow changing the exposed dirty bitmap (whether to a different
bitmap, or removing advertisement of the bitmap), it is nicer to
make the bitmap tied to the export at the time the export is
created, with automatic failure to export if the bitmap is not
available.

The experimental command included an optional 'bitmap-export-name'
field for remapping the name exposed over NBD to be different from
the bitmap name stored on disk.  However, my libvirt demo code
for implementing differential backups on top of persistent bitmaps
did not need to take advantage of that feature (it is instead
possible to create a new temporary bitmap with the desired name,
use block-dirty-bitmap-merge to merge one or more persistent
bitmaps into the temporary, then associate the temporary with the
NBD export, if control is needed over the exported bitmap name).
Hence, I'm not copying that part of the experiment over to the
stable addition. For more details on the libvirt demo, see
https://www.redhat.com/archives/libvir-list/2018-October/msg01254.html,
https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat

This patch focuses on the user interface, and reduces (but does
not completely eliminate) the window where an NBD client can see
the export but not the dirty bitmap, with less work to clean up
after errors.  Later patches will add further cleanups now that
this interface is declared stable via a single QMP command,
including removing the race window.

Update test 223 to use the new interface.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20190111194720.15671-6-eblake@redhat.com>
2019-01-14 10:09:46 -06:00
Eric Blake
702aa50d61 nbd: Only require disabled bitmap for read-only exports
Our initial implementation of x-nbd-server-add-bitmap put
in a restriction because of incremental backups: in that
usage, we are exporting one qcow2 file (the temporary overlay
target of a blockdev-backup sync:none job) and a dirty bitmap
owned by a second qcow2 file (the source of the
blockdev-backup, which is the backing file of the temporary).
While both qcow2 files are still writable (the target in
order to capture copy-on-write of old contents, and the
source in order to track live guest writes in the meantime),
the NBD client expects to see constant data, including the
dirty bitmap.  An enabled bitmap in the source would be
modified by guest writes, which is at odds with the NBD
export being a read-only constant view, hence the initial
code choice of enforcing a disabled bitmap (the intent is
that the exposed bitmap was disabled in the same transaction
that started the blockdev-backup job, although we don't want
to track enough state to actually enforce that).

However, consider the case of a bitmap contained in a read-only
node (including when the bitmap is found in a backing layer of
the active image).  Because the node can't be modified, the
bitmap won't change due to writes, regardless of whether it is
still enabled.  Forbidding the export unless the bitmap is
disabled is awkward, paritcularly since we can't change the
bitmap to be disabled (because the node is read-only).

Alternatively, consider the case of live storage migration,
where management directs the destination to create a writable
NBD server, then performs a drive-mirror from the source to
the target, prior to doing the rest of the live migration.
Since storage migration can be time-consuming, it may be wise
to let the destination include a dirty bitmap to track which
portions it has already received, where even if the migration
is interrupted and restarted, the source can query the
destination block status in order to potentially minimize
re-sending data that has not changed in the meantime on a
second attempt. Such code has not been written, and might not
be trivial (after all, a cluster being marked dirty in the
bitmap does not necessarily guarantee it has the desired
contents), but it makes sense that letting an active dirty
bitmap be exposed and changing alongside writes may prove
useful in the future.

Solve both issues by gating the restriction against a
disabled bitmap to only happen when the caller has requested
a read-only export, and where the BDS that owns the bitmap
(whether or not it is the BDS handed to nbd_export_new() or
from its backing chain) is still writable.  We could drop
the check altogether (if management apps are prepared to
deal with a changing bitmap even on a read-only image), but
for now keeping a check for the read-only case still stands
a chance of preventing management errors.

Update iotest 223 to show the looser behavior by leaving
a bitmap enabled the whole run; note that we have to tear
down and re-export a node when handling an error.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190111194720.15671-4-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-01-14 10:09:46 -06:00
Eric Blake
7801c3a7fd nbd: Forbid nbd-server-stop when server is not running
Since we already forbid other nbd-server commands when not
in the right state, it is unlikely that any caller was relying
on a second stop to behave as a silent no-op.  Update iotest
223 to show the improved behavior.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190111194720.15671-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-01-14 10:09:46 -06:00
Eric Blake
2d2fd67428 nbd: Add some error case testing to iotests 223
Testing success paths is important, but it's also nice to highlight
expected failure handling, to show that we don't crash, and so that
upcoming tests that change behavior can demonstrate the resulting
effects on error paths.

Add the following errors:
Attempting to export without a running server
Attempting to start a second server
Attempting to export a bad node name
Attempting to export a name that is already exported
Attempting to export an enabled bitmap
Attempting to remove an already removed export
Attempting to quit server a second time

All of these properly complain except for a second server-stop,
which will be fixed next.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190111194720.15671-2-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-01-14 10:09:46 -06:00
John Snow
14da540f2a iotests: add iotest 236 for testing bitmap merge
New interface, new smoke test.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20181221093529.23855-12-jsnow@redhat.com>
[eblake: fix last-minute change to echo text]
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-01-14 10:09:46 -06:00
John Snow
55cd64eab5 iotests: implement pretty-print for log and qmp_log
If iotests have lines exceeding >998 characters long, git doesn't
want to send it plaintext to the list. We can solve this by allowing
the iotests to use pretty printed QMP output that we can match against
instead.

As a bonus, it's much nicer for human eyes too.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20181221093529.23855-11-jsnow@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-01-14 10:09:46 -06:00
John Snow
08fcd6111e iotests: change qmp_log filters to expect QMP objects only
As laid out in the previous commit's message:

```
Several places in iotests deal with serializing objects into JSON
strings, but to add pretty-printing it seems desirable to localize
all of those cases.

log() seems like a good candidate for that centralized behavior.
log() can already serialize json objects, but when it does so,
it assumes filters=[] operates on QMP objects, not strings.

qmp_log currently operates by dumping outgoing and incoming QMP
objects into strings and filtering them assuming that filters=[]
are string filters.
```

Therefore:

Change qmp_log to treat filters as if they're always qmp object filters,
then change the logging call to rely on log()'s ability to serialize QMP
objects, so we're not duplicating that effort.

Add a qmp version of filter_testfiles and adjust the only caller using
it for qmp_log to use the qmp version.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20181221093529.23855-10-jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-01-14 10:09:46 -06:00
John Snow
f8ca8609d8 iotests: remove default filters from qmp_log
Several places in iotests deal with serializing objects into JSON
strings, but to add pretty-printing it seems desirable to localize
all of those cases.

log() seems like a good candidate for that centralized behavior.
log() can already serialize json objects, but when it does so,
it assumes filters=[] operates on QMP objects, not strings.

qmp_log currently operates by dumping outgoing and incoming QMP
objects into strings and filtering them assuming that filters=[]
are string filters.

To have qmp_log use log's serialization, qmp_log will need to
accept only qmp filters, not text filters.

However, only a single caller of qmp_log actually requires any
filters at all. I remove the default filter and add it explicitly
to the caller in preparation for refactoring qmp_log to use rich
filters instead.

test 206 is amended to name the filter explicitly and the default
is removed.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20181221093529.23855-9-jsnow@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-01-14 10:09:46 -06:00
John Snow
0706e87d72 iotests: add qmp recursive sorting function
Python before 3.6 does not sort dictionaries (including kwargs).
Therefore, printing QMP objects involves sorting the keys to have
a predictable ordering in the iotests output. This means that
iotests output will sometimes show arguments in an order not
specified by the test author.

Presently, we accomplish this by using json.dumps' sort_keys argument,
where we only serialize the arguments dictionary, but not the command.

However, if we want to pretty-print QMP objects being sent to the
QEMU process, we need to build the entire command before logging it.
Ordinarily, this would then involve "arguments" being sorted above
"execute", which would necessitate a rather ugly and harder-to-read
change to many iotests outputs.

To facilitate pretty-printing AND maintaining predictable output AND
having "arguments" sort after "execute", add a custom sort function
that takes a dictionary and recursively builds an OrderedDict that
maintains the specific key order we wish to see in iotests output.

The qmp_log function uses this to build a QMP object that keeps
"execute" above "arguments", but sorts all keys and keys in any
subdicts in "arguments" lexicographically to maintain consistent
iotests output, with no incompatible changes to any current test.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20181221093529.23855-8-jsnow@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2019-01-14 10:09:46 -06:00