Commit Graph

68571 Commits

Author SHA1 Message Date
Alistair Francis
0feb4a7129
riscv: plic: Fix incorrect irq calculation
This patch fixes four different things, to maintain bisectability they
have been merged into a single patch. The following fixes are below:

sifive_plic: Fix incorrect irq calculation
The irq is incorrectly calculated to be off by one. It has worked in the
past as the priority_base offset has also been set incorrectly. We are
about to fix the priority_base offset so first first the irq
calculation.

sifive_u: Fix PLIC priority base offset and numbering
According to the FU540 manual the PLIC source priority address starts at
an offset of 0x04 and not 0x00. The same manual also specifies that the
PLIC only has 53 source priorities. Fix these two incorrect header
files.

We also need to over extend the plic_gpios[] array as the PLIC sources
count from 1 and not 0.

riscv: sifive_e: Fix PLIC priority base offset
According to the FE31 manual the PLIC source priority address starts at
an offset of 0x04 and not 0x00.

riscv: virt: Fix PLIC priority base offset
Update the virt offsets based on the newly updated SiFive U and SiFive E
offsets.

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2019-04-04 16:36:19 -07:00
Peter Maydell
061b51e919 Update version for v4.0.0-rc2 release
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 17:01:20 +01:00
Peter Maydell
37301a8d0f Miscellaneous patches for 2019-04-02
-----BEGIN PGP SIGNATURE-----
 
 iQIcBAABAgAGBQJco01zAAoJEDhwtADrkYZTzJkP/1izIRaWBMOwhwwDORgdxIro
 Qgd8cgXSiC2WAPtagEdaQQRtYdEwlBTCcnRBqXQixWnJ52FnaijwRdUx2cwTXP2x
 rX0G0inhS7xE7P9Zr9sPhTMafLN5kUyFsm5WsZZbLDcimXMZtAYm7vZAdvneMfMj
 MjOu7FXLtlwvU2Gg1z8Kog3+ShkafHsUm1lQnAp2YQSsSCHRIazTuyyqIpqZEyUt
 ie7zI7GMA0arIJfNQSfrgLiquD4Rj+Cn4RHp4RnKeGTExhNrZJcSk3+IUz25MJIK
 O+Rlftr+iHk/FzoAa0NeASaNuI+cUjbxTEJwvz+sbulSVLBX58pP22vxh4wauslm
 VjrHzj5xRo6w6RKPmkJn0CgJnIAGc7Ulzyt/gpVT+Qm2SRY5EnV3KoEtYcU7xtHw
 uK8YRWThadkG+nHc2LOR8DjjT/h+JitpjwvPadt2NZpoGZkrvGuKZAJ4TKf8Yv0z
 eiyfxV4trZisCEInIcMOVa7cAhO7i7/eUO1/IFNrD+X++/BTWFAlW3cqj41PUl+E
 exGWHHP+Z08TFV2kKobmee1xrv1RRhdFYk8BcWHS83fxqHNGCoPHE/dfB/kJGCMr
 96JE6LTDVIRmMHQjzs0nMHtTZPfaONvdjHuv9kuQj6wnbjYAE8z0ZlcnEa9sSZZQ
 CcaSZLqGJstkoUJwRZQx
 =yOO7
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/armbru/tags/pull-misc-2019-04-02' into staging

Miscellaneous patches for 2019-04-02

# gpg: Signature made Tue 02 Apr 2019 12:54:27 BST
# gpg:                using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-misc-2019-04-02:
  accel: Unbreak accelerator fallback
  vl: Document dependencies hiding in global and compat props
  migration: Support adding migration blockers earlier
  Revert "migration: move only_migratable to MigrationState"
  Revert "vl: Fix to create migration object before block backends again"
  qapi/migration.json: Rename COLOStatus last_mode to last-mode
  qapi/migration.json: Fix ColoStatus member last_mode's version
  vl: Fix error location of positional arguments

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 16:13:59 +01:00
Peter Maydell
436960c959 filemon: various fixes / improvements to file monitor for USB MTP
Ensure watch IDs unique within a monitor and avoid integer wraparound
 issues when many watches are set & unset over time.
 -----BEGIN PGP SIGNATURE-----
 
 iQIcBAABCAAGBQJco1tUAAoJEL6G67QVEE/f9xYP/2OTsAk6dHCcWqc///jXN0JW
 OLcnpmjuh5KgUmOrs7NIRYRo2DARrhNDDXckHjDuX2HmnzoTB5p7DXLWMPgAzxj6
 o83yxYT0Qqw79wxqbDmJ1y3kUVka1r94wCmmvUmxQZjLbDk8u2ytNUDbEOVlM0VJ
 R7T800j8PhbY1pg9PiJJZonm7dxZAVNSCF4AgkrJVIes6MqhvNUhVsck5q/CC3yi
 0jkglsiVTyXjKpJsioM9VhQXeqiHfFl3qjcPi0XQ99lVpTMFysNd4UdnVXhNs9NI
 Xe52yQ7NG3fEggUivtFBmvVplNOg09K6U2vkyk9z8S1WTLZsbabwvwSvkVPW3Ty0
 QaDgI6wNSiBb7Lfl2bbUR6X3j9QXUn8bhlczBsorS4uU4+NdHGyERsZLKLZazYYz
 5t+xm4YBGJhtlt8IglNBjtaJFBuqgjrBEuBkfpevB2ocioWnU6APOdwWkKlSe3+r
 EuKFUcbhRiRAGO05DjntNkePigb66IVkSiJXd4uSid/+akn5kQ19PSD8tXdfvZ7h
 8HYTnWZa3OSLd7tZlwRzimnKuVSIsSnj+NobkIs8HSS3N7oN1t4FC5jH9D6GmK0R
 djQs6a5mHQbonrc0dXxlb+P8u6I+ppxznPpfL/aQJ4pFM18V6gmcZEQluhwALNe9
 rdPUnCudC/9SUNpMf3rI
 =qriK
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/berrange/tags/filemon-next-pull-request' into staging

filemon: various fixes / improvements to file monitor for USB MTP

Ensure watch IDs unique within a monitor and avoid integer wraparound
issues when many watches are set & unset over time.

# gpg: Signature made Tue 02 Apr 2019 13:53:40 BST
# gpg:                using RSA key BE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg:                 aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E  8E3F BE86 EBB4 1510 4FDF

* remotes/berrange/tags/filemon-next-pull-request:
  filemon: fix watch IDs to avoid potential wraparound issues
  filemon: ensure watch IDs are unique to QFileMonitor scope
  tests: refactor file monitor test to make it more understandable

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 14:52:17 +01:00
Peter Maydell
9a363f0bcc Block layer patches:
- file-posix: Ignore unlock failure instead of crashing
 - gluster: Limit the transfer size to 512 MiB
 - stream: Fix backing chain freezing
 - qemu-img: Enable BDRV_REQ_MAY_UNMAP for zero writes in convert
 - iotests fixes
 -----BEGIN PGP SIGNATURE-----
 
 iQIcBAABAgAGBQJco1nvAAoJEH8JsnLIjy/WKSAP/AyAGt4DFbsfVq1SbX2MB/3F
 QtLi3q3Yjy71OaNyMJsqgd8dxoqO1bWKC7MmLxwFZwvwmXhgq7EVbJ6vJ7kzuHw5
 FMsVavm61mL/vsKypFieYHbahiKC31R7sw4Vl7yeKG02oDgi/0SnIQxxO0rmIZxG
 Jx+p5tBDmrJ2WpgCWI0yr3VuYeJ/gJyw8la3Ubf1OzTBuTyfg94vSCsECX7GWm1w
 0YkcRODct96mmqKCf6haHXQs9tTa6FnC9fsReenYcQg7nyqTttyXJz08fuqGQwWm
 sNcmMp2/YxAg2Z85Aq21fmTkXbGopEHtoC2fg71WET1DzpqMvvJPFBIOneBSzUon
 KHrZfZnziAzUESHDqenCQcHlbtWXxPY+7H0lojvhsLVzyEns6eLOSer093HqAh4s
 +av55+ucTUYRl9Uk8/vFPdFBh0bNJRJB2yazp0Fk5Yoo8y70TXePQ8ZkpjH9VSpb
 uQXy+FBc7LXYHB1KW/vhtuVoKgftbHi29rQ3imb9y6HYuJoJYZS9uGgrFpzF0YBN
 /mBG17YqE8Imfuavt0WQqNLSub/km9eqFrejnMqDBka1hZblecOB+3Ujcud5C7Vk
 HtKDdr2/Z+M1KApFMKhNZzEksXwhOVQ6VVZZfRGCjc/EXMOZ2gg8YlqOtotUzA0i
 vV2wMgpqE2Ihx1gIIqw/
 =a4LV
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Ignore unlock failure instead of crashing
- gluster: Limit the transfer size to 512 MiB
- stream: Fix backing chain freezing
- qemu-img: Enable BDRV_REQ_MAY_UNMAP for zero writes in convert
- iotests fixes

# gpg: Signature made Tue 02 Apr 2019 13:47:43 BST
# gpg:                using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  tests/qemu-iotests/235: Allow fallback to tcg
  block: test block-stream with a base node that is used by block-commit
  block: freeze the backing chain earlier in stream_start()
  block: continue until base is found in bdrv_freeze_backing_chain() et al
  block/file-posix: do not fail on unlock bytes
  tests/qemu-iotests: Remove redundant COPYING file
  block/gluster: limit the transfer size to 512 MiB
  qemu-img: Enable BDRV_REQ_MAY_UNMAP in convert
  iotests: Fix test 200 on s390x without virtio-pci

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 14:03:11 +01:00
Daniel P. Berrangé
b4682a63f8 filemon: fix watch IDs to avoid potential wraparound issues
Watch IDs are allocated from incrementing a int counter against
the QFileMonitor object. In very long life QEMU processes with
a huge amount of USB MTP activity creating & deleting directories
it is just about conceivable that the int counter can wrap
around. This would result in incorrect behaviour of the file
monitor watch APIs due to clashing watch IDs.

Instead of trying to detect this situation, this patch changes
the way watch IDs are allocated. It is turned into an int64_t
variable where the high 32 bits are set from the underlying
inotify "int" ID. This gives an ID that is guaranteed unique
for the directory as a whole, and we can rely on the kernel
to enforce this. QFileMonitor then sets the low 32 bits from
a per-directory counter.

The USB MTP device only sets watches on the directory as a
whole, not files within, so there is no risk of guest
triggered wrap around on the low 32 bits.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2019-04-02 13:52:02 +01:00
Daniel P. Berrangé
ff3dc8fefe filemon: ensure watch IDs are unique to QFileMonitor scope
The watch IDs are mistakenly only unique within the scope of the
directory being monitored. This is not useful for clients which are
monitoring multiple directories. They require watch IDs to be unique
globally within the QFileMonitor scope.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Bandan Das <bsd@redhat.com>
Reviewed-by: Bandan Das <bsd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2019-04-02 13:46:33 +01:00
Daniel P. Berrangé
b26c3f9cbd tests: refactor file monitor test to make it more understandable
The current file monitor unit tests are too clever for their own good
making it hard to understand the desired output.

Instead of trying to infer the expected events, explicitly list the
events we expect in the operation sequence.

Instead of dynamically building a matrix of tests, just have one giant
operation sequence that validates all scenarios in a single test.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2019-04-02 13:46:33 +01:00
Markus Armbruster
79b9d4bde7 accel: Unbreak accelerator fallback
When the user specifies a list of accelerators, we pick the first one
that initializes successfully.  Recent commit 1a3ec8c156 broke that.
Reproducer:

    $ qemu-system-x86_64 --machine accel=xen:tcg
    xencall: error: Could not obtain handle on privileged command interface: No such file or directory
    xen be core: xen be core: can't open xen interface
    can't open xen interface
    qemu-system-x86_64: failed to initialize Xen: Operation not permitted
    qemu-system-x86_64: /home/armbru/work/qemu/qom/object.c:436: object_set_accelerator_compat_props: Assertion `!object_compat_props[0]' failed.

Root cause: we register accelerator compat properties even when the
accelerator fails.  The failed assertion is
object_set_accelerator_compat_props() telling us off.  Fix by calling
it only for the accelerator that succeeded.

Fixes: 1a3ec8c156
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20190401090827.20793-6-armbru@redhat.com>
2019-04-02 13:50:09 +02:00
Markus Armbruster
0427b6257e vl: Document dependencies hiding in global and compat props
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190401090827.20793-5-armbru@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
2019-04-02 13:50:01 +02:00
Markus Armbruster
daff7f0bbe migration: Support adding migration blockers earlier
migrate_add_blocker() asserts we have a current_migration object, in
migrate_get_current().  We do only after migration_object_init().

This contributes to the following dependency cycle:

* configure_blockdev() must run before machine_set_property()
  so machine properties can refer to block backends

* machine_set_property() before configure_accelerator()
  so machine properties like kvm-irqchip get applied

* configure_accelerator() before migration_object_init()
  so that Xen's accelerator compat properties get applied.

* migration_object_init() before configure_blockdev()
  so configure_blockdev() can add migration blockers

The cycle was closed when recent commit cda4aa9a5a "Create block
backends before setting machine properties" added the first
dependency, and satisfied it by violating the last one.  Broke block
backends that add migration blockers, as demonstrated by qemu-iotests
055.

To fix it, break the last dependency: make migrate_add_blocker()
usable before migration_object_init().

The previous commit already removed the use of migrate_get_current()
from migrate_add_blocker() itself.  Didn't quite do the trick, as
there's another one hiding in migration_is_idle().

The use there isn't actually necessary: when no migration object has
been created yet, migration is surely idle.  Make migration_is_idle()
return true then.

Fixes: cda4aa9a5a
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190401090827.20793-4-armbru@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
2019-04-02 13:49:36 +02:00
Markus Armbruster
811f865271 Revert "migration: move only_migratable to MigrationState"
This reverts commit 3df663e575.
This reverts commit b605c47b57.

Command line option --only-migratable is for disallowing any
configuration that can block migration.

Initially, --only-migratable set global variable @only_migratable.

Commit 3df663e575 "migration: move only_migratable to MigrationState"
replaced it by MigrationState member @only_migratable.  That was a
mistake.

First, it doesn't make sense on the design level.  MigrationState
captures the state of an individual migration, but --only-migratable
isn't a property of an individual migration, it's a restriction on
QEMU configuration.  With fault tolerance, we could have several
migrations at once.  --only-migratable would certainly protect all of
them.  Storing it in MigrationState feels inappropriate.

Second, it contributes to a dependency cycle that manifests itself as
a bug now.

Putting @only_migratable into MigrationState means its available only
after migration_object_init().

We can't set it before migration_object_init(), so we delay setting it
with a global property (this is fixup commit b605c47b57 "migration:
fix handling for --only-migratable").

We can't get it before migration_object_init(), so anything that uses
it can only run afterwards.

Since migrate_add_blocker() needs to obey --only-migratable, any code
adding migration blockers can run only afterwards.  This contributes
to the following dependency cycle:

* configure_blockdev() must run before machine_set_property()
  so machine properties can refer to block backends

* machine_set_property() before configure_accelerator()
  so machine properties like kvm-irqchip get applied

* configure_accelerator() before migration_object_init()
  so that Xen's accelerator compat properties get applied.

* migration_object_init() before configure_blockdev()
  so configure_blockdev() can add migration blockers

The cycle was closed when recent commit cda4aa9a5a "Create block
backends before setting machine properties" added the first
dependency, and satisfied it by violating the last one.  Broke block
backends that add migration blockers.

Moving @only_migratable into MigrationState was a mistake.  Revert it.

This doesn't quite break the "migration_object_init() before
configure_blockdev() dependency, since migrate_add_blocker() still has
another dependency on migration_object_init().  To be addressed the
next commit.

Note that the reverted commit made -only-migratable sugar for -global
migration.only-migratable=on below the hood.  Documentation has only
ever mentioned -only-migratable.  This commit removes the arcane &
undocumented alternative to -only-migratable again.  Nobody should be
using it.

Conflicts:
	include/migration/misc.h
	migration/migration.c
	migration/migration.h
	vl.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190401090827.20793-3-armbru@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
2019-04-02 13:38:05 +02:00
Markus Armbruster
2fa23277d5 Revert "vl: Fix to create migration object before block backends again"
This reverts commit e60483f2f8.

Recent commit cda4aa9a5a moved block backend creation before machine
property evaluation.  This broke block backends registering migration
blockers.  Commit e60483f2f8 fixed it by moving migration object
creation before block backend creation.  This broke migration with
Xen.  Turns out we need to configure the accelerator before we create
the migration object so that Xen's accelerator compat properties get
applied.  Revert the flawed commit.  This fixes the Xen regression,
but brings back the block backend regression.  The next commits will
fix it again.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190401090827.20793-2-armbru@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
2019-04-02 13:35:00 +02:00
Zhang Chen
5cc8f9eb7a qapi/migration.json: Rename COLOStatus last_mode to last-mode
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Message-Id: <20190402085521.17973-1-chen.zhang@intel.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rephrased]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2019-04-02 13:32:15 +02:00
Zhang Chen
966c0d4932 qapi/migration.json: Fix ColoStatus member last_mode's version
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Message-Id: <20190326174510.13303-1-chen.zhang@intel.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[Commit message tweaked as per Eric's review]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2019-04-02 13:30:25 +02:00
Markus Armbruster
17f30eae12 vl: Fix error location of positional arguments
We blame badness in positional arguments on the last option argument:

    $ qemu-system-x86_64 -vnc :1 bad.img
    qemu-system-x86_64: -vnc :1: Could not open 'foo': No such file or directory

I believe we've done this ever since we reported locations.  Fix it to

    qemu-system-x86_64: bad.img: Could not open 'bad.img': No such file or directory

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190318183312.4684-1-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
2019-04-02 13:30:25 +02:00
Thomas Huth
f18957b854 tests/qemu-iotests/235: Allow fallback to tcg
iotest 235 currently only works with KVM - this is bad for systems where
it is not available, e.g. CI pipelines. The test also works when using
"tcg" as accelerator, so we can simply add that to the list of accelerators,
too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:56 +02:00
Alberto Garcia
d20ba603f2 block: test block-stream with a base node that is used by block-commit
The base node of a block-stream operation indicates the first image
from the backing chain starting from which no data is copied to the
top node.

The block-stream job allows others to use that base image, so a second
block-stream job could be writing to it at the same time. An important
restriction is that the base image must not disappear while the stream
job is ongoing. stream_start() freezes the backing chain from top to
base with that purpose but it does it too late in the code so there is
a race condition there.

This bug was fixed in the previous commit, and this patch contains an
iotest for this scenario.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Alberto Garcia
20509c4b8b block: freeze the backing chain earlier in stream_start()
Commit 6585493369 added code to freeze
the backing chain from 'top' to 'base' for the duration of the
block-stream job.

The problem is that the freezing happens too late in stream_start():
during the bdrv_reopen_set_read_only() call earlier in that function
another job can jump in and remove the base image. If that happens we
have an invalid chain and QEMU crashes.

This patch puts the bdrv_freeze_backing_chain() call at the beginning
of the function.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Alberto Garcia
0f0998f621 block: continue until base is found in bdrv_freeze_backing_chain() et al
All three functions that handle the BdrvChild.frozen attribute walk
the backing chain from 'bs' to 'base' and stop either when 'base' is
found or at the end of the chain if 'base' is NULL.

However if 'base' is not found then the functions return without
errors as if it was NULL.

This is wrong: if the caller passed an incorrect parameter that means
that there is a bug in the code.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Vladimir Sementsov-Ogievskiy
696aaaed57 block/file-posix: do not fail on unlock bytes
bdrv_replace_child() calls bdrv_check_perm() with error_abort on
loosening permissions. However file-locking operations may fail even
in this case, for example on NFS. And this leads to Qemu crash.

Let's avoid such errors. Note, that we ignore such things anyway on
permission update commit and abort.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Thomas Huth
38e694fcc9 tests/qemu-iotests: Remove redundant COPYING file
The file tests/qemu-iotests/COPYING is the same text as in the
COPYING file in the main directory. So as far as I can see, we don't
need the duplicate here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Stefano Garzarella
de23e72bb7 block/gluster: limit the transfer size to 512 MiB
Several versions of GlusterFS (3.12? -> 6.0.1) fail when the
transfer size is greater or equal to 1024 MiB, so we are
limiting the transfer size to 512 MiB to avoid this rare issue.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1691320
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Nir Soffer
a3d6ae2299 qemu-img: Enable BDRV_REQ_MAY_UNMAP in convert
With Kevin's "block: Fix slow pre-zeroing in qemu-img convert"[1]
(commit c9fdcf202f, 'qemu-img: Use BDRV_REQ_NO_FALLBACK for
pre-zeroing') we skip the pre zero step called like this:

    blk_make_zero(s->target, BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK)

And we write zeroes later using:

    blk_co_pwrite_zeroes(s->target,
                         sector_num << BDRV_SECTOR_BITS,
                         n << BDRV_SECTOR_BITS, 0);

Since we use flags=0, this is translated to NBD_CMD_WRITE_ZEROES with
NBD_CMD_FLAG_NO_HOLE flag, which cause the NBD server to allocated space
instead of punching a hole.

Here is an example failure:

$ dd if=/dev/urandom of=src.img bs=1M count=5
$ truncate -s 50m src.img
$ truncate -s 50m dst.img
$ nbdkit -f -v -e '' -U nbd.sock file file=dst.img

$ ./qemu-img convert -n src.img nbd:unix:nbd.sock

We can see in nbdkit log that it received the NBD_CMD_FLAG_NO_HOLE
(may_trim=0):

nbdkit: file[1]: debug: newstyle negotiation: flags: export 0x4d
nbdkit: file[1]: debug: pwrite count=2097152 offset=0
nbdkit: file[1]: debug: pwrite count=2097152 offset=2097152
nbdkit: file[1]: debug: pwrite count=1048576 offset=4194304
nbdkit: file[1]: debug: zero count=33554432 offset=5242880 may_trim=0
nbdkit: file[1]: debug: zero count=13631488 offset=38797312 may_trim=0
nbdkit: file[1]: debug: flush

And the image became fully allocated:

$ qemu-img info dst.img
virtual size: 50M (52428800 bytes)
disk size: 50M

With this change we see that nbdkit did not receive the
NBD_CMD_FLAG_NO_HOLE (may_trim=1):

nbdkit: file[1]: debug: newstyle negotiation: flags: export 0x4d
nbdkit: file[1]: debug: pwrite count=2097152 offset=0
nbdkit: file[1]: debug: pwrite count=2097152 offset=2097152
nbdkit: file[1]: debug: pwrite count=1048576 offset=4194304
nbdkit: file[1]: debug: zero count=33554432 offset=5242880 may_trim=1
nbdkit: file[1]: debug: zero count=13631488 offset=38797312 may_trim=1
nbdkit: file[1]: debug: flush

And the file is sparse as expected:

$ qemu-img info dst.img
virtual size: 50M (52428800 bytes)
disk size: 5.0M

[1] http://lists.nongnu.org/archive/html/qemu-block/2019-03/msg00761.html

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Thomas Huth
e0a59749ef iotests: Fix test 200 on s390x without virtio-pci
virtio-pci is optional on s390x, e.g. in downstream RHEL builds, it
is disabled. On s390x, virtio-ccw should be used instead. Other tests
like 051 or 240 already use virtio-scsi-ccw instead of virtio-scsi-pci
on s390x, so let's do the same here and always use virtio-scsi-ccw on
s390x.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-02 12:04:44 +02:00
Peter Maydell
d61d1a1fb2 fixes for 4.0 (audio, usb),
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQIcBAABAgAGBQJcowU+AAoJEEy22O7T6HE4dLAP/0fvdVi3X+JaYqn3MXI9bmvZ
 j0zI5VovudwRbHhiBof8Sa+ygX6HI/MuQTmS5hjnRe3yh25g2ISRWVsZ09AXqHdT
 mHyg3OgMQaCkPXVN8oxRUmRMP6YIpkAA5COG3gT6DTCnuUunF6OWGlLNj929IZe0
 U/JLsB6YXhIaQ4UIx0njLih3H7gWlxmrr9O6uJosrAUnHezmOVGjayrGmgCDLNwo
 FkqzcZlX0X3EXAztnNp0UhLDQ3je3H9XeG/NxNxgDwTElF8MZH5Sa7CgSdV33c+I
 idFl89twlYBcDqTPeGg2V0pfIelUAL3s+M9dIX9QUvdSIoXmIIw1+/EMsGePMtGC
 MG+rXC1ff5aRftrIZVuqrRFr2ujrU/8D75MJk/DCFAOBEFrghvMkd3/t3kDlRye7
 9sKuVzJu/owY5vaGaI++YJs4Y5/OZqqPfRpWXjHCQKVdVGa4qhHoiYrJQhvnjvqT
 mENu7C9vW+Rc3dxNAxeimTj8w7Lt9yGwhlRo5X04/IzdxSyUxhlqdrtspZhQr5l9
 eLwgvr4898TgNTum2phQtJfrLh3D8peP7/55pIrNu0pgArvEhkZMPsOW4xVDp5N6
 Mf5igXnvgLxHqw3q2gJoQN9/LZR/rTzb4vpgERlrhxCo+5VLuioqXZK4A5nfRloW
 44uQ1WmNXDxJRWkdkOOi
 =v7Pc
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kraxel/tags/fixes-20190402-pull-request' into staging

fixes for 4.0 (audio, usb),

# gpg: Signature made Tue 02 Apr 2019 07:46:22 BST
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>" [full]
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>" [full]
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>" [full]
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/fixes-20190402-pull-request:
  audio: fix audio timer rate conversion bug
  usb-mtp: remove usb_mtp_object_free_one
  usb-mtp: fix return status of delete
  hw/usb/bus.c: Handle "no speed matched" case in usb_mask_to_str()
  Revert "audio: fix pc speaker init"

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 08:18:24 +01:00
Volker Rümelin
be1092afa0 audio: fix audio timer rate conversion bug
Currently the default audio timer frequency is 10000Hz instead of
a period of 10000us. Also the audiodev timer-period property gets
converted like a frequency. Only handling of the legacy
QEMU_AUDIO_TIMER_PERIOD environment variable is correct because
it's actually a frequency.

With this patch the property timer-period is really a timer period
and QEMU_AUDIO_TIMER_PERIOD remains a frequency.

Fixes: 71830221fb "-audiodev command line option basic implementation."
Signed-off-by: Volker Rümelin <vr_qemu@t-online.de>
Reviewed-by: Zoltán Kővágó <DirtY.iCE.hu@gmail.com>
Message-id: 90b95e4f-39ef-2b01-da6a-857ebaee1ec5@t-online.de
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-04-02 07:50:49 +02:00
Bandan Das
b396733df3 usb-mtp: remove usb_mtp_object_free_one
This function is used in the delete path only and can
be replaced by a call to usb_mtp_object_free.

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Bandan Das <bsd@redhat.com>
Message-Id: <20190401211712.19012-3-bsd@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-04-02 07:22:49 +02:00
Bandan Das
4bc1591681 usb-mtp: fix return status of delete
Spotted by Coverity: CID 1399414

mtp delete allows the return status of delete succeeded,
partial_delete or readonly - when none of the objects could be
deleted. Give more meaningful names to return values of the
delete function.

Some initiators recurse over the objects themselves. In that case,
only READ_ONLY can be returned.

Signed-off-by: Bandan Das <bsd@redhat.com>
Message-Id: <20190401211712.19012-2-bsd@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-04-02 07:22:40 +02:00
Peter Maydell
47175951a6 nbd patches for 2019-04-01
- Better behavior of qemu-img map on NBD images
 - Fixes for NBD protocol alignment corner cases:
  - the server has fewer places where it sends reads or block status
    not aligned to its advertised block size
  - the client has more cases where it can work around server
    non-compliance present in qemu 3.1
  - the client now avoids non-compliant requests when interoperating
    with nbdkit or other servers not advertising block size
 -----BEGIN PGP SIGNATURE-----
 
 iQEcBAABCAAGBQJcohr+AAoJEKeha0olJ0Nq+pQH/2AfH172yH2QcTLFbamBhWr1
 gCeU4Kk4U9vD5itKBY7SQfdJxebzSJTkDcSTsGSXI7Q/cadzCNU2HJh7/ALbkbTL
 suDju7uzmvbW7ogMbV/EO4qmomn04cNg+mUkgUgPQZGcnZAaXt2tFtWzPOLuboc1
 kJhsLDfE9A/tFuz+Io7Umrjw6TH8CuerGw+xUJ4hATqiWPd/5k0WazvmKis143io
 zS8ffMZJzJBmpSDtohojz6EFYZOKTFrPNcUwly8cVtRReSV5OiOhFW2Ul6HNT6I7
 2hxkSYFB/7V4zDM0OGPkbhdLAkggFAxAZD16sBzbggFnMyJvqFIj+RUBXuKpv/g=
 =BGOe
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2019-04-01' into staging

nbd patches for 2019-04-01

- Better behavior of qemu-img map on NBD images
- Fixes for NBD protocol alignment corner cases:
 - the server has fewer places where it sends reads or block status
   not aligned to its advertised block size
 - the client has more cases where it can work around server
   non-compliance present in qemu 3.1
 - the client now avoids non-compliant requests when interoperating
   with nbdkit or other servers not advertising block size

# gpg: Signature made Mon 01 Apr 2019 15:06:54 BST
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2019-04-01:
  nbd/client: Trace server noncompliance on structured reads
  nbd/server: Advertise actual minimum block size
  block: Add bdrv_get_request_alignment()
  nbd/client: Support qemu-img convert from unaligned size
  nbd/client: Reject inaccessible tail of inconsistent server
  nbd/client: Report offsets in bdrv_block_status
  nbd/client: Lower min_block for block-status, unaligned size
  iotests: Add 241 to test NBD on unaligned images
  nbd-client: Work around server BLOCK_STATUS misalignment at EOF
  qemu-img: Gracefully shutdown when map can't finish
  nbd: Permit simple error to NBD_CMD_BLOCK_STATUS
  nbd: Don't lose server's error to NBD_CMD_BLOCK_STATUS
  nbd: Tolerate some server non-compliance in NBD_CMD_BLOCK_STATUS
  qemu-img: Report bdrv_block_status failures

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-04-02 03:46:30 +01:00
Eric Blake
75d34eb98c nbd/client: Trace server noncompliance on structured reads
Just as we recently added a trace for a server sending block status
that doesn't match the server's advertised minimum block alignment,
let's do the same for read chunks.  But since qemu 3.1 is such a
server (because it advertised 512-byte alignment, but when serving a
file that ends in data but is not sector-aligned, NBD_CMD_READ would
detect a mid-sector change between data and hole at EOF and the
resulting read chunks are unaligned), we don't want to change our
behavior of otherwise tolerating unaligned reads.

Note that even though we fixed the server for 4.0 to advertise an
actual block alignment (which gets rid of the unaligned reads at EOF
for posix files), we can still trigger it via other means:

$ qemu-nbd --image-opts driver=blkdebug,align=512,image.driver=file,image.filename=/path/to/non-aligned-file

Arguably, that is a bug in the blkdebug block status function, for
leaking a block status that is not aligned. It may also be possible to
observe issues with a backing layer with smaller alignment than the
active layer, although so far I have been unable to write a reliable
iotest for that scenario.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190330165349.32256-1-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-04-01 08:58:04 -05:00
Eric Blake
b0245d6478 nbd/server: Advertise actual minimum block size
Both NBD_CMD_BLOCK_STATUS and structured NBD_CMD_READ will split their
reply according to bdrv_block_status() boundaries. If the block device
has a request_alignment smaller than 512, but we advertise a block
alignment of 512 to the client, then this can result in the server
reply violating client expectations by reporting a smaller region of
the export than what the client is permitted to address (although this
is less of an issue for qemu 4.0 clients, given recent client patches
to overlook our non-compliance at EOF).  Since it's always better to
be strict in what we send, it is worth advertising the actual minimum
block limit rather than blindly rounding it up to 512.

Note that this patch is not foolproof - it is still possible to
provoke non-compliant server behavior using:

$ qemu-nbd --image-opts driver=blkdebug,align=512,image.driver=file,image.filename=/path/to/non-aligned-file

That is arguably a bug in the blkdebug driver (it should never pass
back block status smaller than its alignment, even if it has to make
multiple bdrv_get_status calls and determine the
least-common-denominator status among the group to return). It may
also be possible to observe issues with a backing layer with smaller
alignment than the active layer, although so far I have been unable to
write a reliable iotest for that scenario (but again, an issue like
that could be argued to be a bug in the block layer, or something
where we need a flag to bdrv_block_status() to state whether the
result must be aligned to the current layer's limits or can be
subdivided for accuracy when chasing backing files).

Anyways, as blkdebug is not normally used, and as this patch makes our
server more interoperable with qemu 3.1 clients, it is worth applying
now, even while we still work on a larger patch series for the 4.1
timeframe to have byte-accurate file lengths.

Note that the iotests output changes - for 223 and 233, we can see the
server's better granularity advertisement; and for 241, the three test
cases have the following effects:
- natural alignment: the server's smaller alignment is now advertised,
and the hole reported at EOF is now the right result; we've gotten rid
of the server's non-compliance
- forced server alignment: the server still advertises 512 bytes, but
still sends a mid-sector hole. This is still a server compliance bug,
which needs to be fixed in the block layer in a later patch; output
does not change because the client is already being tolerant of the
non-compliance
- forced client alignment: the server's smaller alignment means that
the client now sees the server's status change mid-sector without any
protocol violations, but the fact that the map shows an unaligned
mid-sector hole is evidence of the block layer problems with aligned
block status, to be fixed in a later patch

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-7-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: rebase to enhanced iotest 241 coverage]
2019-04-01 08:52:28 -05:00
Eric Blake
4841211e0d block: Add bdrv_get_request_alignment()
The next patch needs access to a device's minimum permitted
alignment, since NBD wants to advertise this to clients. Add
an accessor function, borrowing from blk_get_max_transfer()
for accessing a backend's block limits.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20190329042750.14704-6-eblake@redhat.com>
2019-04-01 08:46:52 -05:00
Eric Blake
9cf638508c nbd/client: Support qemu-img convert from unaligned size
If an NBD server advertises a size that is not a multiple of a sector,
the block layer rounds up that size, even though we set info.size to
the exact byte value sent by the server. The block layer then proceeds
to let us read or query block status on the hole that it added past
EOF, which the NBD server is unlikely to be happy with. Fortunately,
qemu as a server never advertizes an unaligned size, so we generally
don't run into this problem; but the nbdkit server makes it easy to
test:

$ printf %1000d 1 > f1
$ ~/nbdkit/nbdkit -fv file f1 & pid=$!
$ qemu-img convert -f raw nbd://localhost:10809 f2
$ kill $pid
$ qemu-img compare f1 f2

Pre-patch, the server attempts a 1024-byte read, which nbdkit
rightfully rejects as going beyond its advertised 1000 byte size; the
conversion fails and the output files differ (not even the first
sector is copied, because qemu-img does not follow ddrescue's habit of
trying smaller reads to get as much information as possible in spite
of errors). Post-patch, the client's attempts to read (and query block
status, for new enough nbdkit) are properly truncated to the server's
length, with sane handling of the hole the block layer forced on
us. Although f2 ends up as a larger file (1024 bytes instead of 1000),
qemu-img compare shows the two images to have identical contents for
display to the guest.

I didn't add iotests coverage since I didn't want to add a dependency
on nbdkit in iotests. I also did NOT patch write, trim, or write
zeroes - these commands continue to fail (usually with ENOSPC, but
whatever the server chose), because we really can't write to the end
of the file, and because 'qemu-img convert' is the most common case
where we care about being tolerant (which is read-only). Perhaps we
could truncate the request if the client is writing zeros to the tail,
but that seems like more work, especially if the block layer is fixed
in 4.1 to track byte-accurate sizing (in which case this patch would
be reverted as unnecessary).

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-5-eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
2019-04-01 08:32:44 -05:00
Eric Blake
3add3ab782 nbd/client: Reject inaccessible tail of inconsistent server
The NBD spec suggests that a server should never advertise a size
inconsistent with its minimum block alignment, as that tail is
effectively inaccessible to a compliant client obeying those block
constraints. Since we have a habit of rounding up rather than
truncating, to avoid losing the last few bytes of user input, and we
cannot access the tail when the server advertises bogus block sizing,
abort the connection to alert the server to fix their bug.  And
rejecting such servers matches what we already did for a min_block
that was not a power of 2 or which was larger than max_block.

Does not impact either qemu (which always sends properly aligned
sizes) or nbdkit (which does not send minimum block requirements yet);
so this is mostly aimed at new NBD server implementations, and ensures
that the rest of our code can assume the size is aligned.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190330155704.24191-1-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-04-01 08:31:16 -05:00
Peter Maydell
5189e30b14 hw/usb/bus.c: Handle "no speed matched" case in usb_mask_to_str()
In usb_mask_to_str() we convert a mask of USB speeds into
a human-readable string (like "full+high") for use in
tracing and error messages. However the conversion code
doesn't do anything to the string buffer if the passed in
speedmask doesn't match any of the recognized speeds,
which means that the tracing and error messages will
end up with random garbage in them. This can happen if
we're doing USB device passthrough.

Handle the "unrecognized speed" case by using the
string "unknown".

Fixes: https://bugs.launchpad.net/qemu/+bug/1603785
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20190328133503.6490-1-peter.maydell@linaro.org
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-04-01 08:53:44 +02:00
Gerd Hoffmann
28605a22f5 Revert "audio: fix pc speaker init"
This reverts commit bd56d37884.

Turned out it isn't that simple as the device needs the pit object link.
So "-device isa-pcspk" isn't going wo work anyway.  We are in freeze, so
just reverting the thing is the best way to handle this for now, trying
to come up with something better can be done in the 4.1 devel cycle.

Also add a comment noting the object link.

Reported-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Message-id: 20190328071121.21147-1-kraxel@redhat.com
2019-04-01 08:53:40 +02:00
Eric Blake
a62a85ef5c nbd/client: Report offsets in bdrv_block_status
It is desirable for 'qemu-img map' to have the same output for a file
whether it is served over file or nbd protocols. However, ever since
we implemented block status for NBD (2.12), the NBD protocol forgot to
inform the block layer that as the final layer in the chain, the
offset is valid; without an offset, the human-readable form of
qemu-img map gives up with the unhelpful:

$ nbdkit -U - data data="1" size=512 --run 'qemu-img map $nbd'
Offset          Length          Mapped to       File
qemu-img: File contains external, encrypted or compressed clusters.

The --output=json form always works, because it is reporting the
lower-level bdrv_block_status results directly rather than trying to
filter out sparse ranges for human consumption - but now it also
shows the offset member.

With this patch, the human output changes to:

Offset          Length          Mapped to       File
0               0x200           0               nbd+unix://?socket=/tmp/nbdkitOxeoLa/socket

This change is observable to several iotests.

Fixes: 78a33ab5
Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-4-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 20:52:29 -05:00
Eric Blake
7da537f70d nbd/client: Lower min_block for block-status, unaligned size
We have a latent bug in our NBD client code, tickled by the brand new
nbdkit 1.11.10 block status support:

$ nbdkit --filter=log --filter=truncate -U - \
           data data="1" size=511 truncate=64K logfile=/dev/stdout \
           --run 'qemu-img convert $nbd /var/tmp/out'
...
qemu-img: block/io.c:2122: bdrv_co_block_status: Assertion `*pnum && QEMU_IS_ALIGNED(*pnum, align) && align > offset - aligned_offset' failed.

The culprit? Our implementation of .bdrv_co_block_status can return
unaligned block status for any server that operates with a lower
actual alignment than what we tell the block layer in
request_alignment, in violation of the block layer's constraints. To
date, we've been unable to trip the bug, because qemu as NBD server
always advertises block sizing (at which point it is a server bug if
the server sends unaligned status - although qemu 3.1 is such a server
and I've sent separate patches for 4.0 both to get the server to obey
the spec, and to let the client to tolerate server oddities at EOF).

But nbdkit does not (yet) advertise block sizing, and therefore is not
in violation of the spec for returning block status at whatever
boundaries it wants, and those unaligned results can occur anywhere
rather than just at EOF. While we are still wise to avoid sending
sub-sector read/write requests to a server of unknown origin, we MUST
consider that a server telling us block status without an advertised
block size is correct.  So, we either have to munge unaligned answers
from the server into aligned ones that we hand back to the block
layer, or we have to tell the block layer about a smaller alignment.

Similarly, if the server advertises an image size that is not
sector-aligned, we might as well assume that the server intends to let
us access those tail bytes, and therefore supports a minimum block
size of 1, regardless of whether the server supports block status
(although we still need more patches to fix the problem that with an
unaligned image, we can send read or block status requests that exceed
EOF to the server). Again, qemu as server cannot trip this problem
(because it rounds images to sector alignment), but nbdkit advertised
unaligned size even before it gained block status support.

Solve both alignment problems at once by using better heuristics on
what alignment to report to the block layer when the server did not
give us something to work with. Note that very few NBD servers
implement block status (to date, only qemu and nbdkit are known to do
so); and as the NBD spec mentioned block sizing constraints prior to
documenting block status, it can be assumed that any future
implementations of block status are aware that they must advertise
block size if they want a minimum size other than 1.

We've had a long history of struggles with picking the right alignment
to use in the block layer, as evidenced by the commit message of
fd8d372d (v2.12) that introduced the current choice of forced 512-byte
alignment.

There is no iotest coverage for this fix, because qemu can't provoke
it, and I didn't want to make test 241 dependent on nbdkit.

Fixes: fd8d372d
Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
2019-03-30 20:52:19 -05:00
Eric Blake
e9dce9cb6e iotests: Add 241 to test NBD on unaligned images
Add a test for the NBD client workaround in the previous patch.  It's
not really feasible for an iotest to assume a specific tracing engine,
so we can't really probe trace_nbd_parse_blockstatus_compliance to see
if the server was fixed vs. whether the client just worked around the
server (other than by rearranging order between code patches and this
test). But having a successful exchange sure beats the previous state
of an error message. Since format probing can change alignment, we can
use that as an easy way to test several configurations.

Not tested yet, but worth adding to this test in future patches: an
NBD server that can advertise a non-sector-aligned size (such as
nbdkit) causes qemu as the NBD client to misbehave when it rounds the
size up and accesses beyond the advertised size. Qemu as NBD server
never advertises a non-sector-aligned size (since bdrv_getlength()
currently rounds up to sector boundaries); until qemu can act as such
a server, testing that flaw will have to rely on external binaries.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190329042750.14704-2-eblake@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: add forced-512 alignment, and nbdkit reproducer comment]
2019-03-30 20:50:58 -05:00
Eric Blake
737d3f5244 nbd-client: Work around server BLOCK_STATUS misalignment at EOF
The NBD spec is clear that a server that advertises a minimum block
size should reply to NBD_CMD_BLOCK_STATUS with extents aligned
accordingly. However, we know that the qemu NBD server implementation
has had a corner-case bug where it is not compliant with the spec,
present since the introduction of NBD_CMD_BLOCK_STATUS in qemu 2.12
(and unlikely to be patched in time for 4.0). Namely, when qemu is
serving a file that is not a multiple of 512 bytes, it rounds the size
advertised over NBD up to the next sector boundary (someday, I'd like
to fix that to be byte-accurate, but it's a much bigger audit not
appropriate for this release); yet if the final sector contains data
prior to EOF, lseek(SEEK_HOLE) will point to the implicit hole
mid-sector which qemu then reported over NBD.

We are well within our rights to hang up on a server that can't follow
the spec, but it is more useful to try and keep the connection alive
in spite of the problem. Do so by tracing a message about the problem,
and then either truncating the request back to an aligned boundary (if
it covered more than the final sector) or widening it out to the full
boundary with a forced status of data (since truncating would result
in 0 bytes, but we have to make progress, and valid since data is a
default-safe answer). And in practice, since the problem only happens
on a sector that starts with data and ends with a hole, we are going
to want to read that full sector anyway (where qemu as the server
fills in the tail beyond EOF with appropriate NUL bytes).

Easy reproduction:
$ printf %1000d 1 > file
$ qemu-nbd -f raw -t file & pid=$!
$ qemu-img map --output=json -f raw nbd://localhost:10809
qemu-img: Could not read file metadata: Invalid argument
$ kill $pid

where the patched version instead succeeds with:
[{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}]

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190326171317.4036-1-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 10:06:08 -05:00
Eric Blake
30065d1424 qemu-img: Gracefully shutdown when map can't finish
Trying 'qemu-img map -f raw nbd://localhost:10809' causes the
NBD server to output a scary message:

qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read

This is because the NBD client, being remote, has no way to expose a
human-readable map (the --output=json data is fine, however). But
because we exit(1) right after the message, causing the client to
bypass all block cleanup, the server sees the abrupt exit and warns,
whereas it would be silent had the client had a chance to send
NBD_CMD_DISC. Other protocols may have similar cleanup issues, where
failure to blk_unref() could cause unintended effects.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190326184043.7544-1-eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2019-03-30 10:06:08 -05:00
Eric Blake
ebd82cd872 nbd: Permit simple error to NBD_CMD_BLOCK_STATUS
The NBD spec is clear that when structured replies are active, a
simple error reply is acceptable to any command except for
NBD_CMD_READ.  However, we were mistakenly requiring structured errors
for NBD_CMD_BLOCK_STATUS, and hanging up on a server that gave a
simple error (since qemu does not behave as such a server, we didn't
notice the problem until now).  Broken since its introduction in
commit 78a33ab5 (v2.12).

Noticed while debugging a separate failure reported by nbdkit while
working out its initial implementation of BLOCK_STATUS, although it
turns out that nbdkit also chose to send structured error replies for
BLOCK_STATUS, so I had to manually provoke the situation by hacking
qemu's server to send a simple error reply:

| diff --git i/nbd/server.c w/nbd/server.c
| index fd013a2817a..833288d7c45 100644
| 00--- i/nbd/server.c
| +++ w/nbd/server.c
| @@ -2269,6 +2269,8 @@ static coroutine_fn int nbd_handle_request(NBDClient *client,
|                                        "discard failed", errp);
|
|      case NBD_CMD_BLOCK_STATUS:
| +        return nbd_co_send_simple_reply(client, request->handle, ENOMEM,
| +                                        NULL, 0, errp);
|          if (!request->len) {
|              return nbd_send_generic_reply(client, request->handle, -EINVAL,
|                                            "need non-zero length", errp);
|

Signed-off-by: Eric Blake <eblake@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20190325190104.30213-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 10:06:08 -05:00
Eric Blake
b29f3a3d2a nbd: Don't lose server's error to NBD_CMD_BLOCK_STATUS
When the server replies with a (structured [*]) error to
NBD_CMD_BLOCK_STATUS, without any extent information sent first, the
client code was blindly throwing away the server's error code and
instead telling the caller that EIO occurred.  This has been broken
since its introduction in 78a33ab5 (v2.12, where we should have called:
   error_setg(&local_err, "Server did not reply with any status extents");
   nbd_iter_error(&iter, false, -EIO, &local_err);
to declare the situation as a non-fatal error if no earlier error had
already been flagged, rather than just blindly slamming iter.err and
iter.ret), although it is more noticeable since commit 7f86068d, which
actually tries hard to preserve the server's code thanks to a separate
iter.request_ret.

[*] The spec is clear that the server is also permitted to reply with
a simple error, but that's a separate fix.

I was able to provoke this scenario with a hack to the server, then
seeing whether ENOMEM makes it back to the caller:

| diff --git a/nbd/server.c b/nbd/server.c
| index fd013a2817a..29c7995de02 100644
| --- a/nbd/server.c
| +++ b/nbd/server.c
| @@ -2269,6 +2269,8 @@ static coroutine_fn int nbd_handle_request(NBDClient *client,
|                                        "discard failed", errp);
|
|      case NBD_CMD_BLOCK_STATUS:
| +        return nbd_send_generic_reply(client, request->handle, -ENOMEM,
| +                                      "no status for you today", errp);
|          if (!request->len) {
|              return nbd_send_generic_reply(client, request->handle, -EINVAL,
|                                            "need non-zero length", errp);
| --

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190325190104.30213-2-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 10:06:08 -05:00
Eric Blake
a39286dd61 nbd: Tolerate some server non-compliance in NBD_CMD_BLOCK_STATUS
The NBD spec states that NBD_CMD_FLAG_REQ_ONE (which we currently
always use) should not reply with an extent larger than our request,
and that the server's response should be exactly one extent. Right
now, that means that if a server sends more than one extent, we treat
the server as broken, fail the block status request, and disconnect,
which prevents all further use of the block device. But while good
software should be strict in what it sends, it should be tolerant in
what it receives.

While trying to implement NBD_CMD_BLOCK_STATUS in nbdkit, we
temporarily had a non-compliant server sending too many extents in
spite of REQ_ONE. Oddly enough, 'qemu-img convert' with qemu 3.1
failed with a somewhat useful message:
  qemu-img: Protocol error: invalid payload for NBD_REPLY_TYPE_BLOCK_STATUS

which then disappeared with commit d8b4bad8, on the grounds that an
error message flagged only at the time of coroutine teardown is
pointless, and instead we should rely on the actual failed API to
report an error - in other words, the 3.1 behavior was masking the
fact that qemu-img was not reporting an error. That has since been
fixed in the previous patch, where qemu-img convert now fails with:
  qemu-img: error while reading block status of sector 0: Invalid argument

But even that is harsh.  Since we already partially relaxed things in
commit acfd8f7a to tolerate a server that exceeds the cap (although
that change was made prior to the NBD spec actually putting a cap on
the extent length during REQ_ONE - in fact, the NBD spec change was
BECAUSE of the qemu behavior prior to that commit), it's not that much
harder to argue that we should also tolerate a server that sends too
many extents.  But at the same time, it's nice to trace when we are
being tolerant of server non-compliance, in order to help server
writers fix their implementations to be more portable (if they refer
to our traces, rather than just stderr).

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190323212639.579-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 10:06:08 -05:00
Eric Blake
2058c2ad26 qemu-img: Report bdrv_block_status failures
If bdrv_block_status_above() fails, we are aborting the convert
process but failing to print an error message.  Broken in commit
690c7301 (v2.4) when rewriting convert's logic.

Discovered when teaching nbdkit to support NBD_CMD_BLOCK_STATUS, and
accidentally violating the protocol by returning more than one extent
in spite of qemu asking for NBD_CMD_FLAG_REQ_ONE.  The qemu NBD code
should probably handle the server's non-compliance more gracefully
than failing with EINVAL, but qemu-img shouldn't be silently
squelching any block status failures. It doesn't help that qemu 3.1
masks the qemu-img bug with extra noise that the nbd code is dumping
to stderr (that noise was cleaned up in d8b4bad8).

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190323212639.579-2-eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2019-03-30 10:06:07 -05:00
Peter Maydell
230ce19814 Update palcode for machine checks.
-----BEGIN PGP SIGNATURE-----
 
 iQFRBAABCgA7FiEEekgeeIaLTbaoWgXAZN846K9+IV8FAlyZX6QdHHJpY2hhcmQu
 aGVuZGVyc29uQGxpbmFyby5vcmcACgkQZN846K9+IV9ajwf/cNMPCvT5D78D9Z0H
 AeBg8QAQAJFHPvKruEm2g4vIx4hN6h5tCLKeqP2CjVcgElI4k0PDFeRIdmVv9qQO
 yhzc2wrMQpme5iy9pVT6D7sYcL2W+VpViGAysdn0gr1bg+1tGsx9nHILwll0DP9m
 Gkybldkjwgqxj8WsKDPJ59ZyYpe3XkhAe7snGcox53P3Ph07xveO2SshlqCxrNOH
 FuJ9NEyfsTGNlc/UR2ggxzPzyfqyFhx1jIzVPo+GP3FOqdpQylMmhaJ4NXUC1wA+
 y6DDuOCtvrjbWJ5ifPPcR91Zl/siFI2MuiCtd1Ysqdbuwg7hg5yZItH6sw7cmiSD
 1EBVog==
 =m6Hm
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/rth/tags/pull-axp-20190325' into staging

Update palcode for machine checks.

# gpg: Signature made Mon 25 Mar 2019 23:09:24 GMT
# gpg:                using RSA key 7A481E78868B4DB6A85A05C064DF38E8AF7E215F
# gpg:                issuer "richard.henderson@linaro.org"
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>" [full]
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* remotes/rth/tags/pull-axp-20190325:
  pc-bios: Update palcode-clipper

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-03-29 19:29:00 +00:00
Peter Maydell
c503849bb5 -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
 
 iQEcBAABAgAGBQJcncmSAAoJEO8Ells5jWIRh8YH/2EXWKEBlN1TSWZQrL3ifq+4
 OS1335yGK34uQrZEvhgpGlHDuvwUn+1ma8YHgkpGGohQaOb91FqDZLQi6aZ1/XBX
 2aLt46XSgvp3y9EO2dVeFepW2dTyAPZMCkWEvJINXpRQQ6X2iu0iDjZoqB3aA8dU
 UP8I3FsoFOg4A2haiMxBgGhldf9VCAJtoKrMoxOOgDhCypwZPqSlmRn3QeqO0MmG
 aVDt5MBqKIXMmG2cUDy+KvsylGXtZ9VisvIU4UAIJaUGEopuYygEYQQhETN9uAXK
 ts6yWdfAU438NcNoFnChrNeHwQS8TuZlV1pcV7xQ2gaKB1cS3I+MGWOzTTWD/zs=
 =4PKd
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging

# gpg: Signature made Fri 29 Mar 2019 07:30:26 GMT
# gpg:                using RSA key EF04965B398D6211
# gpg: Good signature from "Jason Wang (Jason Wang on RedHat) <jasowang@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 215D 46F4 8246 689E C77F  3562 EF04 965B 398D 6211

* remotes/jasowang/tags/net-pull-request:
  net: tap: use qemu_set_nonblock
  MAINTAINERS: Update the latest email address
  e1000: Delay flush queue when receive RCTL
  net/socket: learn to talk with a unix dgram socket

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-03-29 11:51:54 +00:00
Peter Maydell
94c01767aa ppc patch queue 2019-03-29
Here's a set of bugfixes for ppc, aimed at qemu-4.0 during hard freeze.
 
 We have one cleanup that's not strictly a bugfix, but will avoid an
 ugly external interface making it to a released version.
 
 We have one change to generic code to tweak the semantics of
 qemu_getrampagesize() which fixes a bug for ppc.  This does have a
 possible impact on s390x which uses this function for a different
 purpose.  I've discussed with David Hildenbrand and Igor Mammedov,
 however and we think it won't immediately break anything due to some
 existing bugs in the s390 usage.  David H will be following up with
 some s390 fixes in that area.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdfRlhq5hpmzETofcbDjKyiDZs5IFAlydkLUACgkQbDjKyiDZ
 s5KYWw/+MWHPjYrgq2YqWr5VwRFhjV3gg86sQKCq+k/OhRl1ALmQ8DIDA5IBU/zf
 EmFPOdE4hvaCbYRLDyBL5ayVR/obu1+J3SHo1gDNAoXnpSIoTN+a2DeL8qfQekQL
 0EyPnARHlZZiHVM7YWyKIKUKpsptT5TRuZrUmM4pWXNWhk3qF39XQ0gCnMqdfd5U
 n8qcKKz9UxlFPkhyw/mGveMV1eJTlw1EQCyvUsfCgoK0LljDUvlZPCOI/O2jupPn
 mL7CifCm2yPs9ZuEgr/YSNYUCk96gf4hTdN2FiqdZWYbgUQaMDW/HtL6DwR5jI1W
 IXqnD7qsJxrHzsw9ZWBTjcK1PGaw2UMnuHNemI/T6sZP7UIsU3DmKsVA14GEWVBa
 zdO9HxQNa+UjKKtpTijqGxga3jya236a2ssgTr871chWs6cPH2db/iHNUM15ri1J
 wVFmQpAJrLNy5KLGL8Mgs6muH4DVaefKhWdGy3A/l5ThDZp0yAb27sYwVqzHm7P8
 L+jbEoWqRgplkVOtx9jCBjiuca5Sdwi2uhZ5Q6bjEwTPcMU90J6NcsN92/QvEuod
 Tmx/SZs81Pzg8Icq7AMbPbHaVYLy6pDlxIE8KX5wCrhqbWaH2UaxBU2M8mDOdxS5
 AihcQA+KUwnBiRRLaANTGnfJ3FZMNak7qwBP6clNKpazsv53f7Q=
 =sHfP
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-4.0-20190329' into staging

ppc patch queue 2019-03-29

Here's a set of bugfixes for ppc, aimed at qemu-4.0 during hard freeze.

We have one cleanup that's not strictly a bugfix, but will avoid an
ugly external interface making it to a released version.

We have one change to generic code to tweak the semantics of
qemu_getrampagesize() which fixes a bug for ppc.  This does have a
possible impact on s390x which uses this function for a different
purpose.  I've discussed with David Hildenbrand and Igor Mammedov,
however and we think it won't immediately break anything due to some
existing bugs in the s390 usage.  David H will be following up with
some s390 fixes in that area.

# gpg: Signature made Fri 29 Mar 2019 03:27:49 GMT
# gpg:                using RSA key 75F46586AE61A66CC44E87DC6C38CACA20D9B392
# gpg: Good signature from "David Gibson <david@gibson.dropbear.id.au>" [full]
# gpg:                 aka "David Gibson (Red Hat) <dgibson@redhat.com>" [full]
# gpg:                 aka "David Gibson (ozlabs.org) <dgibson@ozlabs.org>" [full]
# gpg:                 aka "David Gibson (kernel.org) <dwg@kernel.org>" [unknown]
# Primary key fingerprint: 75F4 6586 AE61 A66C C44E  87DC 6C38 CACA 20D9 B392

* remotes/dgibson/tags/ppc-for-4.0-20190329:
  exec: Only count mapped memory backends for qemu_getrampagesize()
  spapr/irq: Add XIVE sanity checks on non-P9 machines
  spapr: Simplify handling of host-serial and host-model values
  target/ppc: Fix QEMU crash with stxsdx
  target/ppc: Improve comment of bcctr used for spectre v2 mitigation
  target/ppc: Consolidate 64-bit server processor detection in a helper
  target/ppc: Enable "decrement and test CTR" version of bcctr
  target/ppc: Fix TCG temporary leaks in gen_bcond()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2019-03-29 09:36:29 +00:00
Li Qiang
ab79237a15 net: tap: use qemu_set_nonblock
The fcntl will change the flags directly, use qemu_set_nonblock()
instead.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Li Qiang <liq3ea@gmail.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
2019-03-29 15:22:18 +08:00