Commit Graph

2186 Commits

Author SHA1 Message Date
Daniel P. Berrange
9164b89762 crypto: implement sha224, sha384, sha512 and ripemd160 hashes
Wire up the nettle and gcrypt hash backends so that they can
support the sha224, sha384, sha512 and ripemd160 hash algorithms.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2016-07-04 15:52:36 +01:00
Daniel P. Berrange
0c16c056a4 crypto: switch hash code to use nettle/gcrypt directly
Currently the internal hash code is using the gnutls hash APIs.
GNUTLS in turn is wrapping either nettle or gcrypt. Not only
were the GNUTLS hash APIs not added until GNUTLS 2.9.10, but
they don't expose support for all the algorithms QEMU needs
to use with LUKS.

Address this by directly wrapping nettle/gcrypt in QEMU and
avoiding GNUTLS's extra layer of indirection. This gives us
support for hash functions on a much wider range of platforms
and opens up ability to support more hash functions. It also
avoids a GNUTLS bug which would not correctly handle hashing
of large data blocks if int != size_t.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2016-07-04 10:47:09 +01:00
Daniel P. Berrange
8cbfc94269 crypto: rename OUT to out in xts test to avoid clash on MinGW
On MinGW one of the system headers already has "OUT" defined
which causes a compile failure of the test suite. Rename the
test suite var to 'out' to avoid this clash

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2016-07-04 10:46:59 +01:00
Daniel P. Berrange
8b7cdba386 crypto: fix handling of iv generator hash defaults
When opening an existing LUKS volume, if the iv generator is
essiv, then the iv hash algorithm is mandatory to provide. We
must report an error if it is omitted in the cipher mode spec,
not silently default to hash 0 (md5).  If the iv generator is
not essiv, then we explicitly ignore any iv hash algorithm,
rather than report an error, for compatibility with dm-crypt.

When creating a new LUKS volume, if the iv generator is essiv
and no iv hsah algorithm is provided, we should default to
using the sha256 hash.

Reported-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2016-07-04 10:46:59 +01:00
Eric Blake
9b4e38fe6a qapi: Fix crash on missing alternate member of QAPI struct
If a QAPI struct has a mandatory alternate member which is not
present on input, the input visitor reports an error for the
missing alternate without setting the discriminator, but the
cleanup code for the struct still tries to use the dealloc
visitor to clean up the alternate.

Commit dbf11922 changed visit_start_alternate to set *obj to NULL
when an error occurs, where it was previously left untouched.
Thus, before the patch, the dealloc visitor is blindly trying to
cleanup whatever branch corresponds to (*obj)->type == 0 (that is,
QTYPE_NONE, because *obj still pointed to zeroed memory), which
selects the default branch of the switch and sets an error, but
this second error is ignored by the way the dealloc visitor is
used; but after the patch, the attempt to switch dereferences NULL.

When cleaning up after a partial object parse, we specifically
check for !*obj after visit_start_struct() (see gen_visit_object());
doing the same for alternates fixes the crash. Enhance the testsuite
to give coverage for both missing struct and missing alternate
members.

Also add an abort - we expect visit_start_alternate() to either set an
error or to set (*obj)->type to a valid QType that corresponds to
actual user input, and QTYPE_NONE should never be reachable from valid
input.  Had the abort() been in place earlier, we might have noticed
the dealloc visitor dereferencing bogus zeroed memory prior to when
commit dbf11922 forced our hand by setting *obj to NULL and causing a
fault.

Test case:

{'execute':'blockdev-add', 'arguments':{'options':{'driver':'raw'}}}

The choice of 'driver':'raw' selects a BlockdevOptionsGenericFormat
struct, which has a mandatory 'file':'BlockdevRef' in QAPI.  Since
'file' is missing as a sibling of 'driver', this should report a
graceful error rather than fault.  After this patch, we are back to:

{"error": {"class": "GenericError", "desc": "Parameter 'file' is missing"}}

Generated code in qapi-visit.c changes as:

|@@ -2444,6 +2444,9 @@ void visit_type_BlockdevRef(Visitor *v,
|     if (err) {
|         goto out;
|     }
|+    if (!*obj) {
|+        goto out_obj;
|+    }
|     switch ((*obj)->type) {
|     case QTYPE_QDICT:
|         visit_start_struct(v, name, NULL, 0, &err);
|@@ -2459,10 +2462,13 @@ void visit_type_BlockdevRef(Visitor *v,
|     case QTYPE_QSTRING:
|         visit_type_str(v, name, &(*obj)->u.reference, &err);
|         break;
|+    case QTYPE_NONE:
|+        abort();
|     default:
|         error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
|                    "BlockdevRef");
|     }
|+out_obj:
|     visit_end_alternate(v);

Reported by Kashyap Chamarthy <kchamart@redhat.com>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1466012271-5204-1-git-send-email-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Tested-by: Kashyap Chamarthy <kchamart@redhat.com>
[Commit message tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-06-30 15:24:36 +02:00
Peter Maydell
1ec20c2a3a * serial port fixes (Paolo)
* Q35 modeling improvements (Paolo, Vasily)
 * chardev cleanup improvements (Marc-André)
 * iscsi bugfix (Peter L.)
 * cpu_exec patch from multi-arch patches (Peter C.)
 * pci-assign tweak (Lin Ma)
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQEcBAABAgAGBQJXc+GeAAoJEL/70l94x66DtIAH/3+eUBqSxVJ3SMUxJep2Op07
 lIWqw1GHAdw1gWQDG4HzokKWrVVp/+NFYQjRFcNMfF8L+/Xm6hHAYc7Y4DMkDxSw
 zHX2BT93gPcaFJRz3Md8n2anzFHaWePx7LucPjaoas2OzrbVKXC8JT6n3GGnKQzZ
 0CxDoyW4keI4ZVAOy9SOKsLPxdSvG8uLvaZU98l/YS/TuiGzpv8IWcdHR+k1hua+
 FIenzj7jD9+JFoLEUWkU0pYs33J6yYKPiZn7HgGL9RNWKPFR88+CtMdYXgfOPo7z
 i05L9RTmL4SpahmStPN2r72MC0T0ub0czk/+qxBNms4r/2gBwaSyldmcTfAXM9o=
 =DA8v
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* serial port fixes (Paolo)
* Q35 modeling improvements (Paolo, Vasily)
* chardev cleanup improvements (Marc-André)
* iscsi bugfix (Peter L.)
* cpu_exec patch from multi-arch patches (Peter C.)
* pci-assign tweak (Lin Ma)

# gpg: Signature made Wed 29 Jun 2016 15:56:30 BST
# gpg:                using RSA key 0xBFFBD25F78C7AE83
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>"
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>"
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* remotes/bonzini/tags/for-upstream: (35 commits)
  socket: unlink unix socket on remove
  socket: add listen feature
  char: clean up remaining chardevs when leaving
  vhost-user: disable chardev handlers on close
  vhost-user-test: fix g_cond_wait_until compat implementation
  vl: smp_parse: fix regression
  ich9: implement SCI_IRQ_SEL register
  ich9: implement ACPI_EN register
  serial: reinstate watch after migration
  serial: remove watch on reset
  char: change qemu_chr_fe_add_watch to return unsigned
  serial: separate serial_xmit and serial_watch_cb
  serial: simplify tsr_retry reset
  serial: make tsr_retry unsigned
  iscsi: fix assertion in is_sector_request_lun_aligned
  target-*: Don't redefine cpu_exec()
  pci-assign: Move "Invalid ROM" error message to pci-assign-load-rom.c
  vnc: generalize "VNC server running on ..." message
  scsi: esp: fix migration
  MC146818 RTC: add GPIO access to output IRQ
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-29 19:14:48 +01:00
Marc-André Lureau
74b6ce43e3 socket: unlink unix socket on remove
qemu leaves unix socket files behind when removing a listening chardev
or leaving. qemu could clean that up, even if doing so isn't race-free.

Fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=1347077

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <1466105332-10285-4-git-send-email-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-06-29 16:49:41 +02:00
Paolo Bonzini
634d39b4e3 vhost-user-test: fix g_cond_wait_until compat implementation
This fixes compilation with glib versions up to 2.30, such
as the one in CentOS 6.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-06-29 16:49:40 +02:00
John Snow
ccee3d8f97 iotests: add small-granularity mirror test
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 1466625064-11280-4-git-send-email-jsnow@redhat.com
Signed-off-by: Jeff Cody <jcody@redhat.com>
2016-06-28 22:53:03 -04:00
Thomas Huth
1f5c1cfbae arm: Re-enable tmp105 test
The tmp105 test is currently not executed since the following
line in the Makefile overwrites the check-qtest-arm-y variable
instead of extending it.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 1466760306-21849-1-git-send-email-thuth@redhat.com
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-27 15:37:32 +01:00
Michael S. Tsirkin
d8d69e1f51 acpi-test-data: update expected
switched to new cpu hotplug interface, aml changed.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-24 08:22:07 +03:00
Corey Minyard
f4eda2d429 bios: Add tests for the IPMI ACPI and SMBIOS entries
Signed-off-by: Corey Minyard <cminyard@mvista.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-24 05:13:57 +03:00
Thomas Huth
0ccac16f59 tests: Use '+=' to add additional tests, not '='
The recent commit that added the prom-env-test accidentially
overwrote the check-qtest-ppc-y, check-qtest-ppc64-y and
check-qtest-sparc-y variables instead of extending them.

Fixes: fcbf4a3c0c
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-06-22 11:12:17 +10:00
Peter Maydell
7fa124b273 Error reporting patches for 2016-06-20
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIbBAABAgAGBQJXaAQPAAoJEDhwtADrkYZTbPMP+OmXlf1IwcUONhaHqD0SXAdC
 Q5NNHheNHbIdzrsOmgj2GAxCeGG08REPrdYLb8N9lI3Fev8ggQ0v3d9qiStL6Nvz
 2B7DeTxXba9Qmc2gY86RR4f7T6VK+8k1Kj9f3YcAoXHucdjDLRmtUozwES3Kk8mn
 oeBqd07yuTAO93xQhF6BDv3mfagPZV+kp15Se04ufWPpT9eG3Cp9Pl4/Yad/wJUr
 2B4pIvYnhISVlZIOvpuzdydBaCb/U0YefjWBDkawLwRBtM/kpNKoDdqr2UA0yChQ
 Xulgpt+UQLZ4aOYGLkzqBT7v3kDeEnYUFDdE03xZBLbHddOU05NEhLgzRVUEAdUS
 EAKWvWu692USDh9a2r0AYLqZW6J0+Fyi4fZ34gSuP69oi74CIvjJHDAEwsSdt6bm
 WHCchvfIk9kinzGiFA3hkAbVWAbI0EpB0J7k3mgCVVVelBpF9b9U4M2ydx9ZPufX
 t4ULXxQO9Qyxr7r0uiznugQAjIRwLMDOPPd1F5Vi+LX0wF3Pcrtc/F+3ehoLBjBD
 6zZA5lZVlUEISdzzytfjeX3ch5vRFwlujrTd2anxRuduuBV1/ztvE1v0Nsiii6Sw
 15BaHQJvBnMnAIO5LfhnAaVpIBk0jpQpwVyRcNUpeZ1VU2qlqVS4zxYZ0FDQgyer
 IZa1qYJ0Sz+oMkq9RXU=
 =TptL
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/armbru/tags/pull-error-2016-06-20' into staging

Error reporting patches for 2016-06-20

# gpg: Signature made Mon 20 Jun 2016 15:56:15 BST
# gpg:                using RSA key 0x3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-error-2016-06-20:
  log: Fix qemu_set_log_filename() error handling
  log: Fix qemu_set_dfilter_ranges() error reporting
  log: Plug memory leak on multiple -dfilter
  coccinelle: Remove unnecessary variables for function return value
  error: Remove unnecessary local_err variables
  error: Remove NULL checks on error_propagate() calls
  vl: Error messages need to go to stderr, fix some

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-20 16:19:18 +01:00
Markus Armbruster
daa76aa416 log: Fix qemu_set_log_filename() error handling
When qemu_set_log_filename() detects an invalid file name, it reports
an error, closes the log file (if any), and starts logging to stderr
(unless daemonized or nothing is being logged).

This is wrong.  Asking for an invalid log file on the command line
should be fatal.  Asking for one in the monitor should fail without
messing up an existing logfile.

Fix by converting qemu_set_log_filename() to Error.  Pass it
&error_fatal, except for hmp_logfile report errors.

This also permits testing without a subprocess, so do that.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1466011636-6112-4-git-send-email-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2016-06-20 16:39:08 +02:00
Markus Armbruster
bd6fee9f12 log: Fix qemu_set_dfilter_ranges() error reporting
g_error() is not an acceptable way to report errors to the user:

    $ qemu-system-x86_64 -dfilter 1000+0

    ** (process:17187): ERROR **: Failed to parse range in: 1000+0
    Trace/breakpoint trap (core dumped)

g_assert() isn't, either:

    $ qemu-system-x86_64 -dfilter 1000x+64
    **
    ERROR:/work/armbru/qemu/util/log.c:180:qemu_set_dfilter_ranges: assertion failed: (e == range_op)
    Aborted (core dumped)

Convert qemu_set_dfilter_ranges() to Error.  Rework its deeply nested
control flow.  Touch up the error messages.  Call it with
&error_fatal.

This also permits testing without a subprocess, so do that.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1466011636-6112-3-git-send-email-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2016-06-20 16:38:31 +02:00
Eduardo Habkost
9be385980d coccinelle: Remove unnecessary variables for function return value
Use Coccinelle script to replace 'ret = E; return ret' with
'return E'. The script will do the substitution only when the
function return type and variable type are the same.

Manual fixups:

* audio/audio.c: coding style of "read (...)" and "write (...)"
* block/qcow2-cluster.c: wrap line to make it shorter
* block/qcow2-refcount.c: change indentation of wrapped line
* target-tricore/op_helper.c: fix coding style of
  "remainder|quotient"
* target-mips/dsp_helper.c: reverted changes because I don't
  want to argue about checkpatch.pl
* ui/qemu-pixman.c: fix line indentation
* block/rbd.c: restore blank line between declarations and
  statements

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <1465855078-19435-4-git-send-email-ehabkost@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Unused Coccinelle rule name dropped along with a redundant comment;
whitespace touched up in block/qcow2-cluster.c; stale commit message
paragraph deleted]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-06-20 16:38:13 +02:00
Stefan Hajnoczi
f1d3b99154 libqos: add qvirtqueue_cleanup()
qvirtqueue_setup() allocates the vring and virtqueue state.  So far
there has been no function to free it.  Callers have been using
guest_free() for the vring but forgot to free the QVirtQueue state.

This patch solves the memory leak by introducing qvirtqueue_cleanup().

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
c75f4c061b libqos: drop duplicated virtio_pci.h definitions
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-9-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
74f079a7ee libqos: drop duplicated virtio_scsi.h definitions
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-8-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
4565a3e029 libqos: drop duplicated virtio_blk.h definitions
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-7-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
780b11a097 libqos: drop duplicated virtio_vring.h structs
The descriptor element, used, and avail vring structs are defined in
virtio_ring.h.  There is no need to duplicate them in libqos virtio.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-6-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
ee3b850a70 libqos: drop duplicated virtio_ring.h bit definitions
Note that virtio_ring.h defines feature bits using their bit number:

  #define VIRTIO_RING_F_INDIRECT_DESC     28

On the other hand libqos virtio.h uses the bit mask:

  #define QVIRTIO_F_RING_INDIRECT_DESC    0x10000000

The patch makes the necessary adjustments.

I have used "1u << BITMASK" instead of "1ULL << BITMASK" because the
64-bit feature fields are not implemented in libqos virtio.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-5-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
1373a4c256 libqos: drop duplicated virtio_config.h definitions
Note that VIRTIO_F_ANY_LAYOUT and VIRTIO_F_NOTIFY_ON_EMPTY are bit
numbers in virtio_config.h but bit masks in qtest virtio.h.  Therefore
it's necessary to change users from X to (1u << X).

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-4-git-send-email-stefanha@redhat.com
2016-06-20 11:44:12 +01:00
Stefan Hajnoczi
7ad1e708e6 libqos: drop duplicated PCI vendor ID definition
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-3-git-send-email-stefanha@redhat.com
2016-06-20 11:44:11 +01:00
Stefan Hajnoczi
8ac9e205bd libqos: use virtio_ids.h for device ID definitions
Avoid redefining device IDs.  Use the standard Linux headers that are
already in the source tree.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1462798061-30382-2-git-send-email-stefanha@redhat.com
2016-06-20 11:44:11 +01:00
Peter Maydell
b355438de5 bitops.h: Implement half-shuffle and half-unshuffle ops
A half-shuffle operation takes a word with zeros in the high half:
 0000 0000 0000 0000 ABCD EFGH IJKL MNOP
and spreads the bits out so they are in every other bit of the word:
 0A0B 0C0D 0E0F 0G0H 0I0J 0K0L 0M0N 0O0P
A half-unshuffle performs the reverse operation.

Provide functions in bitops.h which implement these operations
for 32-bit and 64-bit inputs, and add tests for them.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Shannon Zhao <shannon.zhao@linaro.org>
Tested-by: Shannon Zhao <shannon.zhao@linaro.org>
Message-id: 1465915112-29272-3-git-send-email-peter.maydell@linaro.org
2016-06-17 15:23:51 +01:00
Peter Maydell
4acc8fdfd3 ppc patch queue for 2016-06-17
Here's the current accumulated set of spapr, ppc and related patches.
   * The big thing in here is CPU hotplug for spapr
     - This includes a number of acked generic changes adding new
       infrastructure for hotplugging cpu cores
   * A number of TCG bug fixes are also included
   * This adds a new testcase to make it harder to accidentally break
     Macintosh (and other openbios) platforms
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIcBAABAgAGBQJXY5oxAAoJEGw4ysog2bOSMu0QAN5e3H3qt2or5dnxS6/bC7QZ
 9UrDfCFiJ1HppWfesj0lSAQvHlkEdkml8O7fxeG7bzHQDdaIwy4+q2RIvDiAgnOW
 u1he5yhaN6PDLzr9zowtfUySWz6bixMyrBeY2I6/hOKgPKVeifryudmlRNJ5zc1R
 edwiSI4saNBYT2+KlaaQW26g765Xk0CG0FjxAyTTPH62WsUvxB1haSKHd7QTn57c
 ovdrCTHzfhXaQ0WmbDvIz2v0kY6lO71Re9DRFQmlZktVtGr6E07afr+2VzBTuYLb
 r6iHyi/Ed7x+kncH/5F52K7HIddNEp0gbZJgWnfM0gUSIAnya65FP/Qgkc4RdGFd
 yeR6V99s/i5dU5gxoFK0MKNn61/QLLUZJiS9l6XNs92mDZn5C03C/edINOv8tbRe
 TG+/TghwoFfojodi3cEM+TrDZv7E73RgWjoLY/Eq29KOAuOQaWV0ctbPgnKOZApq
 u2LwhPjzU8ae4LCA1c0sMZnMLnskplnNWhmz9sLyvsfy12Lau86/PnT43/rGZ2uu
 lW1SEgrk1zpYkOdbyAB01ZOPw0bQVL8uHQpm7df4/RtiyJuWCG+nPBXJoWXq3hN2
 VO5PEd33Ec8Cdln9modOwHOIkIgH5zMsG8yLcvHoAeDWccPUkr9Z1mv7vZ4ENQFO
 JaOyIOC8JKJJgA/2DQA+
 =AGZ4
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-2.7-20160617' into staging

ppc patch queue for 2016-06-17

Here's the current accumulated set of spapr, ppc and related patches.
  * The big thing in here is CPU hotplug for spapr
    - This includes a number of acked generic changes adding new
      infrastructure for hotplugging cpu cores
  * A number of TCG bug fixes are also included
  * This adds a new testcase to make it harder to accidentally break
    Macintosh (and other openbios) platforms

# gpg: Signature made Fri 17 Jun 2016 07:35:29 BST
# gpg:                using RSA key 0x6C38CACA20D9B392
# gpg: Good signature from "David Gibson <david@gibson.dropbear.id.au>"
# gpg:                 aka "David Gibson (Red Hat) <dgibson@redhat.com>"
# gpg:                 aka "David Gibson (ozlabs.org) <dgibson@ozlabs.org>"
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 75F4 6586 AE61 A66C C44E  87DC 6C38 CACA 20D9 B392

* remotes/dgibson/tags/ppc-for-2.7-20160617:
  spapr: implement query-hotpluggable-cpus callback
  hmp: Add 'info hotpluggable-cpus' HMP command
  QMP: Add query-hotpluggable-cpus
  spapr: CPU hot unplug support
  spapr: CPU hotplug support
  spapr: convert boot CPUs into CPU core devices
  spapr: Move spapr_cpu_init() to spapr_cpu_core.c
  spapr: Abstract CPU core device and type specific core devices
  qom: API to get instance_size of a type
  spapr_drc: Prevent detach racing against attach for CPU DR
  xics,xics_kvm: Handle CPU unplug correctly
  cpu: Abstract CPU core type
  qdev: hotplug: Introduce HotplugHandler.pre_plug() callback
  target-ppc: Fix rlwimi, rlwinm, rlwnm
  vfio: Fix broken EEH
  target-ppc: Bug in BookE wait instruction
  ppc / sparc: Add a tester for checking whether OpenBIOS runs successfully
  hw/ppc/spapr: Silence deprecation message in qtest mode

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-17 12:36:27 +01:00
Peter Maydell
7263a903c3 pc, pci, virtio: new features, cleanups, fixes
Beginning of reconnect support for vhost-user.
 Misc cleanups and fixes.
 
 Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJXY0Q3AAoJECgfDbjSjVRpkVcH/2gTHRE9yUoWe6ROvPV67BKx
 8Iy9GzJ3BMO3RolVZEA5KXIevn5TG+pV274BZEuXMD3AL/molv279p0o/gvBYoqq
 V0jNH2MO+MV6D9OzhUXcgWSejvybF5W07ojPDU/hlgtFXPZFbJDyt95MWaLiilOg
 cCtTuRqgrrRaypcnnk/CIDbC+Ek2kAYdgQHQbfj9ihle3TWO8R0bSXnFqSaqCIkM
 4slMlv8y82fODeiO83nkpfAP1NCnfnRC8r8Gv7hbEUTlZQntavx5DuYdiIx6nsJE
 W0g+Gpe1o0+jRuMnucGIUZvqzZ0e/I0wZuV16Nsfx+Rbd5+4CzTxZda5Qb05v7I=
 =BHbJ
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging

pc, pci, virtio: new features, cleanups, fixes

Beginning of reconnect support for vhost-user.
Misc cleanups and fixes.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

# gpg: Signature made Fri 17 Jun 2016 01:28:39 BST
# gpg:                using RSA key 0x281F0DB8D28D5469
# gpg: Good signature from "Michael S. Tsirkin <mst@kernel.org>"
# gpg:                 aka "Michael S. Tsirkin <mst@redhat.com>"
# Primary key fingerprint: 0270 606B 6F3C DF3D 0B17  0970 C350 3912 AFBE 8E67
#      Subkey fingerprint: 5D09 FD08 71C8 F85B 94CA  8A0D 281F 0DB8 D28D 5469

* remotes/mst/tags/for_upstream:
  MAINTAINERS: add Marcel to PCI
  msi_init: change return value to 0 on success
  fix some coding style problems
  pci core: assert ENOSPC when add capability
  test: start vhost-user reconnect test
  tests: append i386 tests
  vhost-net: save & restore vring enable state
  vhost-net: save & restore vhost-user acked features
  vhost-net: do not crash if backend is not present
  vhost-user: disconnect on start failure
  qemu-char: add qemu_chr_disconnect to close a fd accepted by listen fd
  tests/vhost-user-bridge: workaround stale vring base
  tests/vhost-user-bridge: add client mode
  vhost-user: add ability to know vhost-user backend disconnection
  pci: fix pci_requester_id()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Conflicts:
	tests/Makefile.include
2016-06-17 11:25:46 +01:00
Thomas Huth
fcbf4a3c0c ppc / sparc: Add a tester for checking whether OpenBIOS runs successfully
Since the mac99 and g3beige PowerPC machines recently broke without
being noticed, it would be good to have a tester for "make check"
that detects such issues immediately. A simple way to test the firmware
of these machines is to use the "-prom-env" parameter of QEMU. This
parameter can be used to put some Forth code into the 'boot-command'
firmware variable which then can signal success to the tester by
writing a magic value to a known memory location. And since some of the
Sparc machines are also using OpenBIOS, they are now tested with this
prom-env-tester, too.

Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
[dwg: Removed sparc64, because it trips a TCG bug on 32-bit hosts]
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-06-17 15:57:59 +10:00
Marc-André Lureau
4616e3592b test: start vhost-user reconnect test
This is a simple reconnect test, that simply checks if vhost-user
reconnection is possible and restore the state. A more complete test
would actually manipulate and check the ring contents (such extended
testing would benefit from the libvhost-user proposed in QEMU list to
avoid duplication of ring manipulations)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Victor Kaplansky <victork@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-17 03:28:03 +03:00
Marc-André Lureau
0ee2e9daf8 tests: append i386 tests
Do not overwrite x86-64 tests, re-enable vhost-user-test.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Victor Kaplansky <victork@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-17 03:28:03 +03:00
Marc-André Lureau
523b018dde tests/vhost-user-bridge: workaround stale vring base
This patch is a similar solution to what Yuanhan Liu/Huawei Xie have
suggested for DPDK. When vubr quits (killed or crashed), a restart of
vubr would get stale vring base from QEMU. That would break the kernel
virtio net completely, making it non-work any more, unless a driver
reset is done.

So, instead of getting the stale vring base from QEMU, Huawei suggested
we could get a proper one from used->idx. This works because the queues
packets are processed in order.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Victor Kaplansky <victork@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-17 03:28:02 +03:00
Marc-André Lureau
aef8486ede tests/vhost-user-bridge: add client mode
If -c is specified, vubr will try to connect to the socket instead of
listening for connections.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Victor Kaplansky <victork@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-06-17 03:28:02 +03:00
Paolo Bonzini
e9abfcb57f clean-includes: run it once more
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-06-16 18:39:03 +02:00
Paolo Bonzini
02d0e09503 os-posix: include sys/mman.h
qemu/osdep.h checks whether MAP_ANONYMOUS is defined, but this check
is bogus without a previous inclusion of sys/mman.h.  Include it in
sysemu/os-posix.h and remove it from everywhere else.

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-06-16 18:39:03 +02:00
Max Reitz
3dd48fdc55 iotests: Add test for oVirt-like storage migration
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20160610185750.30956-6-mreitz@redhat.com
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2016-06-16 15:20:37 +02:00
Max Reitz
298c6009dc iotests: Add test for post-mirror backing chains
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20160610185750.30956-5-mreitz@redhat.com
Reviewed-by: Fam Zheng <famz@redhat.com>
[mreitz@redhat.com: Removed unnecessary imports]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2016-06-16 15:20:37 +02:00
Fam Zheng
6ea66b590c iotests: 095: Clean up QEMU before showing image info
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464944872-24484-1-git-send-email-famz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
2016-06-16 15:20:37 +02:00
Thomas Huth
48bea96572 doc: Fix mailing list address in tests/qemu-iotests/README
The address of the mailing list is qemu-devel@nongnu.org
instead of qemu-devel@savannah.nongnu.org. And while we're
at it, also mention the qemu-block mailing list here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-06-16 15:19:55 +02:00
Daniel P. Berrange
8c0dcbc4ad block: drop support for using qcow[2] encryption with system emulators
Back in the 2.3.0 release we declared qcow[2] encryption as
deprecated, warning people that it would be removed in a future
release.

  commit a1f688f415
  Author: Markus Armbruster <armbru@redhat.com>
  Date:   Fri Mar 13 21:09:40 2015 +0100

    block: Deprecate QCOW/QCOW2 encryption

The code still exists today, but by a (happy?) accident we entirely
broke the ability to use qcow[2] encryption in the system emulators
in the 2.4.0 release due to

  commit 8336aafae1
  Author: Daniel P. Berrange <berrange@redhat.com>
  Date:   Tue May 12 17:09:18 2015 +0100

    qcow2/qcow: protect against uninitialized encryption key

This commit was designed to prevent future coding bugs which
might cause QEMU to read/write data on an encrypted block
device in plain text mode before a decryption key is set.

It turns out this preventative measure was a little too good,
because we already had a long standing bug where QEMU read
encrypted data in plain text mode during system emulator
startup, in order to guess disk geometry:

  Thread 10 (Thread 0x7fffd3fff700 (LWP 30373)):
  #0  0x00007fffe90b1a28 in raise () at /lib64/libc.so.6
  #1  0x00007fffe90b362a in abort () at /lib64/libc.so.6
  #2  0x00007fffe90aa227 in __assert_fail_base () at /lib64/libc.so.6
  #3  0x00007fffe90aa2d2 in  () at /lib64/libc.so.6
  #4  0x000055555587ae19 in qcow2_co_readv (bs=0x5555562accb0, sector_num=0, remaining_sectors=1, qiov=0x7fffffffd260) at block/qcow2.c:1229
  #5  0x000055555589b60d in bdrv_aligned_preadv (bs=bs@entry=0x5555562accb0, req=req@entry=0x7fffd3ffea50, offset=offset@entry=0, bytes=bytes@entry=512, align=align@entry=512, qiov=qiov@entry=0x7fffffffd260, flags=0) at block/io.c:908
  #6  0x000055555589b8bc in bdrv_co_do_preadv (bs=0x5555562accb0, offset=0, bytes=512, qiov=0x7fffffffd260, flags=<optimized out>) at block/io.c:999
  #7  0x000055555589c375 in bdrv_rw_co_entry (opaque=0x7fffffffd210) at block/io.c:544
  #8  0x000055555586933b in coroutine_thread (opaque=0x555557876310) at coroutine-gthread.c:134
  #9  0x00007ffff64e1835 in g_thread_proxy (data=0x5555562b5590) at gthread.c:778
  #10 0x00007ffff6bb760a in start_thread () at /lib64/libpthread.so.0
  #11 0x00007fffe917f59d in clone () at /lib64/libc.so.6

  Thread 1 (Thread 0x7ffff7ecab40 (LWP 30343)):
  #0  0x00007fffe91797a9 in syscall () at /lib64/libc.so.6
  #1  0x00007ffff64ff87f in g_cond_wait (cond=cond@entry=0x555555e085f0 <coroutine_cond>, mutex=mutex@entry=0x555555e08600 <coroutine_lock>) at gthread-posix.c:1397
  #2  0x00005555558692c3 in qemu_coroutine_switch (co=<optimized out>) at coroutine-gthread.c:117
  #3  0x00005555558692c3 in qemu_coroutine_switch (from_=0x5555562b5e30, to_=to_@entry=0x555557876310, action=action@entry=COROUTINE_ENTER) at coroutine-gthread.c:175
  #4  0x0000555555868a90 in qemu_coroutine_enter (co=0x555557876310, opaque=0x0) at qemu-coroutine.c:116
  #5  0x0000555555859b84 in thread_pool_completion_bh (opaque=0x7fffd40010e0) at thread-pool.c:187
  #6  0x0000555555859514 in aio_bh_poll (ctx=ctx@entry=0x5555562953b0) at async.c:85
  #7  0x0000555555864d10 in aio_dispatch (ctx=ctx@entry=0x5555562953b0) at aio-posix.c:135
  #8  0x0000555555864f75 in aio_poll (ctx=ctx@entry=0x5555562953b0, blocking=blocking@entry=true) at aio-posix.c:291
  #9  0x000055555589c40d in bdrv_prwv_co (bs=bs@entry=0x5555562accb0, offset=offset@entry=0, qiov=qiov@entry=0x7fffffffd260, is_write=is_write@entry=false, flags=flags@entry=(unknown: 0)) at block/io.c:591
  #10 0x000055555589c503 in bdrv_rw_co (bs=bs@entry=0x5555562accb0, sector_num=sector_num@entry=0, buf=buf@entry=0x7fffffffd2e0 "\321,", nb_sectors=nb_sectors@entry=21845, is_write=is_write@entry=false, flags=flags@entry=(unknown: 0)) at block/io.c:614
  #11 0x000055555589c562 in bdrv_read_unthrottled (nb_sectors=21845, buf=0x7fffffffd2e0 "\321,", sector_num=0, bs=0x5555562accb0) at block/io.c:622
  #12 0x000055555589c562 in bdrv_read_unthrottled (bs=0x5555562accb0, sector_num=sector_num@entry=0, buf=buf@entry=0x7fffffffd2e0 "\321,", nb_sectors=nb_sectors@entry=21845) at block/io.c:634
    nb_sectors@entry=1) at block/block-backend.c:504
  #14 0x0000555555752e9f in guess_disk_lchs (blk=blk@entry=0x5555562a5290, pcylinders=pcylinders@entry=0x7fffffffd52c, pheads=pheads@entry=0x7fffffffd530, psectors=psectors@entry=0x7fffffffd534) at hw/block/hd-geometry.c:68
  #15 0x0000555555752ff7 in hd_geometry_guess (blk=0x5555562a5290, pcyls=pcyls@entry=0x555557875d1c, pheads=pheads@entry=0x555557875d20, psecs=psecs@entry=0x555557875d24, ptrans=ptrans@entry=0x555557875d28) at hw/block/hd-geometry.c:133
  #16 0x0000555555752b87 in blkconf_geometry (conf=conf@entry=0x555557875d00, ptrans=ptrans@entry=0x555557875d28, cyls_max=cyls_max@entry=65536, heads_max=heads_max@entry=16, secs_max=secs_max@entry=255, errp=errp@entry=0x7fffffffd5e0) at hw/block/block.c:71
  #17 0x0000555555799bc4 in ide_dev_initfn (dev=0x555557875c80, kind=IDE_HD) at hw/ide/qdev.c:174
  #18 0x0000555555768394 in device_realize (dev=0x555557875c80, errp=0x7fffffffd640) at hw/core/qdev.c:247
  #19 0x0000555555769a81 in device_set_realized (obj=0x555557875c80, value=<optimized out>, errp=0x7fffffffd730) at hw/core/qdev.c:1058
  #20 0x00005555558240ce in property_set_bool (obj=0x555557875c80, v=<optimized out>, opaque=0x555557875de0, name=<optimized out>, errp=0x7fffffffd730)
        at qom/object.c:1514
  #21 0x0000555555826c87 in object_property_set_qobject (obj=obj@entry=0x555557875c80, value=value@entry=0x55555784bcb0, name=name@entry=0x55555591cb3d "realized", errp=errp@entry=0x7fffffffd730) at qom/qom-qobject.c:24
  #22 0x0000555555825760 in object_property_set_bool (obj=obj@entry=0x555557875c80, value=value@entry=true, name=name@entry=0x55555591cb3d "realized", errp=errp@entry=0x7fffffffd730) at qom/object.c:905
  #23 0x000055555576897b in qdev_init_nofail (dev=dev@entry=0x555557875c80) at hw/core/qdev.c:380
  #24 0x0000555555799ead in ide_create_drive (bus=bus@entry=0x555557629630, unit=unit@entry=0, drive=0x5555562b77e0) at hw/ide/qdev.c:122
  #25 0x000055555579a746 in pci_ide_create_devs (dev=dev@entry=0x555557628db0, hd_table=hd_table@entry=0x7fffffffd830) at hw/ide/pci.c:440
  #26 0x000055555579b165 in pci_piix3_ide_init (bus=<optimized out>, hd_table=0x7fffffffd830, devfn=<optimized out>) at hw/ide/piix.c:218
  #27 0x000055555568ca55 in pc_init1 (machine=0x5555562960a0, pci_enabled=1, kvmclock_enabled=<optimized out>) at /home/berrange/src/virt/qemu/hw/i386/pc_piix.c:256
  #28 0x0000555555603ab2 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4249

So the safety net is correctly preventing QEMU reading cipher
text as if it were plain text, during startup and aborting QEMU
to avoid bad usage of this data.

For added fun this bug only happens if the encrypted qcow2
file happens to have data written to the first cluster,
otherwise the cluster won't be allocated and so qcow2 would
not try the decryption routines at all, just return all 0's.

That no one even noticed, let alone reported, this bug that
has shipped in 2.4.0, 2.5.0 and 2.6.0 shows that the number
of actual users of encrypted qcow2 is approximately zero.

So rather than fix the crash, and backport it to stable
releases, just go ahead with what we have warned users about
and disable any use of qcow2 encryption in the system
emulators. qemu-img/qemu-io/qemu-nbd are still able to access
qcow2 encrypted images for the sake of data conversion.

In the future, qcow2 will gain support for the alternative
luks format, but when this happens it'll be using the
'-object secret' infrastructure for getting keys, which
avoids this problematic scenario entirely.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-06-16 15:19:55 +02:00
Andrea Arcangeli
f5d4579178 tests: fix libqtest socket timeouts
I kept getting timeouts and unix socket accept failures under high
load, the patch fixes it.

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Reviewed-by: Marcel Apfelbaum <marcel@redhat.com>
Message-id: 1465816605-29488-6-git-send-email-dgilbert@redhat.com
Message-Id: <1465816605-29488-6-git-send-email-dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
2016-06-16 09:50:07 +05:30
Dr. David Alan Gilbert
ea0c6d6239 test: Postcopy
This is a postcopy test (x86 only) that actually runs the guest
and checks the memory contents.

The test runs from an x86 boot block with the hex embedded in the test;
the source for this is:

...........

.code16
.org 0x7c00
	.file	"fill.s"
	.text
	.globl	start
	.type	start, @function
start:             # at 0x7c00 ?
        cli
        lgdt gdtdesc
        mov $1,%eax
        mov %eax,%cr0  # Protected mode enable
        data32 ljmp $8,$0x7c20

.org 0x7c20
.code32
        # A20 enable - not sure I actually need this
        inb $0x92,%al
        or  $2,%al
        outb %al, $0x92

        # set up DS for the whole of RAM (needed on KVM)
        mov $16,%eax
        mov %eax,%ds

        mov $65,%ax
        mov $0x3f8,%dx
        outb %al,%dx

        # bl keeps a counter so we limit the output speed
        mov $0, %bl
mainloop:
        # Start from 1MB
        mov $(1024*1024),%eax
innerloop:
        incb (%eax)
        add $4096,%eax
        cmp $(100*1024*1024),%eax
        jl innerloop

        inc %bl
        jnz mainloop

        mov $66,%ax
        mov $0x3f8,%dx
        outb %al,%dx

	jmp mainloop

        # GDT magic from old (GPLv2)  Grub startup.S
        .p2align        2       /* force 4-byte alignment */
gdt:
        .word   0, 0
        .byte   0, 0, 0, 0

        /* -- code segment --
         * base = 0x00000000, limit = 0xFFFFF (4 KiB Granularity), present
         * type = 32bit code execute/read, DPL = 0
         */
        .word   0xFFFF, 0
        .byte   0, 0x9A, 0xCF, 0

        /* -- data segment --
         * base = 0x00000000, limit 0xFFFFF (4 KiB Granularity), present
         * type = 32 bit data read/write, DPL = 0
         */
        .word   0xFFFF, 0
        .byte   0, 0x92, 0xCF, 0

gdtdesc:
        .word   0x27                    /* limit */
        .long   gdt                     /* addr */

/* I'm a bootable disk */
.org 0x7dfe
        .byte 0x55
        .byte 0xAA

...........

and that can be assembled by the following magic:
    as --32 -march=i486 fill.s -o fill.o
    objcopy -O binary fill.o fill.boot
    dd if=fill.boot of=bootsect bs=256 count=2 skip=124
    xxd -i bootsect

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Marcel Apfelbaum <marcel@redhat.com>
Message-id: 1465816605-29488-5-git-send-email-dgilbert@redhat.com
Message-Id: <1465816605-29488-5-git-send-email-dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
2016-06-16 09:50:07 +05:30
Daniel P. Berrange
e7ed11f083 crypto: remove temp files on completion of secrets test
The secret object tests left some temporary files on disk
when completing. Ensure they are unlink, and rename them
to make it more obvious where they come from.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2016-06-13 12:41:17 +01:00
Emilio G. Cota
896a9ee967 qht: add test-qht-par to invoke qht-bench from 'check' target
Signed-off-by: Emilio G. Cota <cota@braap.org>
Message-Id: <1465412133-3029-14-git-send-email-cota@braap.org>
Signed-off-by: Richard Henderson <rth@twiddle.net>
2016-06-11 17:11:16 -07:00
Emilio G. Cota
515864a0d7 qht: add qht-bench, a performance benchmark
This serves as a performance benchmark as well as a stress test
for QHT. We can tweak quite a number of things, including the
number of resize threads and how frequently resizes are triggered.

A performance comparison of QHT vs CLHT[1] and ck_hs[2] using
this same benchmark program can be found here:
  http://imgur.com/a/0Bms4

The tests are run on a 64-core AMD Opteron 6376, pinning threads
to cores favoring same-socket cores. For each run, qht-bench is
invoked with:
  $ tests/qht-bench -d $duration -n $n -u $u -g $range
, where $duration is in seconds, $n is the number of threads,
$u is the update rate (0.0 to 100.0), and $range is the number
of keys.

Note that ck_hs's performance drops significantly as writes go
up, since it requires an external lock (I used a ck_spinlock)
around every write.

Also, note that CLHT instead of using a seqlock, relies on an
allocator that does not ever return the same address during the
same read-critical section. This gives it a slight performance
advantage over QHT on read-heavy workloads, since the seqlock
writes aren't there.

[1] CLHT: https://github.com/LPD-EPFL/CLHT
          https://infoscience.epfl.ch/record/207109/files/ascy_asplos15.pdf

[2] ck_hs: http://concurrencykit.org/
           http://backtrace.io/blog/blog/2015/03/13/workload-specialization/

A few of those plots are shown in text here, since that site
might not be online forever. Throughput is on Mops/s on the Y axis.

                             200K keys, 0 % updates

  450 ++--+------+------+-------+-------+-------+-------+------+-------+--++
      |   +      +      +       +       +       +       +      +      +N+  |
  400 ++                                                           ---+E+ ++
      |                                                       +++----      |
  350 ++          9 ++------+------++                       --+E+    -+H+ ++
      |             |      +H+-     |                 -+N+----   ---- +++  |
  300 ++          8 ++     +E+     ++             -----+E+  --+H+         ++
      |             |      +++      |         -+N+-----+H+--               |
  250 ++          7 ++------+------++  +++-----+E+----                    ++
  200 ++                    1         -+E+-----+H+                        ++
      |                           ----                     qht +-E--+      |
  150 ++                      -+E+                        clht +-H--+     ++
      |                   ----                              ck +-N--+      |
  100 ++               +E+                                                ++
      |            ----                                                    |
   50 ++       -+E+                                                       ++
      |   +E+E+  +      +       +       +       +       +      +       +   |
    0 ++--E------+------+-------+-------+-------+-------+------+-------+--++
          1      8      16      24      32      40      48     56      64
                                Number of threads

                             200K keys, 1 % updates

  350 ++--+------+------+-------+-------+-------+-------+------+-------+--++
      |   +      +      +       +       +       +       +      +     -+E+  |
  300 ++                                                         -----+H+ ++
      |                                                       +E+--        |
      |           9 ++------+------++                  +++----             |
  250 ++            |      +E+   -- |                 -+E+                ++
      |           8 ++         --  ++             ----                     |
  200 ++            |      +++-     |  +++  ---+E+                        ++
      |           7 ++------N------++ -+E+--               qht +-E--+      |
      |                     1  +++----                    clht +-H--+      |
  150 ++                      -+E+                          ck +-N--+     ++
      |                   ----                                             |
  100 ++               +E+                                                ++
      |            ----                                                    |
      |        -+E+                                                        |
   50 ++    +H+-+N+----+N+-----+N+------                                  ++
      |   +E+E+  +      +       +      +N+-----+N+-----+N+----+N+-----+N+  |
    0 ++--E------+------+-------+-------+-------+-------+------+-------+--++
          1      8      16      24      32      40      48     56      64
                                Number of threads

                             200K keys, 20 % updates

  300 ++--+------+------+-------+-------+-------+-------+------+-------+--++
      |   +      +      +       +       +       +       +      +       +   |
      |                                                              -+H+  |
  250 ++                                                         ----     ++
      |           9 ++------+------++                       --+H+  ---+E+  |
      |           8 ++     +H+--   ++                 -+H+----+E+--        |
  200 ++            |      +E+    --|             -----+E+--  +++         ++
      |           7 ++      + ---- ++       ---+H+---- +++ qht +-E--+      |
  150 ++          6 ++------N------++ -+H+-----+E+        clht +-H--+     ++
      |                     1     -----+E+--                ck +-N--+      |
      |                       -+H+----                                     |
  100 ++                  -----+E+                                        ++
      |                +E+--                                               |
      |            ----+++                                                 |
   50 ++       -+E+                                                       ++
      |     +E+ +++                                                        |
      |   +E+N+-+N+-----+       +       +       +       +      +       +   |
    0 ++--E------+------N-------N-------N-------N-------N------N-------N--++
          1      8      16      24      32      40      48     56      64
                                Number of threads

                            200K keys, 100 % updates       qht +-E--+
                                                          clht +-H--+
  160 ++--+------+------+-------+-------+-------+-------+---ck-+-N-----+--++
      |   +      +      +       +       +       +       +      +   ----H   |
  140 ++                                                      +H+--  -+E+ ++
      |                                                +++----   ----      |
  120 ++          8 ++------+------++                 -+H+    +E+         ++
      |           7 ++     +H+---- ++             ---- +++----             |
  100 ++            |      +E+      |  +++  ---+H+    -+E+                ++
      |           6 ++     +++     ++ -+H+--   +++----                     |
   80 ++          5 ++------N----------+E+-----+E+                        ++
      |                     1 -+H+---- +++                                 |
      |                   -----+E+                                         |
   60 ++               +H+---- +++                                        ++
      |            ----+E+                                                 |
   40 ++        +H+----                                                   ++
      |       --+E+                                                        |
   20 ++    +E+                                                           ++
      |  +EE+    +      +       +       +       +       +      +       +   |
    0 ++--+N-N---N------N-------N-------N-------N-------N------N-------N--++
          1      8      16      24      32      40      48     56      64
                                Number of threads

Signed-off-by: Emilio G. Cota <cota@braap.org>
Message-Id: <1465412133-3029-13-git-send-email-cota@braap.org>
Signed-off-by: Richard Henderson <rth@twiddle.net>
2016-06-11 17:11:16 -07:00
Emilio G. Cota
1a95404fbd qht: add test program
Acked-by: Sergey Fedorov <sergey.fedorov@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Emilio G. Cota <cota@braap.org>
Message-Id: <1465412133-3029-12-git-send-email-cota@braap.org>
Signed-off-by: Richard Henderson <rth@twiddle.net>
2016-06-11 23:10:20 +00:00
Emilio G. Cota
ff9249b733 qdist: add test program
Acked-by: Sergey Fedorov <sergey.fedorov@linaro.org>
Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Emilio G. Cota <cota@braap.org>
Message-Id: <1465412133-3029-10-git-send-email-cota@braap.org>
Signed-off-by: Richard Henderson <rth@twiddle.net>
2016-06-11 23:10:19 +00:00
Peter Maydell
6f50f25c82 Block layer patches
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQIcBAABAgAGBQJXV9fqAAoJEH8JsnLIjy/W3/MQAKImbFe+Qm0eHzABKA9rLqRN
 N6RsDgU0/qRs5wOzfsnX5QCIt82cawPWdRL/Ke0JagMJ/A4mV8TsJ7zHk4pgFSyg
 yjhVUqFZy4J+BSkqaFrHbEgyXCDsa6b7Xg30Mrxm4OF5t4Nj96KejzWcyXz6QTGb
 zPwLVCsHlItPPql3w1IUZPMIYaDZbpu+VgULJZ5D4T5RKv7H4i3wi/Nuw4x6XH3E
 zA8Ky4Ws17wM17rUYr3LlcWfULLC8EAPYRYjS6Jii1eknKDQqQ/iQVYbQNJ2gqrs
 /lYygzCOWBGZCAONRnODSO7666CI9ofWhwdyl4gJ85k8YzrZw/ckVmVH3dFYU4u3
 2JVaiZ1PwjgwRuSznsSaK7NhbpPpCbvOuC7Ry3wKlnUgyy1VrhVMlSU0rmVA260k
 2bhnMFnNbf2GsBnibdZCbSAoO5Lb7BJZ29mk+CiP7BnWq1iijfZAm+k+lIOQ3LxD
 ahorWzU3HYcK/ZZAWttPvyhD/n4rbNnodp+Fl7QRc5AaavPRIdfwGpmf0hst4bXa
 xQ+spQWFSfXPpc37qcNbGTdJh9d7NmypSiTeIUeYoSydca9emlQ+M26bSB0lZsCv
 ZzysFz2+UvCoJCwpZXnPL8os7GLCbClDhfLyONfqgcir6cRUIV6TdT5HgwDymG9r
 1M9AfPY5xP73Tx+7voC9
 =joMD
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

# gpg: Signature made Wed 08 Jun 2016 09:31:38 BST
# gpg:                using RSA key 0x7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (31 commits)
  qemu-img bench: Add --flush-interval
  qemu-img bench: Implement -S (step size)
  qemu-img bench: Make start offset configurable
  qemu-img bench: Sequential writes
  qemu-img bench
  block: Don't emulate natively supported pwritev flags
  blockdev: clean up error handling in do_open_tray
  block: Fix bdrv_all_delete_snapshot() error handling
  qcow2: avoid extra flushes in qcow2
  raw-posix: Fetch max sectors for host block device
  block: assert that bs->request_alignment is a power of 2
  migration/block: Convert saving to BlockBackend
  migration/block: Convert load to BlockBackend
  block: Kill bdrv_co_write_zeroes()
  vmdk: Convert to bdrv_co_pwrite_zeroes()
  raw_bsd: Convert to bdrv_co_pwrite_zeroes()
  raw-posix: Convert to bdrv_co_pwrite_zeroes()
  qed: Convert to bdrv_co_pwrite_zeroes()
  gluster: Convert to bdrv_co_pwrite_zeroes()
  blkreplay: Convert to bdrv_co_pwrite_zeroes()
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-08 17:17:16 +01:00
Peter Maydell
d36ebffe94 Docker testing fixes by Paolo.
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2
 
 iQEcBAABCAAGBQJXV8dWAAoJEMo1YkxqkXHG3+4IAJHvNjIo5iE3dJYoFfes9tXc
 OgKFierRAhDvQcZqwRsR7sA1R5C3v6l3UlZZIJznFLKu1C/a69DLuB+AJgeUmpa5
 I9k1+S3QShwv3mXgGBztNmusKNMqwG3vJCv2AfZgfp5AvBOuaMGyRgHhU6H9nvk8
 NPX70GcsVWYUWw8LEc6fGZFIYWMZEAkZwP3VPLRljazLA5qtnJ+jn/Uk9YOv1/k6
 saW0nbV5ts1UQun93XJ9hnm2A4xIIE0vL7t7cn0fMlTFvoQRlPI9xBwOVxYZuyxW
 1yXDnHjKS00Kbo+6PsLYoN92T06tvG+778w2gWQxpLq0tj913jjYVjV+08JPwlI=
 =GUpp
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/famz/tags/pull-docker-20160608' into staging

Docker testing fixes by Paolo.

# gpg: Signature made Wed 08 Jun 2016 08:20:54 BST
# gpg:                using RSA key 0xCA35624C6A9171C6
# gpg: Good signature from "Fam Zheng <famz@redhat.com>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 5003 7CB7 9706 0F76 F021  AD56 CA35 624C 6A91 71C6

* remotes/famz/tags/pull-docker-20160608:
  tests/docker: build all targets in test-clang
  tests/docker: support travis test with fedora image
  tests/docker: remove unused feature "ccache"
  tests/docker: fix test-mingw
  tests/docker: make test-full build all targets, not none
  tests/docker: fix make-archive-maybe

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-08 16:31:53 +01:00