If the guest is using multiple transfers to try and keep the usb bus busy /
used at maximum efficiency, currently we would see / do the following:
1) submit transfer 1 to the device
2) submit transfer 2 to the device
3) report transfer 1 completion to guest
4) report transfer 2 completion to guest
5) submit transfer 1 to the device
6) report transfer 1 completion to guest
7) submit transfer 2 to the device
8) report transfer 2 completion to guest
etc.
So after the initial submission we would effectively only have 1 transfer
in flight, rather then 2. This is caused by us not checking the queue for
addition of new transfers by the guest (ie the resubmission of a recently
finished transfer), while waiting for a pending transfer to complete.
This patch does add a check for this, changing the sequence to:
1) submit transfer 1 to the device
2) submit transfer 2 to the device
3) report transfer 1 completion to guest
4) submit transfer 1 to the device
5) report transfer 2 completion to guest
6) submit transfer 2 to the device
etc.
Thus keeping 2 transfers in flight (most of the time, and always 1),
as intended by the guest.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
For ctrl endpoints Windows (atleast Win7) creates circular td lists, so far
these were not a problem because we would stop filling the queue if altnext
was set. Since further patches in this patchset remove the altnext check this
does become a problem and we need detection for going in circles.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Often the guest will queue up new packets in response to a packet, in the
async schedule with its IOC flag set, completing. By speeding up the
frame-timer, we notice these new packets earlier. This increases the
speed (MB/s) of a Linux guest reading from a USB mass storage device by a
factor of 1.15 on top of the "Improve latency of interrupt delivery"
speed-ups, both with and without input pipelining enabled.
I've not tested the speed-up of this patch without the
"Improve latency of interrupt delivery" patch.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
While doing various performance tests of reading from USB mass storage devices
I noticed the following::
1) When an async handled packet completes, we don't immediately report an
interrupt to the guest, instead we wait for the frame-timer to run and
report it from there
2) If 1) has been fixed and an async handled packet takes a while to complete,
then async_stepdown will become a high value, which means that there
will be a large latency before any new packets queued by the guest in
response to the interrupt get seen
1) was done deliberately as part of commit f0ad01f92:
http://www.kraxel.org/cgit/qemu/commit/?h=usb.57&id=f0ad01f92ca02eee7cadbfd225c5de753ebd5fce
Since setting the interrupt immediately on async packet completion was causing
issues with Linux guests, I believe this recently fixed Linux bug explains
why this is happening:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=361aabf395e4a23cf554cf4ec0c0c6963b8beb01
Note that we can *not* count on this fix being present in all Linux guests!
I was hoping that the recently added support for Interrupt Threshold Control
would fix the issues with Linux guests, but adding a simple ehci_commit_irq()
call to ehci_async_bh() still caused problems with Linux guests.
The problem is, that when doing ehci_commit_irq() from ehci_async_bh(),
the "old" frindex value is used to calculate usbsts_frindex, and when
the frame-timer then runs possibly very shortly after ehci_async_bh(),
it increases the frame-timer, and thus any interrupts raised from that
frame-timer run, will also get reported to the guest immediately, rather
then being delayed to the next frame-timer run.
Luckily the solution for this is simple, this means that we need to
increase frindex before calling ehci_commit_irq() from ehci_async_bh(),
which in the end boils down to simple calling ehci_frame_timer() instead
of ehci_async_bh() from the bh.
This may seem like it causes a lot of extra work to be done, but this
is not true. Any work done from the frame-timer processing the periodic
schedule is work which then does not need to be done the next time the
frame timer runs, also the frame-timer will re-arm itself at (possibly)
a later time then it was armed for saving a vmexit at that time.
As an additional advantage moving to simply calling the frame-timer also
fixes 2) as the packet completion will set async_stepdown to 0, and the
re-arming of the timer with an async_stepdown of 0 ensures that any
newly queued up packets get seen in a reasonable amount of time.
This improves the speed (MB/s) of a Linux guest reading from a USB mass
storage device by a factor of 1.5 - 1.7 with input pipelining disabled,
and by a factor of 1.8 with input pipelining enabled.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
According to 4.15.1.2 an interrupt must be raised when a short packet
is received. If we don't do this it may take a significant time for
the guest to notice a short trasnfer has completed, since only the last td
will have its IOC flag set, and a short transfer may complete in an earlier
packet.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
This field is used in some places to track the tbytes field of the token, but
in other places the field is used directly, use it directly everywhere for
consistency.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Rather then having a special check to start queuing after the first packet,
and then another check for the other packets in uhci_fill_queue(), simply
check the previous packet beforehand in uhci_fill_queue()
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Packets with an invalid pid, or which were cancelled have
usb_packet_map() called on them on init, but not usb_packet_unmap()
before being freed.
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Commit ad0b5321f1 forgot to add
QEMU_MADV_HUGEPAGE macros for when CONFIG_MADVISE is not defined.
This broke the build for Windows. Fix it.
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
* bonzini/nbd-next: (30 commits)
qmp: add NBD server commands
block: add close notifiers
block: prepare code for adding block notifiers
qemu-sockets: add socket_listen, socket_connect, socket_parse
tests: do not include tools-obj-y Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
qemu-sockets: return InetSocketAddress from inet_parse
qapi: add socket address types
build: add QAPI files to the tools
vnc: drop QERR_VNC_SERVER_FAILED
qemu-sockets: add error propagation to Unix socket functions
qemu-sockets: add error propagation to inet_parse
qemu-sockets: add error propagation to inet_dgram_opts
qemu-sockets: add error propagation to inet_connect_addr
qemu-sockets: include strerror or gai_strerror output in error messages
vnc: add error propagation to vnc_display_open
vnc: reorganize code for reverse mode
vnc: introduce a single label for error returns
vnc: avoid Yoda conditionals
qemu-ga: ask and print error information from qemu-sockets
nbd: ask and print error information from qemu-sockets
...
At the end of migration the machine has started already, and cannot be
destroyed without losing the guest's data. Hence, prelaunch is the
wrong state. Go to the paused state instead. QEMU would reach that
state anyway (after running the guest for the blink of an eye) if the
"stop" command had been received after the start of migration.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
Right now, stop followed by an incoming migration will let the
virtual machine start. cont before an incoming migration instead
will fail.
This is bad because the actual behavior is not predictable; it is
racy with respect to the start of the incoming migration. That's
because incoming migration is blocking, and thus will delay the
processing of stop/cont until the end of the migration.
In addition, there's nothing that really prevents the user from
typing the block device's passwords before incoming migration is
done, so returning the DeviceEncrypted error is also helpful in
the QMP case.
Both things can be fixed by just toggling the autostart variable when
stop/cont are called in INMIGRATE state.
Note that libvirt is currently working around the race by looping
if the MigrationExpected answer is returned. After this patch, the
command will return right away without ever raising an error.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
On sparc targets, info cpus returns this kind of output:
| info cpus
| * CPU #0: pc=0x0000000000424d18pc=0x0000000000424d18npc=0x0000000000424d1c thread_id=19460
pc is printed twice, there is no space between pc, pc and npc.
With this patch, pc is not printed anymore when has_npc is set. In addition
the space is printed before pc/nip/npc/PC instead of after the colon so that
multiple prints are possible. This result on the following kind of input on
sparc targets:
| info cpus
| * CPU #0: pc=0x0000000000424d18 npc=0x0000000000424d1c thread_id=19460
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Blue Swirl <blauwirbel@gmail.com>
Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
Remove an out of date FIXME regarding the saturating arithmetic helpers:
we now do pass a pointer to CPUARMState to these helpers, and since
the AREG0 changes went in there is no difference between helper.c
and op_helper.c and therefore no point in moving the functions.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Implement abs_i32 inline (with movcond) rather than using a helper
function.
Reviewed-by: Aurelien Jarno <aurelien@aurel32.net>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Use the TCG operation to do Neon 64 bit negations rather than calling
a helper routine for it.
Reviewed-by: Aurelien Jarno <aurelien@aurel32.net>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Rework the handling of arguments to ARM semihosting calls so that we
handle a possible failure return from get_user_ual() or put_user_ual().
(This incidentally silences a lot of warnings from clang about
"expression result unused").
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
qemu_dup_flags() currently limits the flags that can be set on the
fcntl() F_SETFL call to those that we currently know can be set with
fcntl() F_SETFL. The problem with this is that it will prevent use
of new flags in the future without a code update.
This patch relaxes the checking and lets fcntl() determine the flags
it can set.
Signed-off-by: Corey Bryant <coreyb@linux.vnet.ibm.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The new options are tested with blkdebug on both the source and the
target.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This function is unlike get_events in that it makes it easy to process
one event at a time. This is useful in the mirroring test cases, where
we want to process just one event (BLOCK_JOB_ERROR) and leave the others
to a helper function.
Acked-by: Luiz Capitulino <lcapitulino@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Error management is important for mirroring; otherwise, an error on the
target (even something as "innocent" as ENOSPC) requires to start again
with a full copy. Similar to on_read_error/on_write_error, two separate
knobs are provided for on_source_error (reads) and on_target_error (writes).
The default is 'report' for both.
The 'ignore' policy will leave the sector dirty, so that it will be
retried later. Thus, it will not cause corruption.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Switching to the target of the migration is done mostly asynchronously,
and reported to management via the BLOCK_JOB_COMPLETED event; the only
synchronous phase is opening the backing files. bdrv_open_backing_file
can always be done, even for migration of the full image (aka sync:
'full'). In this case, qmp_drive_mirror will create the target disk
with no backing file at all, and bdrv_open_backing_file will be a no-op.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This adds the monitor commands that start the mirroring job.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This patch adds the implementation of a new job that mirrors a disk to
a new image while letting the guest continue using the old image.
The target is treated as a "black box" and data is copied from the
source to the target in the background. This can be used for several
purposes, including storage migration, continuous replication, and
observation of the guest I/O in an external program. It is also a
first step in replacing the inefficient block migration code that is
part of QEMU.
The job is possibly never-ending, but it is logically structured into
two phases: 1) copy all data as fast as possible until the target
first gets in sync with the source; 2) keep target in sync and
ensure that reopening to the target gets a correct (full) copy
of the source data.
The second phase is indicated by the progress in "info block-jobs"
reporting the current offset to be equal to the length of the file.
When the job is cancelled in the second phase, QEMU will run the
job until the source is clean and quiescent, then it will report
successful completion of the job.
In other words, the BLOCK_JOB_CANCELLED event means that the target
may _not_ be consistent with a past state of the source; the
BLOCK_JOB_COMPLETED event means that the target is consistent with
a past state of the source. (Note that it could already happen
that management lost the race against QEMU and got a completion
event instead of cancellation).
It is not yet possible to complete the job and switch over to the target
disk. The next patches will fix this and add many refinements to the
basic idea introduced here. These include improved error management,
some tunable knobs and performance optimizations.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Even for jobs that need to be manually completed, management may want
to take care itself of the completion, not requiring the user to issue
a command to terminate the job. In this case we want to avoid that
they poll us continuously, waiting for completion to become available.
Thus, add a new event that signals the phase switch and the availability
of the block-job-complete command.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
While streaming can be dropped as soon as it progressed through the whole
image, mirroring needs to be completed manually for two reasons: 1) so that
management knows exactly when the VM switches to the target; 2) because
for other use cases such as replication, we may leave the operation running
for the whole life of the virtual machine.
Add a new block job command that manually completes background operations.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Assert that write_compressed is never used with the dirty bitmap.
Setting the bits early is wrong, because a coroutine might concurrently
examine them and copy incomplete data from the source.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Mirroring runs without the backing file so that it can be copied outside
QEMU. However, we need to add it at the time the job is completed and
QEMU switches to the target. Factor out the common bits of opening an
image and completing a mirroring operation.
The new function does not assume that the file is closed immediately after
it returns failure, so it keeps the BDRV_O_NO_BACKING flag up-to-date.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qmp_query_blockstat cannot have errors, remove the Error argument and
create a new public function bdrv_query_stats out of it.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This option can be used for passing file descriptors on the
command line. It mirrors the existing add-fd QMP command which
allows an fd to be passed to QEMU via SCM_RIGHTS and added to an
fd set.
This can be combined with commands such as -drive to link file
descriptors in an fd set to a drive:
qemu-kvm -add-fd fd=3,set=2,opaque="rdwr:/path/to/file"
-add-fd fd=4,set=2,opaque="rdonly:/path/to/file"
-drive file=/dev/fdset/2,index=0,media=disk
This example adds dups of fds 3 and 4, and the accompanying opaque
strings to the fd set with ID=2. qemu_open() already knows how
to handle a filename of this format. qemu_open() searches the
corresponding fd set for an fd and when it finds a match, QEMU
goes on to use a dup of that fd just like it would have used an
fd that it opened itself.
Signed-off-by: Corey Bryant <coreyb@linux.vnet.ibm.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean
it up after QEMU initialization is complete.
Signed-off-by: Corey Bryant <coreyb@linux.vnet.ibm.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qmp_add_fd() gets an fd that was received over a socket with
SCM_RIGHTS and adds it to an fd set. This patch adds support
that will enable adding an fd that was inherited on the
command line to an fd set.
Note: All of the code added to monitor_fdset_add_fd(), with the
exception of the error path for non-valid fdset-id, is code motion
from qmp_add_fd().
Signed-off-by: Corey Bryant <coreyb@linux.vnet.ibm.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The first call to add an fd to an fd set was previously not
allowed to choose the fd set ID. The ID was generated as
the first available and ensuing calls could add more fds by
specifying the fd set ID. This change allows users to
choose the fd set ID on the first call.
Signed-off-by: Corey Bryant <coreyb@linux.vnet.ibm.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kashyap Chamarthy <kashyap.cv@gmail.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This new test verifies that qemu-img info --backing-chain safely aborts
when an image file has a backing file infinite loop.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The previous block commit used absolute filenames for all block-commit
images and commands; this adds relative filenames for the same tests.
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This simplifies some code and error checking, and also fixes a bug.
bdrv_find_backing_image() should only be passed absolute filenames,
or filenames relative to the chain. In the QMP message handler for
block commit, when looking up the base do so from the determined top
image, so we know it is reachable from top.
Some of the error messages put out by block-commit have changed
slightly, which causes 2 tests cases for block-commit to fail.
This patch updates the test cases to look for the correct error
output.
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Currently, bdrv_find_backing_image compares bs->backing_file with
what is passed in as a backing_file name. Mismatches may occur,
however, when bs->backing_file and backing_file are not both
absolute or relative.
Use path_combine() to make sure any relative backing filenames are
relative to the current image filename being searched, and then use
realpath() to make all comparisons based on absolute filenames.
If either backing_file or bs->backing_file is determine to be a
protocol, then no filename normalization is performed.
This also changes bdrv_find_backing_image to no longer be recursive,
but iterative.
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This patch allows an empty filename to be passed as the new base image name
for qemu-img rebase to mean base the image on no backing file (i.e.
independent of any backing file). According to Eric Blake, qemu-img rebase
already supports this when '-u' is used; this adds support when -u is not
used.
Signed-off-by: Alex Bligh <alex@alex.org.uk>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
In QEMUMonitorProtocol, commit e9d17b6 removed the __sockfile creation
from __negotiate_capabilities(), which breaks _accept(). This causes
failures in qemu-io python based tests (i.e. tests 030 and 040).
This patch creates the sockfile in __accept() as well.
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Luiz Capitulino <lcapitulino@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Adding an NBD server inside QEMU is trivial, since all the logic is
in nbd.c and can be shared easily between qemu-nbd and QEMU itself.
The main difference is that qemu-nbd serves a single unnamed export,
while QEMU serves named exports.
Acked-by: Luiz Capitulino <lcapitulino@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
