2015-04-28 16:27:52 +03:00
|
|
|
/*
|
|
|
|
* Block layer I/O functions
|
|
|
|
*
|
|
|
|
* Copyright (c) 2003 Fabrice Bellard
|
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
|
|
* of this software and associated documentation files (the "Software"), to deal
|
|
|
|
* in the Software without restriction, including without limitation the rights
|
|
|
|
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
|
|
* copies of the Software, and to permit persons to whom the Software is
|
|
|
|
* furnished to do so, subject to the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice shall be included in
|
|
|
|
* all copies or substantial portions of the Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
|
|
|
|
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
|
|
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
|
|
* THE SOFTWARE.
|
|
|
|
*/
|
|
|
|
|
2016-01-18 21:01:42 +03:00
|
|
|
#include "qemu/osdep.h"
|
2015-04-28 16:27:52 +03:00
|
|
|
#include "trace.h"
|
2015-10-19 18:53:21 +03:00
|
|
|
#include "sysemu/block-backend.h"
|
2018-02-16 19:50:12 +03:00
|
|
|
#include "block/aio-wait.h"
|
2015-04-28 16:27:52 +03:00
|
|
|
#include "block/blockjob.h"
|
2017-05-08 17:13:03 +03:00
|
|
|
#include "block/blockjob_int.h"
|
2015-04-28 16:27:52 +03:00
|
|
|
#include "block/block_int.h"
|
2016-03-20 20:16:19 +03:00
|
|
|
#include "qemu/cutils.h"
|
include/qemu/osdep.h: Don't include qapi/error.h
Commit 57cb38b included qapi/error.h into qemu/osdep.h to get the
Error typedef. Since then, we've moved to include qemu/osdep.h
everywhere. Its file comment explains: "To avoid getting into
possible circular include dependencies, this file should not include
any other QEMU headers, with the exceptions of config-host.h,
compiler.h, os-posix.h and os-win32.h, all of which are doing a
similar job to this file and are under similar constraints."
qapi/error.h doesn't do a similar job, and it doesn't adhere to
similar constraints: it includes qapi-types.h. That's in excess of
100KiB of crap most .c files don't actually need.
Add the typedef to qemu/typedefs.h, and include that instead of
qapi/error.h. Include qapi/error.h in .c files that need it and don't
get it now. Include qapi-types.h in qom/object.h for uint16List.
Update scripts/clean-includes accordingly. Update it further to match
reality: replace config.h by config-target.h, add sysemu/os-posix.h,
sysemu/os-win32.h. Update the list of includes in the qemu/osdep.h
comment quoted above similarly.
This reduces the number of objects depending on qapi/error.h from "all
of them" to less than a third. Unfortunately, the number depending on
qapi-types.h shrinks only a little. More work is needed for that one.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
[Fix compilation without the spice devel packages. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-03-14 11:01:28 +03:00
|
|
|
#include "qapi/error.h"
|
2015-03-17 20:29:20 +03:00
|
|
|
#include "qemu/error-report.h"
|
Include qemu/main-loop.h less
In my "build everything" tree, changing qemu/main-loop.h triggers a
recompile of some 5600 out of 6600 objects (not counting tests and
objects that don't depend on qemu/osdep.h). It includes block/aio.h,
which in turn includes qemu/event_notifier.h, qemu/notify.h,
qemu/processor.h, qemu/qsp.h, qemu/queue.h, qemu/thread-posix.h,
qemu/thread.h, qemu/timer.h, and a few more.
Include qemu/main-loop.h only where it's needed. Touching it now
recompiles only some 1700 objects. For block/aio.h and
qemu/event_notifier.h, these numbers drop from 5600 to 2800. For the
others, they shrink only slightly.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190812052359.30071-21-armbru@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-12 08:23:50 +03:00
|
|
|
#include "qemu/main-loop.h"
|
2019-09-17 14:58:08 +03:00
|
|
|
#include "sysemu/replay.h"
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
#define NOT_DONE 0x7fffffff /* used while emulated sync operation in progress */
|
|
|
|
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
/* Maximum bounce buffer for copy-on-read and write zeroes, in bytes */
|
|
|
|
#define MAX_BOUNCE_BUFFER (32768 << BDRV_SECTOR_BITS)
|
|
|
|
|
2018-07-10 09:31:20 +03:00
|
|
|
static void bdrv_parent_cb_resize(BlockDriverState *bs);
|
2016-06-02 00:10:03 +03:00
|
|
|
static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs,
|
2017-06-09 13:18:08 +03:00
|
|
|
int64_t offset, int bytes, BdrvRequestFlags flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-07-19 12:26:12 +03:00
|
|
|
static void bdrv_parent_drained_begin(BlockDriverState *bs, BdrvChild *ignore,
|
|
|
|
bool ignore_bds_parents)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-11-29 13:25:10 +03:00
|
|
|
BdrvChild *c, *next;
|
2016-03-21 14:56:44 +03:00
|
|
|
|
2017-11-29 13:25:10 +03:00
|
|
|
QLIST_FOREACH_SAFE(c, &bs->parents, next_parent, next) {
|
2018-05-29 18:17:45 +03:00
|
|
|
if (c == ignore || (ignore_bds_parents && c->role->parent_is_bds)) {
|
2017-12-07 15:03:13 +03:00
|
|
|
continue;
|
|
|
|
}
|
2018-06-29 19:01:31 +03:00
|
|
|
bdrv_parent_drained_begin_single(c, false);
|
2016-04-07 19:33:33 +03:00
|
|
|
}
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
static void bdrv_parent_drained_end_single_no_poll(BdrvChild *c,
|
|
|
|
int *drained_end_counter)
|
block: Introduce BdrvChild.parent_quiesce_counter
Commit 5cb2737e925042e6c7cd3fb0b01313950b03cddf laid out why
bdrv_do_drained_end() must decrement the quiesce_counter after
bdrv_drain_invoke(). It did not give a very good reason why it has to
happen after bdrv_parent_drained_end(), instead only claiming symmetry
to bdrv_do_drained_begin().
It turns out that delaying it for so long is wrong.
Situation: We have an active commit job (i.e. a mirror job) from top to
base for the following graph:
filter
|
[file]
|
v
top --[backing]--> base
Now the VM is closed, which results in the job being cancelled and a
bdrv_drain_all() happening pretty much simultaneously.
Beginning the drain means the job is paused once whenever one of its
nodes is quiesced. This is reversed when the drain ends.
With how the code currently is, after base's drain ends (which means
that it will have unpaused the job once), its quiesce_counter remains at
1 while it goes to undrain its parents (bdrv_parent_drained_end()). For
some reason or another, undraining filter causes the job to be kicked
and enter mirror_exit_common(), where it proceeds to invoke
block_job_remove_all_bdrv().
Now base will be detached from the job. Because its quiesce_counter is
still 1, it will unpause the job once more. So in total, undraining
base will unpause the job twice. Eventually, this will lead to the
job's pause_count going negative -- well, it would, were there not an
assertion against this, which crashes qemu.
The general problem is that if in bdrv_parent_drained_end() we undrain
parent A, and then undrain parent B, which then leads to A detaching the
child, bdrv_replace_child_noperm() will undrain A as if we had not done
so yet; that is, one time too many.
It follows that we cannot decrement the quiesce_counter after invoking
bdrv_parent_drained_end().
Unfortunately, decrementing it before bdrv_parent_drained_end() would be
wrong, too. Imagine the above situation in reverse: Undraining A leads
to B detaching the child. If we had already decremented the
quiesce_counter by that point, bdrv_replace_child_noperm() would undrain
B one time too little; because it expects bdrv_parent_drained_end() to
issue this undrain. But bdrv_parent_drained_end() won't do that,
because B is no longer a parent.
Therefore, we have to do something else. This patch opts for
introducing a second quiesce_counter that counts how many times a
child's parent has been quiesced (though c->role->drained_*). With
that, bdrv_replace_child_noperm() just has to undrain the parent exactly
that many times when removing a child, and it will always be right.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:09 +03:00
|
|
|
{
|
|
|
|
assert(c->parent_quiesce_counter > 0);
|
|
|
|
c->parent_quiesce_counter--;
|
|
|
|
if (c->role->drained_end) {
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
c->role->drained_end(c, drained_end_counter);
|
block: Introduce BdrvChild.parent_quiesce_counter
Commit 5cb2737e925042e6c7cd3fb0b01313950b03cddf laid out why
bdrv_do_drained_end() must decrement the quiesce_counter after
bdrv_drain_invoke(). It did not give a very good reason why it has to
happen after bdrv_parent_drained_end(), instead only claiming symmetry
to bdrv_do_drained_begin().
It turns out that delaying it for so long is wrong.
Situation: We have an active commit job (i.e. a mirror job) from top to
base for the following graph:
filter
|
[file]
|
v
top --[backing]--> base
Now the VM is closed, which results in the job being cancelled and a
bdrv_drain_all() happening pretty much simultaneously.
Beginning the drain means the job is paused once whenever one of its
nodes is quiesced. This is reversed when the drain ends.
With how the code currently is, after base's drain ends (which means
that it will have unpaused the job once), its quiesce_counter remains at
1 while it goes to undrain its parents (bdrv_parent_drained_end()). For
some reason or another, undraining filter causes the job to be kicked
and enter mirror_exit_common(), where it proceeds to invoke
block_job_remove_all_bdrv().
Now base will be detached from the job. Because its quiesce_counter is
still 1, it will unpause the job once more. So in total, undraining
base will unpause the job twice. Eventually, this will lead to the
job's pause_count going negative -- well, it would, were there not an
assertion against this, which crashes qemu.
The general problem is that if in bdrv_parent_drained_end() we undrain
parent A, and then undrain parent B, which then leads to A detaching the
child, bdrv_replace_child_noperm() will undrain A as if we had not done
so yet; that is, one time too many.
It follows that we cannot decrement the quiesce_counter after invoking
bdrv_parent_drained_end().
Unfortunately, decrementing it before bdrv_parent_drained_end() would be
wrong, too. Imagine the above situation in reverse: Undraining A leads
to B detaching the child. If we had already decremented the
quiesce_counter by that point, bdrv_replace_child_noperm() would undrain
B one time too little; because it expects bdrv_parent_drained_end() to
issue this undrain. But bdrv_parent_drained_end() won't do that,
because B is no longer a parent.
Therefore, we have to do something else. This patch opts for
introducing a second quiesce_counter that counts how many times a
child's parent has been quiesced (though c->role->drained_*). With
that, bdrv_replace_child_noperm() just has to undrain the parent exactly
that many times when removing a child, and it will always be right.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:09 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
void bdrv_parent_drained_end_single(BdrvChild *c)
|
|
|
|
{
|
|
|
|
int drained_end_counter = 0;
|
|
|
|
bdrv_parent_drained_end_single_no_poll(c, &drained_end_counter);
|
|
|
|
BDRV_POLL_WHILE(c->bs, atomic_read(&drained_end_counter) > 0);
|
|
|
|
}
|
|
|
|
|
2019-07-19 12:26:12 +03:00
|
|
|
static void bdrv_parent_drained_end(BlockDriverState *bs, BdrvChild *ignore,
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
bool ignore_bds_parents,
|
|
|
|
int *drained_end_counter)
|
2016-04-07 19:33:33 +03:00
|
|
|
{
|
2019-07-19 12:26:16 +03:00
|
|
|
BdrvChild *c;
|
2016-03-21 14:56:44 +03:00
|
|
|
|
2019-07-19 12:26:16 +03:00
|
|
|
QLIST_FOREACH(c, &bs->parents, next_parent) {
|
2018-05-29 18:17:45 +03:00
|
|
|
if (c == ignore || (ignore_bds_parents && c->role->parent_is_bds)) {
|
2017-12-07 15:03:13 +03:00
|
|
|
continue;
|
|
|
|
}
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
bdrv_parent_drained_end_single_no_poll(c, drained_end_counter);
|
2016-03-21 14:56:44 +03:00
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2018-06-29 19:01:31 +03:00
|
|
|
static bool bdrv_parent_drained_poll_single(BdrvChild *c)
|
|
|
|
{
|
|
|
|
if (c->role->drained_poll) {
|
|
|
|
return c->role->drained_poll(c);
|
|
|
|
}
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2018-05-29 18:17:45 +03:00
|
|
|
static bool bdrv_parent_drained_poll(BlockDriverState *bs, BdrvChild *ignore,
|
|
|
|
bool ignore_bds_parents)
|
2018-03-22 16:11:20 +03:00
|
|
|
{
|
|
|
|
BdrvChild *c, *next;
|
|
|
|
bool busy = false;
|
|
|
|
|
|
|
|
QLIST_FOREACH_SAFE(c, &bs->parents, next_parent, next) {
|
2018-05-29 18:17:45 +03:00
|
|
|
if (c == ignore || (ignore_bds_parents && c->role->parent_is_bds)) {
|
2018-03-22 16:11:20 +03:00
|
|
|
continue;
|
|
|
|
}
|
2018-06-29 19:01:31 +03:00
|
|
|
busy |= bdrv_parent_drained_poll_single(c);
|
2018-03-22 16:11:20 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
return busy;
|
|
|
|
}
|
|
|
|
|
2018-06-29 19:01:31 +03:00
|
|
|
void bdrv_parent_drained_begin_single(BdrvChild *c, bool poll)
|
|
|
|
{
|
block: Introduce BdrvChild.parent_quiesce_counter
Commit 5cb2737e925042e6c7cd3fb0b01313950b03cddf laid out why
bdrv_do_drained_end() must decrement the quiesce_counter after
bdrv_drain_invoke(). It did not give a very good reason why it has to
happen after bdrv_parent_drained_end(), instead only claiming symmetry
to bdrv_do_drained_begin().
It turns out that delaying it for so long is wrong.
Situation: We have an active commit job (i.e. a mirror job) from top to
base for the following graph:
filter
|
[file]
|
v
top --[backing]--> base
Now the VM is closed, which results in the job being cancelled and a
bdrv_drain_all() happening pretty much simultaneously.
Beginning the drain means the job is paused once whenever one of its
nodes is quiesced. This is reversed when the drain ends.
With how the code currently is, after base's drain ends (which means
that it will have unpaused the job once), its quiesce_counter remains at
1 while it goes to undrain its parents (bdrv_parent_drained_end()). For
some reason or another, undraining filter causes the job to be kicked
and enter mirror_exit_common(), where it proceeds to invoke
block_job_remove_all_bdrv().
Now base will be detached from the job. Because its quiesce_counter is
still 1, it will unpause the job once more. So in total, undraining
base will unpause the job twice. Eventually, this will lead to the
job's pause_count going negative -- well, it would, were there not an
assertion against this, which crashes qemu.
The general problem is that if in bdrv_parent_drained_end() we undrain
parent A, and then undrain parent B, which then leads to A detaching the
child, bdrv_replace_child_noperm() will undrain A as if we had not done
so yet; that is, one time too many.
It follows that we cannot decrement the quiesce_counter after invoking
bdrv_parent_drained_end().
Unfortunately, decrementing it before bdrv_parent_drained_end() would be
wrong, too. Imagine the above situation in reverse: Undraining A leads
to B detaching the child. If we had already decremented the
quiesce_counter by that point, bdrv_replace_child_noperm() would undrain
B one time too little; because it expects bdrv_parent_drained_end() to
issue this undrain. But bdrv_parent_drained_end() won't do that,
because B is no longer a parent.
Therefore, we have to do something else. This patch opts for
introducing a second quiesce_counter that counts how many times a
child's parent has been quiesced (though c->role->drained_*). With
that, bdrv_replace_child_noperm() just has to undrain the parent exactly
that many times when removing a child, and it will always be right.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:09 +03:00
|
|
|
c->parent_quiesce_counter++;
|
2018-06-29 19:01:31 +03:00
|
|
|
if (c->role->drained_begin) {
|
|
|
|
c->role->drained_begin(c);
|
|
|
|
}
|
|
|
|
if (poll) {
|
|
|
|
BDRV_POLL_WHILE(c->bs, bdrv_parent_drained_poll_single(c));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-06-24 01:37:23 +03:00
|
|
|
static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)
|
|
|
|
{
|
|
|
|
dst->opt_transfer = MAX(dst->opt_transfer, src->opt_transfer);
|
|
|
|
dst->max_transfer = MIN_NON_ZERO(dst->max_transfer, src->max_transfer);
|
|
|
|
dst->opt_mem_alignment = MAX(dst->opt_mem_alignment,
|
|
|
|
src->opt_mem_alignment);
|
|
|
|
dst->min_mem_alignment = MAX(dst->min_mem_alignment,
|
|
|
|
src->min_mem_alignment);
|
|
|
|
dst->max_iov = MIN_NON_ZERO(dst->max_iov, src->max_iov);
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
|
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
|
|
|
Error *local_err = NULL;
|
|
|
|
|
|
|
|
memset(&bs->bl, 0, sizeof(bs->bl));
|
|
|
|
|
|
|
|
if (!drv) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2016-06-24 01:37:18 +03:00
|
|
|
/* Default alignment based on whether driver has byte interface */
|
block: Support byte-based aio callbacks
We are gradually moving away from sector-based interfaces, towards
byte-based. Add new sector-based aio callbacks for read and write,
to match the fact that bdrv_aio_pdiscard is already byte-based.
Ideally, drivers should be converted to use coroutine callbacks
rather than aio; but that is not quite as trivial (and if we were
to do that conversion, the null-aio driver would disappear), so for
the short term, converting the signature but keeping things with
aio is easier. However, we CAN declare that a driver that uses
the byte-based aio interfaces now defaults to byte-based
operations, and must explicitly provide a refresh_limits override
to stick with larger alignments (making the alignment issues more
obvious directly in the drivers touched in the next few patches).
Once all drivers are converted, the sector-based aio callbacks will
be removed; in the meantime, a FIXME comment is added due to a
slight inefficiency that will be touched up as part of that later
cleanup.
Simplify some instances of 'bs->drv' into 'drv' while touching this,
since the local variable already exists to reduce typing.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2018-04-24 22:25:01 +03:00
|
|
|
bs->bl.request_alignment = (drv->bdrv_co_preadv ||
|
2019-06-04 19:15:06 +03:00
|
|
|
drv->bdrv_aio_preadv ||
|
|
|
|
drv->bdrv_co_preadv_part) ? 1 : 512;
|
2016-06-24 01:37:18 +03:00
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/* Take some limits from the children as a default */
|
|
|
|
if (bs->file) {
|
2015-06-16 15:19:22 +03:00
|
|
|
bdrv_refresh_limits(bs->file->bs, &local_err);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (local_err) {
|
|
|
|
error_propagate(errp, local_err);
|
|
|
|
return;
|
|
|
|
}
|
2016-06-24 01:37:23 +03:00
|
|
|
bdrv_merge_limits(&bs->bl, &bs->file->bs->bl);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
2015-05-12 17:30:55 +03:00
|
|
|
bs->bl.min_mem_alignment = 512;
|
2019-10-13 05:11:45 +03:00
|
|
|
bs->bl.opt_mem_alignment = qemu_real_host_page_size;
|
2015-07-09 12:56:44 +03:00
|
|
|
|
|
|
|
/* Safe default since most protocols use readv()/writev()/etc */
|
|
|
|
bs->bl.max_iov = IOV_MAX;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2015-06-17 15:55:21 +03:00
|
|
|
if (bs->backing) {
|
|
|
|
bdrv_refresh_limits(bs->backing->bs, &local_err);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (local_err) {
|
|
|
|
error_propagate(errp, local_err);
|
|
|
|
return;
|
|
|
|
}
|
2016-06-24 01:37:23 +03:00
|
|
|
bdrv_merge_limits(&bs->bl, &bs->backing->bs->bl);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Then let the driver override it */
|
|
|
|
if (drv->bdrv_refresh_limits) {
|
|
|
|
drv->bdrv_refresh_limits(bs, errp);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* The copy-on-read flag is actually a reference count so multiple users may
|
|
|
|
* use the feature without worrying about clobbering its previous state.
|
|
|
|
* Copy-on-read stays enabled until all users have called to disable it.
|
|
|
|
*/
|
|
|
|
void bdrv_enable_copy_on_read(BlockDriverState *bs)
|
|
|
|
{
|
2017-06-05 15:38:50 +03:00
|
|
|
atomic_inc(&bs->copy_on_read);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_disable_copy_on_read(BlockDriverState *bs)
|
|
|
|
{
|
2017-06-05 15:38:50 +03:00
|
|
|
int old = atomic_fetch_dec(&bs->copy_on_read);
|
|
|
|
assert(old >= 1);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-06-29 16:27:45 +03:00
|
|
|
typedef struct {
|
|
|
|
Coroutine *co;
|
|
|
|
BlockDriverState *bs;
|
|
|
|
bool done;
|
2017-09-23 14:14:09 +03:00
|
|
|
bool begin;
|
2017-12-06 19:05:44 +03:00
|
|
|
bool recursive;
|
2018-03-23 14:40:41 +03:00
|
|
|
bool poll;
|
2017-12-07 15:03:13 +03:00
|
|
|
BdrvChild *parent;
|
2018-05-29 18:17:45 +03:00
|
|
|
bool ignore_bds_parents;
|
2019-07-19 12:26:11 +03:00
|
|
|
int *drained_end_counter;
|
2017-06-29 16:27:45 +03:00
|
|
|
} BdrvCoDrainData;
|
|
|
|
|
|
|
|
static void coroutine_fn bdrv_drain_invoke_entry(void *opaque)
|
|
|
|
{
|
|
|
|
BdrvCoDrainData *data = opaque;
|
|
|
|
BlockDriverState *bs = data->bs;
|
|
|
|
|
2017-09-23 14:14:09 +03:00
|
|
|
if (data->begin) {
|
2017-09-23 14:14:10 +03:00
|
|
|
bs->drv->bdrv_co_drain_begin(bs);
|
2017-09-23 14:14:09 +03:00
|
|
|
} else {
|
|
|
|
bs->drv->bdrv_co_drain_end(bs);
|
|
|
|
}
|
2017-06-29 16:27:45 +03:00
|
|
|
|
2019-07-22 16:30:53 +03:00
|
|
|
/* Set data->done and decrement drained_end_counter before bdrv_wakeup() */
|
2017-06-29 16:27:45 +03:00
|
|
|
atomic_mb_set(&data->done, true);
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
if (!data->begin) {
|
2019-07-19 12:26:11 +03:00
|
|
|
atomic_dec(data->drained_end_counter);
|
|
|
|
}
|
2019-07-22 16:30:53 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2019-07-19 12:26:11 +03:00
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
g_free(data);
|
2017-06-29 16:27:45 +03:00
|
|
|
}
|
|
|
|
|
2017-12-05 14:52:09 +03:00
|
|
|
/* Recursively call BlockDriver.bdrv_co_drain_begin/end callbacks */
|
2019-07-19 12:26:11 +03:00
|
|
|
static void bdrv_drain_invoke(BlockDriverState *bs, bool begin,
|
|
|
|
int *drained_end_counter)
|
2017-06-29 16:27:45 +03:00
|
|
|
{
|
2018-03-23 22:29:24 +03:00
|
|
|
BdrvCoDrainData *data;
|
2017-06-29 16:27:45 +03:00
|
|
|
|
2017-09-23 14:14:10 +03:00
|
|
|
if (!bs->drv || (begin && !bs->drv->bdrv_co_drain_begin) ||
|
2017-09-23 14:14:09 +03:00
|
|
|
(!begin && !bs->drv->bdrv_co_drain_end)) {
|
2017-06-29 16:27:45 +03:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2018-03-23 22:29:24 +03:00
|
|
|
data = g_new(BdrvCoDrainData, 1);
|
|
|
|
*data = (BdrvCoDrainData) {
|
|
|
|
.bs = bs,
|
|
|
|
.done = false,
|
2019-07-19 12:26:11 +03:00
|
|
|
.begin = begin,
|
|
|
|
.drained_end_counter = drained_end_counter,
|
2018-03-23 22:29:24 +03:00
|
|
|
};
|
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
if (!begin) {
|
2019-07-19 12:26:11 +03:00
|
|
|
atomic_inc(drained_end_counter);
|
|
|
|
}
|
|
|
|
|
2018-03-23 22:29:24 +03:00
|
|
|
/* Make sure the driver callback completes during the polling phase for
|
|
|
|
* drain_begin. */
|
|
|
|
bdrv_inc_in_flight(bs);
|
|
|
|
data->co = qemu_coroutine_create(bdrv_drain_invoke_entry, data);
|
|
|
|
aio_co_schedule(bdrv_get_aio_context(bs), data->co);
|
2017-06-29 16:27:45 +03:00
|
|
|
}
|
|
|
|
|
2018-03-22 12:57:14 +03:00
|
|
|
/* Returns true if BDRV_POLL_WHILE() should go into a blocking aio_poll() */
|
2018-03-23 14:40:41 +03:00
|
|
|
bool bdrv_drain_poll(BlockDriverState *bs, bool recursive,
|
2018-05-29 18:17:45 +03:00
|
|
|
BdrvChild *ignore_parent, bool ignore_bds_parents)
|
2018-03-22 16:11:20 +03:00
|
|
|
{
|
2018-03-23 14:40:41 +03:00
|
|
|
BdrvChild *child, *next;
|
|
|
|
|
2018-05-29 18:17:45 +03:00
|
|
|
if (bdrv_parent_drained_poll(bs, ignore_parent, ignore_bds_parents)) {
|
2018-03-22 16:11:20 +03:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2018-03-23 14:40:41 +03:00
|
|
|
if (atomic_read(&bs->in_flight)) {
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (recursive) {
|
2018-05-29 18:17:45 +03:00
|
|
|
assert(!ignore_bds_parents);
|
2018-03-23 14:40:41 +03:00
|
|
|
QLIST_FOREACH_SAFE(child, &bs->children, next, next) {
|
2018-05-29 18:17:45 +03:00
|
|
|
if (bdrv_drain_poll(child->bs, recursive, child, false)) {
|
2018-03-23 14:40:41 +03:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
2018-03-22 16:11:20 +03:00
|
|
|
}
|
|
|
|
|
2018-03-23 14:40:41 +03:00
|
|
|
static bool bdrv_drain_poll_top_level(BlockDriverState *bs, bool recursive,
|
2018-03-22 16:11:20 +03:00
|
|
|
BdrvChild *ignore_parent)
|
2018-03-22 12:57:14 +03:00
|
|
|
{
|
2018-05-29 18:17:45 +03:00
|
|
|
return bdrv_drain_poll(bs, recursive, ignore_parent, false);
|
2018-03-22 12:57:14 +03:00
|
|
|
}
|
|
|
|
|
2017-12-06 19:05:44 +03:00
|
|
|
static void bdrv_do_drained_begin(BlockDriverState *bs, bool recursive,
|
2018-05-29 18:17:45 +03:00
|
|
|
BdrvChild *parent, bool ignore_bds_parents,
|
|
|
|
bool poll);
|
2017-12-06 19:05:44 +03:00
|
|
|
static void bdrv_do_drained_end(BlockDriverState *bs, bool recursive,
|
2019-07-19 12:26:11 +03:00
|
|
|
BdrvChild *parent, bool ignore_bds_parents,
|
|
|
|
int *drained_end_counter);
|
2017-12-07 15:03:13 +03:00
|
|
|
|
2016-04-05 14:20:52 +03:00
|
|
|
static void bdrv_co_drain_bh_cb(void *opaque)
|
|
|
|
{
|
|
|
|
BdrvCoDrainData *data = opaque;
|
|
|
|
Coroutine *co = data->co;
|
2016-10-27 13:48:52 +03:00
|
|
|
BlockDriverState *bs = data->bs;
|
2016-04-05 14:20:52 +03:00
|
|
|
|
2018-04-10 17:07:55 +03:00
|
|
|
if (bs) {
|
2018-08-17 19:54:18 +03:00
|
|
|
AioContext *ctx = bdrv_get_aio_context(bs);
|
|
|
|
AioContext *co_ctx = qemu_coroutine_get_aio_context(co);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* When the coroutine yielded, the lock for its home context was
|
|
|
|
* released, so we need to re-acquire it here. If it explicitly
|
|
|
|
* acquired a different context, the lock is still held and we don't
|
|
|
|
* want to lock it a second time (or AIO_WAIT_WHILE() would hang).
|
|
|
|
*/
|
|
|
|
if (ctx == co_ctx) {
|
|
|
|
aio_context_acquire(ctx);
|
|
|
|
}
|
2018-04-10 17:07:55 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
|
|
|
if (data->begin) {
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
assert(!data->drained_end_counter);
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin(bs, data->recursive, data->parent,
|
|
|
|
data->ignore_bds_parents, data->poll);
|
2018-04-10 17:07:55 +03:00
|
|
|
} else {
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
assert(!data->poll);
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_end(bs, data->recursive, data->parent,
|
2019-07-19 12:26:11 +03:00
|
|
|
data->ignore_bds_parents,
|
|
|
|
data->drained_end_counter);
|
2018-04-10 17:07:55 +03:00
|
|
|
}
|
2018-08-17 19:54:18 +03:00
|
|
|
if (ctx == co_ctx) {
|
|
|
|
aio_context_release(ctx);
|
|
|
|
}
|
2017-09-23 14:14:09 +03:00
|
|
|
} else {
|
2018-04-10 17:07:55 +03:00
|
|
|
assert(data->begin);
|
|
|
|
bdrv_drain_all_begin();
|
2017-09-23 14:14:09 +03:00
|
|
|
}
|
|
|
|
|
2016-04-05 14:20:52 +03:00
|
|
|
data->done = true;
|
2017-02-13 16:52:31 +03:00
|
|
|
aio_co_wake(co);
|
2016-04-05 14:20:52 +03:00
|
|
|
}
|
|
|
|
|
2017-09-23 14:14:09 +03:00
|
|
|
static void coroutine_fn bdrv_co_yield_to_drain(BlockDriverState *bs,
|
2017-12-06 19:05:44 +03:00
|
|
|
bool begin, bool recursive,
|
2018-05-29 18:17:45 +03:00
|
|
|
BdrvChild *parent,
|
|
|
|
bool ignore_bds_parents,
|
2019-07-19 12:26:11 +03:00
|
|
|
bool poll,
|
|
|
|
int *drained_end_counter)
|
2016-04-05 14:20:52 +03:00
|
|
|
{
|
|
|
|
BdrvCoDrainData data;
|
|
|
|
|
|
|
|
/* Calling bdrv_drain() from a BH ensures the current coroutine yields and
|
2018-03-22 18:28:33 +03:00
|
|
|
* other coroutines run if they were queued by aio_co_enter(). */
|
2016-04-05 14:20:52 +03:00
|
|
|
|
|
|
|
assert(qemu_in_coroutine());
|
|
|
|
data = (BdrvCoDrainData) {
|
|
|
|
.co = qemu_coroutine_self(),
|
|
|
|
.bs = bs,
|
|
|
|
.done = false,
|
2017-09-23 14:14:09 +03:00
|
|
|
.begin = begin,
|
2017-12-06 19:05:44 +03:00
|
|
|
.recursive = recursive,
|
2017-12-07 15:03:13 +03:00
|
|
|
.parent = parent,
|
2018-05-29 18:17:45 +03:00
|
|
|
.ignore_bds_parents = ignore_bds_parents,
|
2018-03-23 14:40:41 +03:00
|
|
|
.poll = poll,
|
2019-07-19 12:26:11 +03:00
|
|
|
.drained_end_counter = drained_end_counter,
|
2016-04-05 14:20:52 +03:00
|
|
|
};
|
2019-07-19 12:26:11 +03:00
|
|
|
|
2018-04-10 17:07:55 +03:00
|
|
|
if (bs) {
|
|
|
|
bdrv_inc_in_flight(bs);
|
|
|
|
}
|
2019-09-17 14:58:19 +03:00
|
|
|
replay_bh_schedule_oneshot_event(bdrv_get_aio_context(bs),
|
|
|
|
bdrv_co_drain_bh_cb, &data);
|
2016-04-05 14:20:52 +03:00
|
|
|
|
|
|
|
qemu_coroutine_yield();
|
|
|
|
/* If we are resumed from some other event (such as an aio completion or a
|
|
|
|
* timer callback), it is a bug in the caller that should be fixed. */
|
|
|
|
assert(data.done);
|
|
|
|
}
|
|
|
|
|
2018-03-23 17:57:20 +03:00
|
|
|
void bdrv_do_drained_begin_quiesce(BlockDriverState *bs,
|
2018-05-29 18:17:45 +03:00
|
|
|
BdrvChild *parent, bool ignore_bds_parents)
|
2016-05-23 17:08:55 +03:00
|
|
|
{
|
2018-03-23 17:57:20 +03:00
|
|
|
assert(!qemu_in_coroutine());
|
2016-10-27 13:48:53 +03:00
|
|
|
|
2017-12-06 13:00:59 +03:00
|
|
|
/* Stop things in parent-to-child order */
|
2017-06-05 15:38:51 +03:00
|
|
|
if (atomic_fetch_inc(&bs->quiesce_counter) == 0) {
|
2016-05-23 17:08:55 +03:00
|
|
|
aio_disable_external(bdrv_get_aio_context(bs));
|
|
|
|
}
|
|
|
|
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_parent_drained_begin(bs, parent, ignore_bds_parents);
|
2019-07-19 12:26:11 +03:00
|
|
|
bdrv_drain_invoke(bs, true, NULL);
|
2018-03-23 17:57:20 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
static void bdrv_do_drained_begin(BlockDriverState *bs, bool recursive,
|
2018-05-29 18:17:45 +03:00
|
|
|
BdrvChild *parent, bool ignore_bds_parents,
|
|
|
|
bool poll)
|
2018-03-23 17:57:20 +03:00
|
|
|
{
|
|
|
|
BdrvChild *child, *next;
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_co_yield_to_drain(bs, true, recursive, parent, ignore_bds_parents,
|
2019-07-19 12:26:11 +03:00
|
|
|
poll, NULL);
|
2018-03-23 17:57:20 +03:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin_quiesce(bs, parent, ignore_bds_parents);
|
2018-03-22 16:35:58 +03:00
|
|
|
|
2017-12-06 19:05:44 +03:00
|
|
|
if (recursive) {
|
2018-05-29 18:17:45 +03:00
|
|
|
assert(!ignore_bds_parents);
|
2017-12-18 18:05:48 +03:00
|
|
|
bs->recursive_quiesce_counter++;
|
2017-12-06 19:05:44 +03:00
|
|
|
QLIST_FOREACH_SAFE(child, &bs->children, next, next) {
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin(child->bs, true, child, ignore_bds_parents,
|
|
|
|
false);
|
2017-12-06 19:05:44 +03:00
|
|
|
}
|
|
|
|
}
|
2018-03-23 14:40:41 +03:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Wait for drained requests to finish.
|
|
|
|
*
|
|
|
|
* Calling BDRV_POLL_WHILE() only once for the top-level node is okay: The
|
|
|
|
* call is needed so things in this AioContext can make progress even
|
|
|
|
* though we don't return to the main AioContext loop - this automatically
|
|
|
|
* includes other nodes in the same AioContext and therefore all child
|
|
|
|
* nodes.
|
|
|
|
*/
|
|
|
|
if (poll) {
|
2018-05-29 18:17:45 +03:00
|
|
|
assert(!ignore_bds_parents);
|
2018-03-23 14:40:41 +03:00
|
|
|
BDRV_POLL_WHILE(bs, bdrv_drain_poll_top_level(bs, recursive, parent));
|
|
|
|
}
|
2016-05-23 17:08:55 +03:00
|
|
|
}
|
|
|
|
|
2017-12-07 15:03:13 +03:00
|
|
|
void bdrv_drained_begin(BlockDriverState *bs)
|
|
|
|
{
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin(bs, false, NULL, false, true);
|
2017-12-06 19:05:44 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_subtree_drained_begin(BlockDriverState *bs)
|
|
|
|
{
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin(bs, true, NULL, false, true);
|
2017-12-07 15:03:13 +03:00
|
|
|
}
|
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
/**
|
|
|
|
* This function does not poll, nor must any of its recursively called
|
|
|
|
* functions. The *drained_end_counter pointee will be incremented
|
|
|
|
* once for every background operation scheduled, and decremented once
|
|
|
|
* the operation settles. Therefore, the pointer must remain valid
|
|
|
|
* until the pointee reaches 0. That implies that whoever sets up the
|
|
|
|
* pointee has to poll until it is 0.
|
|
|
|
*
|
|
|
|
* We use atomic operations to access *drained_end_counter, because
|
|
|
|
* (1) when called from bdrv_set_aio_context_ignore(), the subgraph of
|
|
|
|
* @bs may contain nodes in different AioContexts,
|
|
|
|
* (2) bdrv_drain_all_end() uses the same counter for all nodes,
|
|
|
|
* regardless of which AioContext they are in.
|
|
|
|
*/
|
2018-05-29 18:17:45 +03:00
|
|
|
static void bdrv_do_drained_end(BlockDriverState *bs, bool recursive,
|
2019-07-19 12:26:11 +03:00
|
|
|
BdrvChild *parent, bool ignore_bds_parents,
|
|
|
|
int *drained_end_counter)
|
2016-05-23 17:08:55 +03:00
|
|
|
{
|
2019-07-19 12:26:16 +03:00
|
|
|
BdrvChild *child;
|
2017-12-13 20:14:18 +03:00
|
|
|
int old_quiesce_counter;
|
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
assert(drained_end_counter != NULL);
|
|
|
|
|
2017-09-23 14:14:09 +03:00
|
|
|
if (qemu_in_coroutine()) {
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_co_yield_to_drain(bs, false, recursive, parent, ignore_bds_parents,
|
2019-07-19 12:26:11 +03:00
|
|
|
false, drained_end_counter);
|
2017-09-23 14:14:09 +03:00
|
|
|
return;
|
|
|
|
}
|
2016-05-23 17:08:55 +03:00
|
|
|
assert(bs->quiesce_counter > 0);
|
|
|
|
|
2017-12-06 13:00:59 +03:00
|
|
|
/* Re-enable things in child-to-parent order */
|
2019-07-19 12:26:11 +03:00
|
|
|
bdrv_drain_invoke(bs, false, drained_end_counter);
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
bdrv_parent_drained_end(bs, parent, ignore_bds_parents,
|
|
|
|
drained_end_counter);
|
block/io: Delay decrementing the quiesce_counter
When ending a drained section, bdrv_do_drained_end() currently first
decrements the quiesce_counter, and only then actually ends the drain.
The bdrv_drain_invoke(bs, false) call may cause graph changes. Say the
graph change involves replacing an existing BB's ("blk") BDS
(blk_bs(blk)) by @bs. Let us introducing the following values:
- bs_oqc = old_quiesce_counter
(so bs->quiesce_counter == bs_oqc - 1)
- obs_qc = blk_bs(blk)->quiesce_counter (before bdrv_drain_invoke())
Let us assume there is no blk_pread_unthrottled() involved, so
blk->quiesce_counter == obs_qc (before bdrv_drain_invoke()).
Now replacing blk_bs(blk) by @bs will reduce blk->quiesce_counter by
obs_qc (making it 0) and increase it by bs_oqc-1 (making it bs_oqc-1).
bdrv_drain_invoke() returns and we invoke bdrv_parent_drained_end().
This will decrement blk->quiesce_counter by one, so it would be -1 --
were there not an assertion against that in blk_root_drained_end().
We therefore have to keep the quiesce_counter up at least until
bdrv_drain_invoke() returns, so that bdrv_parent_drained_end() does the
right thing for the parents @bs got during bdrv_drain_invoke().
But let us delay it even further, namely until bdrv_parent_drained_end()
returns, because then it mirrors bdrv_do_drained_begin(): There, we
first increment the quiesce_counter, then begin draining the parents,
and then call bdrv_drain_invoke(). It makes sense to let
bdrv_do_drained_end() unravel this exactly in reverse.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-05-22 17:40:36 +03:00
|
|
|
|
|
|
|
old_quiesce_counter = atomic_fetch_dec(&bs->quiesce_counter);
|
2017-12-13 20:14:18 +03:00
|
|
|
if (old_quiesce_counter == 1) {
|
|
|
|
aio_enable_external(bdrv_get_aio_context(bs));
|
|
|
|
}
|
2017-12-06 19:05:44 +03:00
|
|
|
|
|
|
|
if (recursive) {
|
2018-05-29 18:17:45 +03:00
|
|
|
assert(!ignore_bds_parents);
|
2017-12-18 18:05:48 +03:00
|
|
|
bs->recursive_quiesce_counter--;
|
2019-07-19 12:26:16 +03:00
|
|
|
QLIST_FOREACH(child, &bs->children, next) {
|
2019-07-19 12:26:11 +03:00
|
|
|
bdrv_do_drained_end(child->bs, true, child, ignore_bds_parents,
|
|
|
|
drained_end_counter);
|
2017-12-06 19:05:44 +03:00
|
|
|
}
|
|
|
|
}
|
2016-05-23 17:08:55 +03:00
|
|
|
}
|
|
|
|
|
2017-12-07 15:03:13 +03:00
|
|
|
void bdrv_drained_end(BlockDriverState *bs)
|
|
|
|
{
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
int drained_end_counter = 0;
|
|
|
|
bdrv_do_drained_end(bs, false, NULL, false, &drained_end_counter);
|
|
|
|
BDRV_POLL_WHILE(bs, atomic_read(&drained_end_counter) > 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_drained_end_no_poll(BlockDriverState *bs, int *drained_end_counter)
|
|
|
|
{
|
|
|
|
bdrv_do_drained_end(bs, false, NULL, false, drained_end_counter);
|
2017-12-06 19:05:44 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_subtree_drained_end(BlockDriverState *bs)
|
|
|
|
{
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
int drained_end_counter = 0;
|
|
|
|
bdrv_do_drained_end(bs, true, NULL, false, &drained_end_counter);
|
|
|
|
BDRV_POLL_WHILE(bs, atomic_read(&drained_end_counter) > 0);
|
2017-12-07 15:03:13 +03:00
|
|
|
}
|
|
|
|
|
2017-12-18 18:05:48 +03:00
|
|
|
void bdrv_apply_subtree_drain(BdrvChild *child, BlockDriverState *new_parent)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < new_parent->recursive_quiesce_counter; i++) {
|
2018-05-29 18:17:45 +03:00
|
|
|
bdrv_do_drained_begin(child->bs, true, child, false, true);
|
2017-12-18 18:05:48 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_unapply_subtree_drain(BdrvChild *child, BlockDriverState *old_parent)
|
|
|
|
{
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
int drained_end_counter = 0;
|
2017-12-18 18:05:48 +03:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < old_parent->recursive_quiesce_counter; i++) {
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
bdrv_do_drained_end(child->bs, true, child, false,
|
|
|
|
&drained_end_counter);
|
2017-12-18 18:05:48 +03:00
|
|
|
}
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
|
|
|
|
BDRV_POLL_WHILE(child->bs, atomic_read(&drained_end_counter) > 0);
|
2017-12-18 18:05:48 +03:00
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
2015-11-09 13:16:53 +03:00
|
|
|
* Wait for pending requests to complete on a single BlockDriverState subtree,
|
|
|
|
* and suspend block driver's internal I/O until next request arrives.
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
|
|
|
* Note that unlike bdrv_drain_all(), the caller must hold the BlockDriverState
|
|
|
|
* AioContext.
|
|
|
|
*/
|
2016-04-07 19:33:32 +03:00
|
|
|
void coroutine_fn bdrv_co_drain(BlockDriverState *bs)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2016-05-23 17:08:55 +03:00
|
|
|
assert(qemu_in_coroutine());
|
|
|
|
bdrv_drained_begin(bs);
|
|
|
|
bdrv_drained_end(bs);
|
2016-04-07 19:33:32 +03:00
|
|
|
}
|
2015-06-10 14:38:17 +03:00
|
|
|
|
2016-04-07 19:33:32 +03:00
|
|
|
void bdrv_drain(BlockDriverState *bs)
|
|
|
|
{
|
2016-05-23 17:08:55 +03:00
|
|
|
bdrv_drained_begin(bs);
|
|
|
|
bdrv_drained_end(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-12-14 13:25:16 +03:00
|
|
|
static void bdrv_drain_assert_idle(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
BdrvChild *child, *next;
|
|
|
|
|
|
|
|
assert(atomic_read(&bs->in_flight) == 0);
|
|
|
|
QLIST_FOREACH_SAFE(child, &bs->children, next, next) {
|
|
|
|
bdrv_drain_assert_idle(child->bs);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
unsigned int bdrv_drain_all_count = 0;
|
|
|
|
|
|
|
|
static bool bdrv_drain_all_poll(void)
|
|
|
|
{
|
|
|
|
BlockDriverState *bs = NULL;
|
|
|
|
bool result = false;
|
|
|
|
|
|
|
|
/* bdrv_drain_poll() can't make changes to the graph and we are holding the
|
|
|
|
* main AioContext lock, so iterating bdrv_next_all_states() is safe. */
|
|
|
|
while ((bs = bdrv_next_all_states(bs))) {
|
|
|
|
AioContext *aio_context = bdrv_get_aio_context(bs);
|
|
|
|
aio_context_acquire(aio_context);
|
|
|
|
result |= bdrv_drain_poll(bs, false, NULL, true);
|
|
|
|
aio_context_release(aio_context);
|
|
|
|
}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
|
|
|
* Wait for pending requests to complete across all BlockDriverStates
|
|
|
|
*
|
|
|
|
* This function does not flush data to disk, use bdrv_flush_all() for that
|
|
|
|
* after calling this function.
|
2016-10-28 10:08:02 +03:00
|
|
|
*
|
|
|
|
* This pauses all block jobs and disables external clients. It must
|
|
|
|
* be paired with bdrv_drain_all_end().
|
|
|
|
*
|
|
|
|
* NOTE: no new block jobs or BlockDriverStates can be created between
|
|
|
|
* the bdrv_drain_all_begin() and bdrv_drain_all_end() calls.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2016-10-28 10:08:02 +03:00
|
|
|
void bdrv_drain_all_begin(void)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2018-03-28 19:29:18 +03:00
|
|
|
BlockDriverState *bs = NULL;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2018-04-10 17:07:55 +03:00
|
|
|
if (qemu_in_coroutine()) {
|
2019-07-19 12:26:11 +03:00
|
|
|
bdrv_co_yield_to_drain(NULL, true, false, NULL, true, true, NULL);
|
2018-04-10 17:07:55 +03:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2019-09-17 14:58:08 +03:00
|
|
|
/*
|
|
|
|
* bdrv queue is managed by record/replay,
|
|
|
|
* waiting for finishing the I/O requests may
|
|
|
|
* be infinite
|
|
|
|
*/
|
|
|
|
if (replay_events_enabled()) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
/* AIO_WAIT_WHILE() with a NULL context can only be called from the main
|
|
|
|
* loop AioContext, so make sure we're in the main context. */
|
2017-12-15 11:33:21 +03:00
|
|
|
assert(qemu_get_current_aio_context() == qemu_get_aio_context());
|
2018-03-28 19:29:18 +03:00
|
|
|
assert(bdrv_drain_all_count < INT_MAX);
|
|
|
|
bdrv_drain_all_count++;
|
2017-12-15 11:33:21 +03:00
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
/* Quiesce all nodes, without polling in-flight requests yet. The graph
|
|
|
|
* cannot change during this loop. */
|
|
|
|
while ((bs = bdrv_next_all_states(bs))) {
|
2015-04-28 16:27:52 +03:00
|
|
|
AioContext *aio_context = bdrv_get_aio_context(bs);
|
|
|
|
|
|
|
|
aio_context_acquire(aio_context);
|
2018-03-28 19:29:18 +03:00
|
|
|
bdrv_do_drained_begin(bs, false, NULL, true, false);
|
2015-04-28 16:27:52 +03:00
|
|
|
aio_context_release(aio_context);
|
|
|
|
}
|
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
/* Now poll the in-flight requests */
|
2018-09-18 18:09:16 +03:00
|
|
|
AIO_WAIT_WHILE(NULL, bdrv_drain_all_poll());
|
2018-03-28 19:29:18 +03:00
|
|
|
|
|
|
|
while ((bs = bdrv_next_all_states(bs))) {
|
2017-12-14 13:25:16 +03:00
|
|
|
bdrv_drain_assert_idle(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2016-10-28 10:08:02 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_drain_all_end(void)
|
|
|
|
{
|
2018-03-28 19:29:18 +03:00
|
|
|
BlockDriverState *bs = NULL;
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
int drained_end_counter = 0;
|
2016-10-28 10:08:02 +03:00
|
|
|
|
2019-09-17 14:58:08 +03:00
|
|
|
/*
|
|
|
|
* bdrv queue is managed by record/replay,
|
|
|
|
* waiting for finishing the I/O requests may
|
|
|
|
* be endless
|
|
|
|
*/
|
|
|
|
if (replay_events_enabled()) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
while ((bs = bdrv_next_all_states(bs))) {
|
2015-04-28 16:27:52 +03:00
|
|
|
AioContext *aio_context = bdrv_get_aio_context(bs);
|
|
|
|
|
|
|
|
aio_context_acquire(aio_context);
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
bdrv_do_drained_end(bs, false, NULL, true, &drained_end_counter);
|
2015-04-28 16:27:52 +03:00
|
|
|
aio_context_release(aio_context);
|
|
|
|
}
|
2018-03-28 19:29:18 +03:00
|
|
|
|
block: Do not poll in bdrv_do_drained_end()
We should never poll anywhere in bdrv_do_drained_end() (including its
recursive callees like bdrv_drain_invoke()), because it does not cope
well with graph changes. In fact, it has been written based on the
postulation that no graph changes will happen in it.
Instead, the callers that want to poll must poll, i.e. all currently
globally available wrappers: bdrv_drained_end(),
bdrv_subtree_drained_end(), bdrv_unapply_subtree_drain(), and
bdrv_drain_all_end(). Graph changes there do not matter.
They can poll simply by passing a pointer to a drained_end_counter and
wait until it reaches 0.
This patch also adds a non-polling global wrapper for
bdrv_do_drained_end() that takes a drained_end_counter pointer. We need
such a variant because now no function called anywhere from
bdrv_do_drained_end() must poll. This includes
BdrvChildRole.drained_end(), which already must not poll according to
its interface documentation, but bdrv_child_cb_drained_end() just
violates that by invoking bdrv_drained_end() (which does poll).
Therefore, BdrvChildRole.drained_end() must take a *drained_end_counter
parameter, which bdrv_child_cb_drained_end() can pass on to the new
bdrv_drained_end_no_poll() function.
Note that we now have a pattern of all drained_end-related functions
either polling or receiving a *drained_end_counter to let the caller
poll based on that.
A problem with a single poll loop is that when the drained section in
bdrv_set_aio_context_ignore() ends, some nodes in the subgraph may be in
the old contexts, while others are in the new context already. To let
the collective poll in bdrv_drained_end() work correctly, we must not
hold a lock to the old context, so that the old context can make
progress in case it is different from the current context.
(In the process, remove the comment saying that the current context is
always the old context, because it is wrong.)
In all other places, all nodes in a subtree must be in the same context,
so we can just poll that. The exception of course is
bdrv_drain_all_end(), but that always runs in the main context, so we
can just poll NULL (like bdrv_drain_all_begin() does).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-07-19 12:26:14 +03:00
|
|
|
assert(qemu_get_current_aio_context() == qemu_get_aio_context());
|
|
|
|
AIO_WAIT_WHILE(NULL, atomic_read(&drained_end_counter) > 0);
|
|
|
|
|
2018-03-28 19:29:18 +03:00
|
|
|
assert(bdrv_drain_all_count > 0);
|
|
|
|
bdrv_drain_all_count--;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-10-28 10:08:02 +03:00
|
|
|
void bdrv_drain_all(void)
|
|
|
|
{
|
|
|
|
bdrv_drain_all_begin();
|
|
|
|
bdrv_drain_all_end();
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/**
|
|
|
|
* Remove an active request from the tracked requests list
|
|
|
|
*
|
|
|
|
* This function should be called when a tracked request is completing.
|
|
|
|
*/
|
|
|
|
static void tracked_request_end(BdrvTrackedRequest *req)
|
|
|
|
{
|
|
|
|
if (req->serialising) {
|
2017-06-05 15:38:53 +03:00
|
|
|
atomic_dec(&req->bs->serialising_in_flight);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_lock(&req->bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
QLIST_REMOVE(req, list);
|
|
|
|
qemu_co_queue_restart_all(&req->wait_queue);
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_unlock(&req->bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Add an active request to the tracked requests list
|
|
|
|
*/
|
|
|
|
static void tracked_request_begin(BdrvTrackedRequest *req,
|
|
|
|
BlockDriverState *bs,
|
|
|
|
int64_t offset,
|
2018-07-10 09:31:18 +03:00
|
|
|
uint64_t bytes,
|
2015-11-09 13:16:46 +03:00
|
|
|
enum BdrvTrackedRequestType type)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2018-07-10 09:31:18 +03:00
|
|
|
assert(bytes <= INT64_MAX && offset <= INT64_MAX - bytes);
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
*req = (BdrvTrackedRequest){
|
|
|
|
.bs = bs,
|
|
|
|
.offset = offset,
|
|
|
|
.bytes = bytes,
|
2015-11-09 13:16:46 +03:00
|
|
|
.type = type,
|
2015-04-28 16:27:52 +03:00
|
|
|
.co = qemu_coroutine_self(),
|
|
|
|
.serialising = false,
|
|
|
|
.overlap_offset = offset,
|
|
|
|
.overlap_bytes = bytes,
|
|
|
|
};
|
|
|
|
|
|
|
|
qemu_co_queue_init(&req->wait_queue);
|
|
|
|
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_lock(&bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
QLIST_INSERT_HEAD(&bs->tracked_requests, req, list);
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_unlock(&bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2020-01-08 17:55:56 +03:00
|
|
|
static bool tracked_request_overlaps(BdrvTrackedRequest *req,
|
|
|
|
int64_t offset, uint64_t bytes)
|
|
|
|
{
|
|
|
|
/* aaaa bbbb */
|
|
|
|
if (offset >= req->overlap_offset + req->overlap_bytes) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
/* bbbb aaaa */
|
|
|
|
if (req->overlap_offset >= offset + bytes) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool coroutine_fn
|
|
|
|
bdrv_wait_serialising_requests_locked(BlockDriverState *bs,
|
|
|
|
BdrvTrackedRequest *self)
|
|
|
|
{
|
|
|
|
BdrvTrackedRequest *req;
|
|
|
|
bool retry;
|
|
|
|
bool waited = false;
|
|
|
|
|
|
|
|
do {
|
|
|
|
retry = false;
|
|
|
|
QLIST_FOREACH(req, &bs->tracked_requests, list) {
|
|
|
|
if (req == self || (!req->serialising && !self->serialising)) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
if (tracked_request_overlaps(req, self->overlap_offset,
|
|
|
|
self->overlap_bytes))
|
|
|
|
{
|
|
|
|
/* Hitting this means there was a reentrant request, for
|
|
|
|
* example, a block driver issuing nested requests. This must
|
|
|
|
* never happen since it means deadlock.
|
|
|
|
*/
|
|
|
|
assert(qemu_coroutine_self() != req->co);
|
|
|
|
|
|
|
|
/* If the request is already (indirectly) waiting for us, or
|
|
|
|
* will wait for us as soon as it wakes up, then just go on
|
|
|
|
* (instead of producing a deadlock in the former case). */
|
|
|
|
if (!req->waiting_for) {
|
|
|
|
self->waiting_for = req;
|
|
|
|
qemu_co_queue_wait(&req->wait_queue, &bs->reqs_lock);
|
|
|
|
self->waiting_for = NULL;
|
|
|
|
retry = true;
|
|
|
|
waited = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
} while (retry);
|
|
|
|
return waited;
|
|
|
|
}
|
|
|
|
|
2020-01-08 17:55:55 +03:00
|
|
|
bool bdrv_mark_request_serialising(BdrvTrackedRequest *req, uint64_t align)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2020-01-08 17:55:56 +03:00
|
|
|
BlockDriverState *bs = req->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
int64_t overlap_offset = req->offset & ~(align - 1);
|
2018-07-10 09:31:18 +03:00
|
|
|
uint64_t overlap_bytes = ROUND_UP(req->offset + req->bytes, align)
|
2015-04-28 16:27:52 +03:00
|
|
|
- overlap_offset;
|
2020-01-08 17:55:56 +03:00
|
|
|
bool waited;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2020-01-08 17:55:56 +03:00
|
|
|
qemu_co_mutex_lock(&bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (!req->serialising) {
|
2017-06-05 15:38:53 +03:00
|
|
|
atomic_inc(&req->bs->serialising_in_flight);
|
2015-04-28 16:27:52 +03:00
|
|
|
req->serialising = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
req->overlap_offset = MIN(req->overlap_offset, overlap_offset);
|
|
|
|
req->overlap_bytes = MAX(req->overlap_bytes, overlap_bytes);
|
2020-01-08 17:55:56 +03:00
|
|
|
waited = bdrv_wait_serialising_requests_locked(bs, req);
|
|
|
|
qemu_co_mutex_unlock(&bs->reqs_lock);
|
|
|
|
return waited;
|
2018-07-09 19:37:18 +03:00
|
|
|
}
|
|
|
|
|
2019-11-01 18:25:09 +03:00
|
|
|
/**
|
|
|
|
* Return the tracked request on @bs for the current coroutine, or
|
|
|
|
* NULL if there is none.
|
|
|
|
*/
|
|
|
|
BdrvTrackedRequest *coroutine_fn bdrv_co_get_self_request(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
BdrvTrackedRequest *req;
|
|
|
|
Coroutine *self = qemu_coroutine_self();
|
|
|
|
|
|
|
|
QLIST_FOREACH(req, &bs->tracked_requests, list) {
|
|
|
|
if (req->co == self) {
|
|
|
|
return req;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2016-06-02 12:41:52 +03:00
|
|
|
/**
|
|
|
|
* Round a region to cluster boundaries
|
|
|
|
*/
|
|
|
|
void bdrv_round_to_clusters(BlockDriverState *bs,
|
2017-10-12 06:46:59 +03:00
|
|
|
int64_t offset, int64_t bytes,
|
2016-06-02 12:41:52 +03:00
|
|
|
int64_t *cluster_offset,
|
2017-10-12 06:46:59 +03:00
|
|
|
int64_t *cluster_bytes)
|
2016-06-02 12:41:52 +03:00
|
|
|
{
|
|
|
|
BlockDriverInfo bdi;
|
|
|
|
|
|
|
|
if (bdrv_get_info(bs, &bdi) < 0 || bdi.cluster_size == 0) {
|
|
|
|
*cluster_offset = offset;
|
|
|
|
*cluster_bytes = bytes;
|
|
|
|
} else {
|
|
|
|
int64_t c = bdi.cluster_size;
|
|
|
|
*cluster_offset = QEMU_ALIGN_DOWN(offset, c);
|
|
|
|
*cluster_bytes = QEMU_ALIGN_UP(offset - *cluster_offset + bytes, c);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
static int bdrv_get_cluster_size(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
BlockDriverInfo bdi;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = bdrv_get_info(bs, &bdi);
|
|
|
|
if (ret < 0 || bdi.cluster_size == 0) {
|
2016-06-24 01:37:24 +03:00
|
|
|
return bs->bl.request_alignment;
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
|
|
|
return bdi.cluster_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
void bdrv_inc_in_flight(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
atomic_inc(&bs->in_flight);
|
|
|
|
}
|
|
|
|
|
2016-10-27 13:49:05 +03:00
|
|
|
void bdrv_wakeup(BlockDriverState *bs)
|
|
|
|
{
|
2018-09-18 18:09:16 +03:00
|
|
|
aio_wait_kick();
|
2016-10-27 13:49:05 +03:00
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
void bdrv_dec_in_flight(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
atomic_dec(&bs->in_flight);
|
2016-10-27 13:49:05 +03:00
|
|
|
bdrv_wakeup(bs);
|
2016-10-27 13:48:52 +03:00
|
|
|
}
|
|
|
|
|
2020-01-08 17:55:55 +03:00
|
|
|
static bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
BlockDriverState *bs = self->bs;
|
|
|
|
bool waited = false;
|
|
|
|
|
2017-06-05 15:38:53 +03:00
|
|
|
if (!atomic_read(&bs->serialising_in_flight)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2020-01-08 17:55:56 +03:00
|
|
|
qemu_co_mutex_lock(&bs->reqs_lock);
|
|
|
|
waited = bdrv_wait_serialising_requests_locked(bs, self);
|
|
|
|
qemu_co_mutex_unlock(&bs->reqs_lock);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
return waited;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int bdrv_check_byte_request(BlockDriverState *bs, int64_t offset,
|
|
|
|
size_t size)
|
|
|
|
{
|
2019-05-14 16:57:35 +03:00
|
|
|
if (size > BDRV_REQUEST_MAX_BYTES) {
|
2015-04-28 16:27:52 +03:00
|
|
|
return -EIO;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!bdrv_is_inserted(bs)) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (offset < 0) {
|
|
|
|
return -EIO;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
typedef struct RwCo {
|
2016-05-30 16:00:40 +03:00
|
|
|
BdrvChild *child;
|
2015-04-28 16:27:52 +03:00
|
|
|
int64_t offset;
|
|
|
|
QEMUIOVector *qiov;
|
|
|
|
bool is_write;
|
|
|
|
int ret;
|
|
|
|
BdrvRequestFlags flags;
|
|
|
|
} RwCo;
|
|
|
|
|
|
|
|
static void coroutine_fn bdrv_rw_co_entry(void *opaque)
|
|
|
|
{
|
|
|
|
RwCo *rwco = opaque;
|
|
|
|
|
|
|
|
if (!rwco->is_write) {
|
2016-06-20 22:31:46 +03:00
|
|
|
rwco->ret = bdrv_co_preadv(rwco->child, rwco->offset,
|
2016-04-25 15:57:23 +03:00
|
|
|
rwco->qiov->size, rwco->qiov,
|
|
|
|
rwco->flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
2016-06-20 22:31:46 +03:00
|
|
|
rwco->ret = bdrv_co_pwritev(rwco->child, rwco->offset,
|
2016-04-25 15:57:23 +03:00
|
|
|
rwco->qiov->size, rwco->qiov,
|
|
|
|
rwco->flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Process a vectored synchronous request using coroutines
|
|
|
|
*/
|
2016-05-30 16:00:40 +03:00
|
|
|
static int bdrv_prwv_co(BdrvChild *child, int64_t offset,
|
2015-04-28 16:27:52 +03:00
|
|
|
QEMUIOVector *qiov, bool is_write,
|
|
|
|
BdrvRequestFlags flags)
|
|
|
|
{
|
|
|
|
Coroutine *co;
|
|
|
|
RwCo rwco = {
|
2016-05-30 16:00:40 +03:00
|
|
|
.child = child,
|
2015-04-28 16:27:52 +03:00
|
|
|
.offset = offset,
|
|
|
|
.qiov = qiov,
|
|
|
|
.is_write = is_write,
|
|
|
|
.ret = NOT_DONE,
|
|
|
|
.flags = flags,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
/* Fast-path if already in coroutine context */
|
|
|
|
bdrv_rw_co_entry(&rwco);
|
|
|
|
} else {
|
coroutine: move entry argument to qemu_coroutine_create
In practice the entry argument is always known at creation time, and
it is confusing that sometimes qemu_coroutine_enter is used with a
non-NULL argument to re-enter a coroutine (this happens in
block/sheepdog.c and tests/test-coroutine.c). So pass the opaque value
at creation time, for consistency with e.g. aio_bh_new.
Mostly done with the following semantic patch:
@ entry1 @
expression entry, arg, co;
@@
- co = qemu_coroutine_create(entry);
+ co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry2 @
expression entry, arg;
identifier co;
@@
- Coroutine *co = qemu_coroutine_create(entry);
+ Coroutine *co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry3 @
expression entry, arg;
@@
- qemu_coroutine_enter(qemu_coroutine_create(entry), arg);
+ qemu_coroutine_enter(qemu_coroutine_create(entry, arg));
@ reentry @
expression co;
@@
- qemu_coroutine_enter(co, NULL);
+ qemu_coroutine_enter(co);
except for the aforementioned few places where the semantic patch
stumbled (as expected) and for test_co_queue, which would otherwise
produce an uninitialized variable warning.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-07-04 20:10:01 +03:00
|
|
|
co = qemu_coroutine_create(bdrv_rw_co_entry, &rwco);
|
block: Use bdrv_coroutine_enter to start I/O coroutines
BDRV_POLL_WHILE waits for the started I/O by releasing bs's ctx then polling
the main context, which relies on the yielded coroutine continuing on bs->ctx
before notifying qemu_aio_context with bdrv_wakeup().
Thus, using qemu_coroutine_enter to start I/O is wrong because if the coroutine
is entered from main loop, co->ctx will be qemu_aio_context, as a result of the
"release, poll, acquire" loop of BDRV_POLL_WHILE, race conditions happen when
both main thread and the iothread access the same BDS:
main loop iothread
-----------------------------------------------------------------------
blockdev_snapshot
aio_context_acquire(bs->ctx)
virtio_scsi_data_plane_handle_cmd
bdrv_drained_begin(bs->ctx)
bdrv_flush(bs)
bdrv_co_flush(bs) aio_context_acquire(bs->ctx).enter
...
qemu_coroutine_yield(co)
BDRV_POLL_WHILE()
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx).return
...
aio_co_wake(co)
aio_poll(qemu_aio_context) ...
co_schedule_bh_cb() ...
qemu_coroutine_enter(co) ...
/* (A) bdrv_co_flush(bs) /* (B) I/O on bs */
continues... */
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx)
Note that in above case, bdrv_drained_begin() doesn't do the "release,
poll, acquire" in BDRV_POLL_WHILE, because bs->in_flight == 0.
Fix this by using bdrv_coroutine_enter and enter coroutine in the right
context.
iotests 109 output is updated because the coroutine reenter flow during
mirror job complete is different (now through co_queue_wakeup, instead
of the unconditional qemu_coroutine_switch before), making the end job
len different.
Signed-off-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2017-04-10 15:20:17 +03:00
|
|
|
bdrv_coroutine_enter(child->bs, co);
|
2016-10-27 13:48:55 +03:00
|
|
|
BDRV_POLL_WHILE(child->bs, rwco.ret == NOT_DONE);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
return rwco.ret;
|
|
|
|
}
|
|
|
|
|
2016-06-16 16:13:15 +03:00
|
|
|
int bdrv_pwrite_zeroes(BdrvChild *child, int64_t offset,
|
2017-06-09 13:18:08 +03:00
|
|
|
int bytes, BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2019-02-18 17:09:11 +03:00
|
|
|
QEMUIOVector qiov = QEMU_IOVEC_INIT_BUF(qiov, NULL, bytes);
|
2016-06-02 00:10:04 +03:00
|
|
|
|
2016-05-30 16:00:40 +03:00
|
|
|
return bdrv_prwv_co(child, offset, &qiov, true,
|
2016-06-02 00:10:04 +03:00
|
|
|
BDRV_REQ_ZERO_WRITE | flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2016-06-02 00:10:04 +03:00
|
|
|
* Completely zero out a block device with the help of bdrv_pwrite_zeroes.
|
2015-04-28 16:27:52 +03:00
|
|
|
* The operation is sped up by checking the block status and only writing
|
|
|
|
* zeroes to the device if they currently do not return zeroes. Optional
|
2016-06-02 00:10:04 +03:00
|
|
|
* flags are passed through to bdrv_pwrite_zeroes (e.g. BDRV_REQ_MAY_UNMAP,
|
block: Honor BDRV_REQ_FUA during write_zeroes
The block layer has a couple of cases where it can lose
Force Unit Access semantics when writing a large block of
zeroes, such that the request returns before the zeroes
have been guaranteed to land on underlying media.
SCSI does not support FUA during WRITESAME(10/16); FUA is only
supported if it falls back to WRITE(10/16). But where the
underlying device is new enough to not need a fallback, it
means that any upper layer request with FUA semantics was
silently ignoring BDRV_REQ_FUA.
Conversely, NBD has situations where it can support FUA but not
ZERO_WRITE; when that happens, the generic block layer fallback
to bdrv_driver_pwritev() (or the older bdrv_co_writev() in qemu
2.6) was losing the FUA flag.
The problem of losing flags unrelated to ZERO_WRITE has been
latent in bdrv_co_do_write_zeroes() since commit aa7bfbff, but
back then, it did not matter because there was no FUA flag. It
became observable when commit 93f5e6d8 paved the way for flags
that can impact correctness, when we should have been using
bdrv_co_writev_flags() with modified flags. Compare to commit
9eeb6dd, which got flag manipulation right in
bdrv_co_do_zero_pwritev().
Symptoms: I tested with qemu-io with default writethrough cache
(which is supposed to use FUA semantics on every write), and
targetted an NBD client connected to a server that intentionally
did not advertise NBD_FLAG_SEND_FUA. When doing 'write 0 512',
the NBD client sent two operations (NBD_CMD_WRITE then
NBD_CMD_FLUSH) to get the fallback FUA semantics; but when doing
'write -z 0 512', the NBD client sent only NBD_CMD_WRITE.
The fix is do to a cleanup bdrv_co_flush() at the end of the
operation if any step in the middle relied on a BDS that does
not natively support FUA for that step (note that we don't
need to flush after every operation, if the operation is broken
into chunks based on bounce-buffer sizing). Each BDS gains a
new flag .supported_zero_flags, which parallels the use of
.supported_write_flags but only when accessing a zero write
operation (the flags MUST be different, because of SCSI having
different semantics based on WRITE vs. WRITESAME; and also
because BDRV_REQ_MAY_UNMAP only makes sense on zero writes).
Also fix some documentation to describe -ENOTSUP semantics,
particularly since iscsi depends on those semantics.
Down the road, we may want to add a driver where its
.bdrv_co_pwritev() honors all three of BDRV_REQ_FUA,
BDRV_REQ_ZERO_WRITE, and BDRV_REQ_MAY_UNMAP, and advertise
this via bs->supported_write_flags for blocks opened by that
driver; such a driver should NOT supply .bdrv_co_write_zeroes
nor .supported_zero_flags. But none of the drivers touched
in this patch want to do that (the act of writing zeroes is
different enough from normal writes to deserve a second
callback).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-05-04 01:39:07 +03:00
|
|
|
* BDRV_REQ_FUA).
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
|
|
|
* Returns < 0 on error, 0 on success. For error codes see bdrv_write().
|
|
|
|
*/
|
2016-06-16 16:13:15 +03:00
|
|
|
int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
int ret;
|
|
|
|
int64_t target_size, bytes, offset = 0;
|
2016-06-16 16:13:15 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-10-12 06:47:01 +03:00
|
|
|
target_size = bdrv_getlength(bs);
|
|
|
|
if (target_size < 0) {
|
|
|
|
return target_size;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
for (;;) {
|
2017-10-12 06:47:01 +03:00
|
|
|
bytes = MIN(target_size - offset, BDRV_REQUEST_MAX_BYTES);
|
|
|
|
if (bytes <= 0) {
|
2015-04-28 16:27:52 +03:00
|
|
|
return 0;
|
|
|
|
}
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
ret = bdrv_block_status(bs, offset, bytes, &bytes, NULL, NULL);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
if (ret & BDRV_BLOCK_ZERO) {
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
offset += bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
continue;
|
|
|
|
}
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
ret = bdrv_pwrite_zeroes(child, offset, bytes, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
offset += bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-06-20 19:24:02 +03:00
|
|
|
int bdrv_preadv(BdrvChild *child, int64_t offset, QEMUIOVector *qiov)
|
2016-06-09 17:36:00 +03:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-05-30 16:00:40 +03:00
|
|
|
ret = bdrv_prwv_co(child, offset, qiov, false, 0);
|
2016-06-09 17:36:00 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return qiov->size;
|
|
|
|
}
|
|
|
|
|
2019-05-01 21:13:58 +03:00
|
|
|
/* See bdrv_pwrite() for the return codes */
|
2016-06-20 19:24:02 +03:00
|
|
|
int bdrv_pread(BdrvChild *child, int64_t offset, void *buf, int bytes)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2019-02-18 17:09:11 +03:00
|
|
|
QEMUIOVector qiov = QEMU_IOVEC_INIT_BUF(qiov, buf, bytes);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
if (bytes < 0) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2016-06-20 19:24:02 +03:00
|
|
|
return bdrv_preadv(child, offset, &qiov);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-06-20 21:09:15 +03:00
|
|
|
int bdrv_pwritev(BdrvChild *child, int64_t offset, QEMUIOVector *qiov)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-05-30 16:00:40 +03:00
|
|
|
ret = bdrv_prwv_co(child, offset, qiov, true, 0);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return qiov->size;
|
|
|
|
}
|
|
|
|
|
2019-05-01 21:13:58 +03:00
|
|
|
/* Return no. of bytes on success or < 0 on error. Important errors are:
|
|
|
|
-EIO generic I/O error (may happen for all errors)
|
|
|
|
-ENOMEDIUM No media inserted.
|
|
|
|
-EINVAL Invalid offset or number of bytes
|
|
|
|
-EACCES Trying to write a read-only device
|
|
|
|
*/
|
2016-06-20 21:09:15 +03:00
|
|
|
int bdrv_pwrite(BdrvChild *child, int64_t offset, const void *buf, int bytes)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2019-02-18 17:09:11 +03:00
|
|
|
QEMUIOVector qiov = QEMU_IOVEC_INIT_BUF(qiov, buf, bytes);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
if (bytes < 0) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2016-06-20 21:09:15 +03:00
|
|
|
return bdrv_pwritev(child, offset, &qiov);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Writes to the file and ensures that no writes are reordered across this
|
|
|
|
* request (acts as a barrier)
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno in error cases.
|
|
|
|
*/
|
2016-06-20 21:09:15 +03:00
|
|
|
int bdrv_pwrite_sync(BdrvChild *child, int64_t offset,
|
|
|
|
const void *buf, int count)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-06-20 21:09:15 +03:00
|
|
|
ret = bdrv_pwrite(child, offset, buf, count);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-06-20 21:09:15 +03:00
|
|
|
ret = bdrv_flush(child->bs);
|
2016-03-04 16:16:51 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-04-25 15:13:12 +03:00
|
|
|
typedef struct CoroutineIOCompletion {
|
|
|
|
Coroutine *coroutine;
|
|
|
|
int ret;
|
|
|
|
} CoroutineIOCompletion;
|
|
|
|
|
|
|
|
static void bdrv_co_io_em_complete(void *opaque, int ret)
|
|
|
|
{
|
|
|
|
CoroutineIOCompletion *co = opaque;
|
|
|
|
|
|
|
|
co->ret = ret;
|
2017-02-13 16:52:32 +03:00
|
|
|
aio_co_wake(co->coroutine);
|
2016-04-25 15:13:12 +03:00
|
|
|
}
|
|
|
|
|
2016-04-25 12:46:41 +03:00
|
|
|
static int coroutine_fn bdrv_driver_preadv(BlockDriverState *bs,
|
|
|
|
uint64_t offset, uint64_t bytes,
|
2019-06-04 19:15:06 +03:00
|
|
|
QEMUIOVector *qiov,
|
|
|
|
size_t qiov_offset, int flags)
|
2016-04-25 12:46:41 +03:00
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
2016-04-25 12:25:18 +03:00
|
|
|
int64_t sector_num;
|
|
|
|
unsigned int nb_sectors;
|
2019-06-04 19:15:06 +03:00
|
|
|
QEMUIOVector local_qiov;
|
|
|
|
int ret;
|
2016-04-25 12:25:18 +03:00
|
|
|
|
2016-06-13 21:56:35 +03:00
|
|
|
assert(!(flags & ~BDRV_REQ_MASK));
|
2019-03-22 15:38:43 +03:00
|
|
|
assert(!(flags & BDRV_REQ_NO_FALLBACK));
|
2016-06-13 21:56:35 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
if (drv->bdrv_co_preadv_part) {
|
|
|
|
return drv->bdrv_co_preadv_part(bs, offset, bytes, qiov, qiov_offset,
|
|
|
|
flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (qiov_offset > 0 || bytes != qiov->size) {
|
|
|
|
qemu_iovec_init_slice(&local_qiov, qiov, qiov_offset, bytes);
|
|
|
|
qiov = &local_qiov;
|
|
|
|
}
|
|
|
|
|
2016-04-25 12:25:18 +03:00
|
|
|
if (drv->bdrv_co_preadv) {
|
2019-06-04 19:15:06 +03:00
|
|
|
ret = drv->bdrv_co_preadv(bs, offset, bytes, qiov, flags);
|
|
|
|
goto out;
|
2016-04-25 12:25:18 +03:00
|
|
|
}
|
|
|
|
|
2018-04-24 22:25:06 +03:00
|
|
|
if (drv->bdrv_aio_preadv) {
|
2016-04-25 15:13:12 +03:00
|
|
|
BlockAIOCB *acb;
|
|
|
|
CoroutineIOCompletion co = {
|
|
|
|
.coroutine = qemu_coroutine_self(),
|
|
|
|
};
|
|
|
|
|
2018-04-24 22:25:06 +03:00
|
|
|
acb = drv->bdrv_aio_preadv(bs, offset, bytes, qiov, flags,
|
|
|
|
bdrv_co_io_em_complete, &co);
|
2016-04-25 15:13:12 +03:00
|
|
|
if (acb == NULL) {
|
2019-06-04 19:15:06 +03:00
|
|
|
ret = -EIO;
|
|
|
|
goto out;
|
2016-04-25 15:13:12 +03:00
|
|
|
} else {
|
|
|
|
qemu_coroutine_yield();
|
2019-06-04 19:15:06 +03:00
|
|
|
ret = co.ret;
|
|
|
|
goto out;
|
2016-04-25 15:13:12 +03:00
|
|
|
}
|
|
|
|
}
|
2018-04-24 22:25:06 +03:00
|
|
|
|
|
|
|
sector_num = offset >> BDRV_SECTOR_BITS;
|
|
|
|
nb_sectors = bytes >> BDRV_SECTOR_BITS;
|
|
|
|
|
2019-08-27 21:59:12 +03:00
|
|
|
assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
|
|
|
|
assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
|
2019-05-14 16:57:35 +03:00
|
|
|
assert(bytes <= BDRV_REQUEST_MAX_BYTES);
|
2018-04-24 22:25:06 +03:00
|
|
|
assert(drv->bdrv_co_readv);
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
ret = drv->bdrv_co_readv(bs, sector_num, nb_sectors, qiov);
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (qiov == &local_qiov) {
|
|
|
|
qemu_iovec_destroy(&local_qiov);
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
2016-04-25 12:46:41 +03:00
|
|
|
}
|
|
|
|
|
2016-04-25 13:13:39 +03:00
|
|
|
static int coroutine_fn bdrv_driver_pwritev(BlockDriverState *bs,
|
|
|
|
uint64_t offset, uint64_t bytes,
|
2019-06-04 19:15:06 +03:00
|
|
|
QEMUIOVector *qiov,
|
|
|
|
size_t qiov_offset, int flags)
|
2016-04-25 13:13:39 +03:00
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
2016-04-25 12:25:18 +03:00
|
|
|
int64_t sector_num;
|
|
|
|
unsigned int nb_sectors;
|
2019-06-04 19:15:06 +03:00
|
|
|
QEMUIOVector local_qiov;
|
2016-04-25 13:13:39 +03:00
|
|
|
int ret;
|
|
|
|
|
2016-06-13 21:56:35 +03:00
|
|
|
assert(!(flags & ~BDRV_REQ_MASK));
|
2019-03-22 15:38:43 +03:00
|
|
|
assert(!(flags & BDRV_REQ_NO_FALLBACK));
|
2016-06-13 21:56:35 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
if (drv->bdrv_co_pwritev_part) {
|
|
|
|
ret = drv->bdrv_co_pwritev_part(bs, offset, bytes, qiov, qiov_offset,
|
|
|
|
flags & bs->supported_write_flags);
|
|
|
|
flags &= ~bs->supported_write_flags;
|
|
|
|
goto emulate_flags;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (qiov_offset > 0 || bytes != qiov->size) {
|
|
|
|
qemu_iovec_init_slice(&local_qiov, qiov, qiov_offset, bytes);
|
|
|
|
qiov = &local_qiov;
|
|
|
|
}
|
|
|
|
|
2016-04-25 12:25:18 +03:00
|
|
|
if (drv->bdrv_co_pwritev) {
|
2016-06-07 16:51:28 +03:00
|
|
|
ret = drv->bdrv_co_pwritev(bs, offset, bytes, qiov,
|
|
|
|
flags & bs->supported_write_flags);
|
|
|
|
flags &= ~bs->supported_write_flags;
|
2016-04-25 12:25:18 +03:00
|
|
|
goto emulate_flags;
|
|
|
|
}
|
|
|
|
|
2018-04-24 22:25:06 +03:00
|
|
|
if (drv->bdrv_aio_pwritev) {
|
2016-04-25 15:13:12 +03:00
|
|
|
BlockAIOCB *acb;
|
|
|
|
CoroutineIOCompletion co = {
|
|
|
|
.coroutine = qemu_coroutine_self(),
|
|
|
|
};
|
|
|
|
|
2018-04-24 22:25:06 +03:00
|
|
|
acb = drv->bdrv_aio_pwritev(bs, offset, bytes, qiov,
|
|
|
|
flags & bs->supported_write_flags,
|
|
|
|
bdrv_co_io_em_complete, &co);
|
|
|
|
flags &= ~bs->supported_write_flags;
|
2016-04-25 15:13:12 +03:00
|
|
|
if (acb == NULL) {
|
2016-04-25 12:25:18 +03:00
|
|
|
ret = -EIO;
|
2016-04-25 15:13:12 +03:00
|
|
|
} else {
|
|
|
|
qemu_coroutine_yield();
|
2016-04-25 12:25:18 +03:00
|
|
|
ret = co.ret;
|
2016-04-25 15:13:12 +03:00
|
|
|
}
|
2018-04-24 22:25:06 +03:00
|
|
|
goto emulate_flags;
|
|
|
|
}
|
|
|
|
|
|
|
|
sector_num = offset >> BDRV_SECTOR_BITS;
|
|
|
|
nb_sectors = bytes >> BDRV_SECTOR_BITS;
|
|
|
|
|
2019-08-27 21:59:12 +03:00
|
|
|
assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
|
|
|
|
assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
|
2019-05-14 16:57:35 +03:00
|
|
|
assert(bytes <= BDRV_REQUEST_MAX_BYTES);
|
2018-04-24 22:25:06 +03:00
|
|
|
|
2018-04-25 01:01:57 +03:00
|
|
|
assert(drv->bdrv_co_writev);
|
|
|
|
ret = drv->bdrv_co_writev(bs, sector_num, nb_sectors, qiov,
|
|
|
|
flags & bs->supported_write_flags);
|
|
|
|
flags &= ~bs->supported_write_flags;
|
2016-04-25 13:13:39 +03:00
|
|
|
|
2016-04-25 12:25:18 +03:00
|
|
|
emulate_flags:
|
2016-05-04 01:39:06 +03:00
|
|
|
if (ret == 0 && (flags & BDRV_REQ_FUA)) {
|
2016-04-25 13:13:39 +03:00
|
|
|
ret = bdrv_co_flush(bs);
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
if (qiov == &local_qiov) {
|
|
|
|
qemu_iovec_destroy(&local_qiov);
|
|
|
|
}
|
|
|
|
|
2016-04-25 13:13:39 +03:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-07-22 11:17:42 +03:00
|
|
|
static int coroutine_fn
|
|
|
|
bdrv_driver_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
|
2019-06-04 19:15:06 +03:00
|
|
|
uint64_t bytes, QEMUIOVector *qiov,
|
|
|
|
size_t qiov_offset)
|
2016-07-22 11:17:42 +03:00
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
2019-06-04 19:15:06 +03:00
|
|
|
QEMUIOVector local_qiov;
|
|
|
|
int ret;
|
2016-07-22 11:17:42 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
if (!block_driver_can_compress(drv)) {
|
2016-07-22 11:17:42 +03:00
|
|
|
return -ENOTSUP;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
if (drv->bdrv_co_pwritev_compressed_part) {
|
|
|
|
return drv->bdrv_co_pwritev_compressed_part(bs, offset, bytes,
|
|
|
|
qiov, qiov_offset);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (qiov_offset == 0) {
|
|
|
|
return drv->bdrv_co_pwritev_compressed(bs, offset, bytes, qiov);
|
|
|
|
}
|
|
|
|
|
|
|
|
qemu_iovec_init_slice(&local_qiov, qiov, qiov_offset, bytes);
|
|
|
|
ret = drv->bdrv_co_pwritev_compressed(bs, offset, bytes, &local_qiov);
|
|
|
|
qemu_iovec_destroy(&local_qiov);
|
|
|
|
|
|
|
|
return ret;
|
2016-07-22 11:17:42 +03:00
|
|
|
}
|
|
|
|
|
2017-02-09 17:58:43 +03:00
|
|
|
static int coroutine_fn bdrv_co_do_copy_on_readv(BdrvChild *child,
|
2019-07-25 13:05:48 +03:00
|
|
|
int64_t offset, unsigned int bytes, QEMUIOVector *qiov,
|
2019-06-04 19:15:07 +03:00
|
|
|
size_t qiov_offset, int flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-02-09 17:58:43 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/* Perform I/O through a temporary buffer so that users who scribble over
|
|
|
|
* their read buffer while the operation is in progress do not end up
|
|
|
|
* modifying the image file. This is critical for zero-copy guest I/O
|
|
|
|
* where anything might happen inside guest memory.
|
|
|
|
*/
|
2019-06-04 19:15:08 +03:00
|
|
|
void *bounce_buffer = NULL;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
BlockDriver *drv = bs->drv;
|
2016-06-02 12:41:52 +03:00
|
|
|
int64_t cluster_offset;
|
2017-10-12 06:46:59 +03:00
|
|
|
int64_t cluster_bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
size_t skip_bytes;
|
|
|
|
int ret;
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
int max_transfer = MIN_NON_ZERO(bs->bl.max_transfer,
|
|
|
|
BDRV_REQUEST_MAX_BYTES);
|
|
|
|
unsigned int progress = 0;
|
2019-10-01 20:48:26 +03:00
|
|
|
bool skip_write;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2019-10-01 20:48:26 +03:00
|
|
|
/*
|
|
|
|
* Do not write anything when the BDS is inactive. That is not
|
|
|
|
* allowed, and it would not help.
|
|
|
|
*/
|
|
|
|
skip_write = (bs->open_flags & BDRV_O_INACTIVE);
|
|
|
|
|
2017-04-07 13:29:05 +03:00
|
|
|
/* FIXME We cannot require callers to have write permissions when all they
|
|
|
|
* are doing is a read request. If we did things right, write permissions
|
|
|
|
* would be obtained anyway, but internally by the copy-on-read code. As
|
2017-09-28 22:13:25 +03:00
|
|
|
* long as it is implemented here rather than in a separate filter driver,
|
2017-04-07 13:29:05 +03:00
|
|
|
* the copy-on-read code doesn't have its own BdrvChild, however, for which
|
|
|
|
* it could request permissions. Therefore we have to bypass the permission
|
|
|
|
* system for the moment. */
|
|
|
|
// assert(child->perm & (BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE));
|
2017-02-09 18:49:53 +03:00
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/* Cover entire cluster so no additional backing file I/O is required when
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
* allocating cluster in the image file. Note that this value may exceed
|
|
|
|
* BDRV_REQUEST_MAX_BYTES (even when the original read did not), which
|
|
|
|
* is one reason we loop rather than doing it all at once.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2016-06-02 12:41:52 +03:00
|
|
|
bdrv_round_to_clusters(bs, offset, bytes, &cluster_offset, &cluster_bytes);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
skip_bytes = offset - cluster_offset;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-02 12:41:52 +03:00
|
|
|
trace_bdrv_co_do_copy_on_readv(bs, offset, bytes,
|
|
|
|
cluster_offset, cluster_bytes);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
while (cluster_bytes) {
|
|
|
|
int64_t pnum;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-10-01 20:48:26 +03:00
|
|
|
if (skip_write) {
|
|
|
|
ret = 1; /* "already allocated", so nothing will be copied */
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
pnum = MIN(cluster_bytes, max_transfer);
|
2019-10-01 20:48:26 +03:00
|
|
|
} else {
|
|
|
|
ret = bdrv_is_allocated(bs, cluster_offset,
|
|
|
|
MIN(cluster_bytes, max_transfer), &pnum);
|
|
|
|
if (ret < 0) {
|
|
|
|
/*
|
|
|
|
* Safe to treat errors in querying allocation as if
|
|
|
|
* unallocated; we'll probably fail again soon on the
|
|
|
|
* read, but at least that will set a decent errno.
|
|
|
|
*/
|
|
|
|
pnum = MIN(cluster_bytes, max_transfer);
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-10-01 20:48:26 +03:00
|
|
|
/* Stop at EOF if the image ends in the middle of the cluster */
|
|
|
|
if (ret == 0 && pnum == 0) {
|
|
|
|
assert(progress >= bytes);
|
|
|
|
break;
|
|
|
|
}
|
2018-07-06 19:41:07 +03:00
|
|
|
|
2019-10-01 20:48:26 +03:00
|
|
|
assert(skip_bytes < pnum);
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
if (ret <= 0) {
|
2019-06-04 19:15:07 +03:00
|
|
|
QEMUIOVector local_qiov;
|
|
|
|
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
/* Must copy-on-read; use the bounce buffer */
|
2019-02-18 17:09:11 +03:00
|
|
|
pnum = MIN(pnum, MAX_BOUNCE_BUFFER);
|
2019-06-04 19:15:08 +03:00
|
|
|
if (!bounce_buffer) {
|
|
|
|
int64_t max_we_need = MAX(pnum, cluster_bytes - pnum);
|
|
|
|
int64_t max_allowed = MIN(max_transfer, MAX_BOUNCE_BUFFER);
|
|
|
|
int64_t bounce_buffer_len = MIN(max_we_need, max_allowed);
|
|
|
|
|
|
|
|
bounce_buffer = qemu_try_blockalign(bs, bounce_buffer_len);
|
|
|
|
if (!bounce_buffer) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
}
|
2019-02-18 17:09:11 +03:00
|
|
|
qemu_iovec_init_buf(&local_qiov, bounce_buffer, pnum);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
ret = bdrv_driver_preadv(bs, cluster_offset, pnum,
|
2019-06-04 19:15:06 +03:00
|
|
|
&local_qiov, 0, 0);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
bdrv_debug_event(bs, BLKDBG_COR_WRITE);
|
|
|
|
if (drv->bdrv_co_pwrite_zeroes &&
|
|
|
|
buffer_is_zero(bounce_buffer, pnum)) {
|
|
|
|
/* FIXME: Should we (perhaps conditionally) be setting
|
|
|
|
* BDRV_REQ_MAY_UNMAP, if it will allow for a sparser copy
|
|
|
|
* that still correctly reads as zero? */
|
2018-04-21 16:29:24 +03:00
|
|
|
ret = bdrv_co_do_pwrite_zeroes(bs, cluster_offset, pnum,
|
|
|
|
BDRV_REQ_WRITE_UNCHANGED);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
} else {
|
|
|
|
/* This does not change the data on the disk, it is not
|
|
|
|
* necessary to flush even in cache=writethrough mode.
|
|
|
|
*/
|
|
|
|
ret = bdrv_driver_pwritev(bs, cluster_offset, pnum,
|
2019-06-04 19:15:06 +03:00
|
|
|
&local_qiov, 0,
|
2018-04-21 16:29:24 +03:00
|
|
|
BDRV_REQ_WRITE_UNCHANGED);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
if (ret < 0) {
|
|
|
|
/* It might be okay to ignore write errors for guest
|
|
|
|
* requests. If this is a deliberate copy-on-read
|
|
|
|
* then we don't want to ignore the error. Simply
|
|
|
|
* report it in all cases.
|
|
|
|
*/
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2019-07-25 13:05:48 +03:00
|
|
|
if (!(flags & BDRV_REQ_PREFETCH)) {
|
2019-06-04 19:15:07 +03:00
|
|
|
qemu_iovec_from_buf(qiov, qiov_offset + progress,
|
|
|
|
bounce_buffer + skip_bytes,
|
2020-03-12 11:19:49 +03:00
|
|
|
MIN(pnum - skip_bytes, bytes - progress));
|
2019-07-25 13:05:48 +03:00
|
|
|
}
|
|
|
|
} else if (!(flags & BDRV_REQ_PREFETCH)) {
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
/* Read directly into the destination */
|
2019-06-04 19:15:07 +03:00
|
|
|
ret = bdrv_driver_preadv(bs, offset + progress,
|
|
|
|
MIN(pnum - skip_bytes, bytes - progress),
|
|
|
|
qiov, qiov_offset + progress, 0);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
cluster_offset += pnum;
|
|
|
|
cluster_bytes -= pnum;
|
|
|
|
progress += pnum - skip_bytes;
|
|
|
|
skip_bytes = 0;
|
|
|
|
}
|
|
|
|
ret = 0;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
err:
|
|
|
|
qemu_vfree(bounce_buffer);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Forwards an already correctly aligned request to the BlockDriver. This
|
2016-07-15 21:31:59 +03:00
|
|
|
* handles copy on read, zeroing after EOF, and fragmentation of large
|
|
|
|
* reads; any other features must be implemented by the caller.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2017-02-09 17:58:43 +03:00
|
|
|
static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child,
|
2015-04-28 16:27:52 +03:00
|
|
|
BdrvTrackedRequest *req, int64_t offset, unsigned int bytes,
|
2019-06-04 19:15:09 +03:00
|
|
|
int64_t align, QEMUIOVector *qiov, size_t qiov_offset, int flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-02-09 17:58:43 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2016-06-01 18:13:47 +03:00
|
|
|
int64_t total_bytes, max_bytes;
|
2016-07-15 21:31:59 +03:00
|
|
|
int ret = 0;
|
|
|
|
uint64_t bytes_remaining = bytes;
|
|
|
|
int max_transfer;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-03 17:17:28 +03:00
|
|
|
assert(is_power_of_2(align));
|
|
|
|
assert((offset & (align - 1)) == 0);
|
|
|
|
assert((bytes & (align - 1)) == 0);
|
2016-03-21 17:11:42 +03:00
|
|
|
assert((bs->open_flags & BDRV_O_NO_IO) == 0);
|
2016-07-15 21:31:59 +03:00
|
|
|
max_transfer = QEMU_ALIGN_DOWN(MIN_NON_ZERO(bs->bl.max_transfer, INT_MAX),
|
|
|
|
align);
|
2016-06-24 01:37:06 +03:00
|
|
|
|
|
|
|
/* TODO: We would need a per-BDS .supported_read_flags and
|
|
|
|
* potential fallback support, if we ever implement any read flags
|
|
|
|
* to pass through to drivers. For now, there aren't any
|
|
|
|
* passthrough flags. */
|
2020-01-08 17:55:54 +03:00
|
|
|
assert(!(flags & ~(BDRV_REQ_COPY_ON_READ | BDRV_REQ_PREFETCH)));
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
/* Handle Copy on Read and associated serialisation */
|
|
|
|
if (flags & BDRV_REQ_COPY_ON_READ) {
|
|
|
|
/* If we touch the same cluster it counts as an overlap. This
|
|
|
|
* guarantees that allocating writes will be serialized and not race
|
|
|
|
* with each other for the same cluster. For example, in copy-on-read
|
|
|
|
* it ensures that the CoR read and write operations are atomic and
|
|
|
|
* guest writes cannot interleave between them. */
|
2019-11-01 18:25:08 +03:00
|
|
|
bdrv_mark_request_serialising(req, bdrv_get_cluster_size(bs));
|
2020-01-08 17:55:55 +03:00
|
|
|
} else {
|
|
|
|
bdrv_wait_serialising_requests(req);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
if (flags & BDRV_REQ_COPY_ON_READ) {
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
int64_t pnum;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-10-12 06:47:18 +03:00
|
|
|
ret = bdrv_is_allocated(bs, offset, bytes, &pnum);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2017-10-12 06:47:18 +03:00
|
|
|
if (!ret || pnum != bytes) {
|
2019-06-04 19:15:09 +03:00
|
|
|
ret = bdrv_co_do_copy_on_readv(child, offset, bytes,
|
|
|
|
qiov, qiov_offset, flags);
|
2019-07-25 13:05:48 +03:00
|
|
|
goto out;
|
|
|
|
} else if (flags & BDRV_REQ_PREFETCH) {
|
2015-04-28 16:27:52 +03:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-07-15 21:31:59 +03:00
|
|
|
/* Forward the request to the BlockDriver, possibly fragmenting it */
|
2016-06-01 18:13:47 +03:00
|
|
|
total_bytes = bdrv_getlength(bs);
|
|
|
|
if (total_bytes < 0) {
|
|
|
|
ret = total_bytes;
|
|
|
|
goto out;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-01 18:13:47 +03:00
|
|
|
max_bytes = ROUND_UP(MAX(0, total_bytes - offset), align);
|
2016-07-15 21:31:59 +03:00
|
|
|
if (bytes <= max_bytes && bytes <= max_transfer) {
|
2019-06-04 19:15:09 +03:00
|
|
|
ret = bdrv_driver_preadv(bs, offset, bytes, qiov, qiov_offset, 0);
|
2016-07-15 21:31:59 +03:00
|
|
|
goto out;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-07-15 21:31:59 +03:00
|
|
|
while (bytes_remaining) {
|
|
|
|
int num;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-07-15 21:31:59 +03:00
|
|
|
if (max_bytes) {
|
|
|
|
num = MIN(bytes_remaining, MIN(max_bytes, max_transfer));
|
|
|
|
assert(num);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-07-15 21:31:59 +03:00
|
|
|
ret = bdrv_driver_preadv(bs, offset + bytes - bytes_remaining,
|
2019-06-04 19:15:09 +03:00
|
|
|
num, qiov, bytes - bytes_remaining, 0);
|
2016-07-15 21:31:59 +03:00
|
|
|
max_bytes -= num;
|
|
|
|
} else {
|
|
|
|
num = bytes_remaining;
|
|
|
|
ret = qemu_iovec_memset(qiov, bytes - bytes_remaining, 0,
|
|
|
|
bytes_remaining);
|
|
|
|
}
|
|
|
|
if (ret < 0) {
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
bytes_remaining -= num;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
2016-07-15 21:31:59 +03:00
|
|
|
return ret < 0 ? ret : 0;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-06-04 19:15:05 +03:00
|
|
|
* Request padding
|
|
|
|
*
|
|
|
|
* |<---- align ----->| |<----- align ---->|
|
|
|
|
* |<- head ->|<------------- bytes ------------->|<-- tail -->|
|
|
|
|
* | | | | | |
|
|
|
|
* -*----------$-------*-------- ... --------*-----$------------*---
|
|
|
|
* | | | | | |
|
|
|
|
* | offset | | end |
|
|
|
|
* ALIGN_DOWN(offset) ALIGN_UP(offset) ALIGN_DOWN(end) ALIGN_UP(end)
|
|
|
|
* [buf ... ) [tail_buf )
|
|
|
|
*
|
|
|
|
* @buf is an aligned allocation needed to store @head and @tail paddings. @head
|
|
|
|
* is placed at the beginning of @buf and @tail at the @end.
|
|
|
|
*
|
|
|
|
* @tail_buf is a pointer to sub-buffer, corresponding to align-sized chunk
|
|
|
|
* around tail, if tail exists.
|
|
|
|
*
|
|
|
|
* @merge_reads is true for small requests,
|
|
|
|
* if @buf_len == @head + bytes + @tail. In this case it is possible that both
|
|
|
|
* head and tail exist but @buf_len == align and @tail_buf == @buf.
|
|
|
|
*/
|
|
|
|
typedef struct BdrvRequestPadding {
|
|
|
|
uint8_t *buf;
|
|
|
|
size_t buf_len;
|
|
|
|
uint8_t *tail_buf;
|
|
|
|
size_t head;
|
|
|
|
size_t tail;
|
|
|
|
bool merge_reads;
|
|
|
|
QEMUIOVector local_qiov;
|
|
|
|
} BdrvRequestPadding;
|
|
|
|
|
|
|
|
static bool bdrv_init_padding(BlockDriverState *bs,
|
|
|
|
int64_t offset, int64_t bytes,
|
|
|
|
BdrvRequestPadding *pad)
|
|
|
|
{
|
|
|
|
uint64_t align = bs->bl.request_alignment;
|
|
|
|
size_t sum;
|
|
|
|
|
|
|
|
memset(pad, 0, sizeof(*pad));
|
|
|
|
|
|
|
|
pad->head = offset & (align - 1);
|
|
|
|
pad->tail = ((offset + bytes) & (align - 1));
|
|
|
|
if (pad->tail) {
|
|
|
|
pad->tail = align - pad->tail;
|
|
|
|
}
|
|
|
|
|
2020-02-06 19:42:45 +03:00
|
|
|
if (!pad->head && !pad->tail) {
|
2019-06-04 19:15:05 +03:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2020-02-06 19:42:45 +03:00
|
|
|
assert(bytes); /* Nothing good in aligning zero-length requests */
|
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
sum = pad->head + bytes + pad->tail;
|
|
|
|
pad->buf_len = (sum > align && pad->head && pad->tail) ? 2 * align : align;
|
|
|
|
pad->buf = qemu_blockalign(bs, pad->buf_len);
|
|
|
|
pad->merge_reads = sum == pad->buf_len;
|
|
|
|
if (pad->tail) {
|
|
|
|
pad->tail_buf = pad->buf + pad->buf_len - align;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int bdrv_padding_rmw_read(BdrvChild *child,
|
|
|
|
BdrvTrackedRequest *req,
|
|
|
|
BdrvRequestPadding *pad,
|
|
|
|
bool zero_middle)
|
|
|
|
{
|
|
|
|
QEMUIOVector local_qiov;
|
|
|
|
BlockDriverState *bs = child->bs;
|
|
|
|
uint64_t align = bs->bl.request_alignment;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
assert(req->serialising && pad->buf);
|
|
|
|
|
|
|
|
if (pad->head || pad->merge_reads) {
|
|
|
|
uint64_t bytes = pad->merge_reads ? pad->buf_len : align;
|
|
|
|
|
|
|
|
qemu_iovec_init_buf(&local_qiov, pad->buf, bytes);
|
|
|
|
|
|
|
|
if (pad->head) {
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_HEAD);
|
|
|
|
}
|
|
|
|
if (pad->merge_reads && pad->tail) {
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_TAIL);
|
|
|
|
}
|
|
|
|
ret = bdrv_aligned_preadv(child, req, req->overlap_offset, bytes,
|
2019-06-04 19:15:09 +03:00
|
|
|
align, &local_qiov, 0, 0);
|
2019-06-04 19:15:05 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
if (pad->head) {
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_AFTER_HEAD);
|
|
|
|
}
|
|
|
|
if (pad->merge_reads && pad->tail) {
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_AFTER_TAIL);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (pad->merge_reads) {
|
|
|
|
goto zero_mem;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (pad->tail) {
|
|
|
|
qemu_iovec_init_buf(&local_qiov, pad->tail_buf, align);
|
|
|
|
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_TAIL);
|
|
|
|
ret = bdrv_aligned_preadv(
|
|
|
|
child, req,
|
|
|
|
req->overlap_offset + req->overlap_bytes - align,
|
2019-06-04 19:15:09 +03:00
|
|
|
align, align, &local_qiov, 0, 0);
|
2019-06-04 19:15:05 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_RMW_AFTER_TAIL);
|
|
|
|
}
|
|
|
|
|
|
|
|
zero_mem:
|
|
|
|
if (zero_middle) {
|
|
|
|
memset(pad->buf + pad->head, 0, pad->buf_len - pad->head - pad->tail);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void bdrv_padding_destroy(BdrvRequestPadding *pad)
|
|
|
|
{
|
|
|
|
if (pad->buf) {
|
|
|
|
qemu_vfree(pad->buf);
|
|
|
|
qemu_iovec_destroy(&pad->local_qiov);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* bdrv_pad_request
|
|
|
|
*
|
|
|
|
* Exchange request parameters with padded request if needed. Don't include RMW
|
|
|
|
* read of padding, bdrv_padding_rmw_read() should be called separately if
|
|
|
|
* needed.
|
|
|
|
*
|
|
|
|
* All parameters except @bs are in-out: they represent original request at
|
|
|
|
* function call and padded (if padding needed) at function finish.
|
|
|
|
*
|
|
|
|
* Function always succeeds.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2019-06-04 19:15:11 +03:00
|
|
|
static bool bdrv_pad_request(BlockDriverState *bs,
|
|
|
|
QEMUIOVector **qiov, size_t *qiov_offset,
|
2019-06-04 19:15:05 +03:00
|
|
|
int64_t *offset, unsigned int *bytes,
|
|
|
|
BdrvRequestPadding *pad)
|
|
|
|
{
|
|
|
|
if (!bdrv_init_padding(bs, *offset, *bytes, pad)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
qemu_iovec_init_extended(&pad->local_qiov, pad->buf, pad->head,
|
2019-06-04 19:15:11 +03:00
|
|
|
*qiov, *qiov_offset, *bytes,
|
2019-06-04 19:15:05 +03:00
|
|
|
pad->buf + pad->buf_len - pad->tail, pad->tail);
|
|
|
|
*bytes += pad->head + pad->tail;
|
|
|
|
*offset -= pad->head;
|
|
|
|
*qiov = &pad->local_qiov;
|
2019-06-04 19:15:11 +03:00
|
|
|
*qiov_offset = 0;
|
2019-06-04 19:15:05 +03:00
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2016-06-20 22:31:46 +03:00
|
|
|
int coroutine_fn bdrv_co_preadv(BdrvChild *child,
|
2015-04-28 16:27:52 +03:00
|
|
|
int64_t offset, unsigned int bytes, QEMUIOVector *qiov,
|
|
|
|
BdrvRequestFlags flags)
|
2019-06-04 19:15:11 +03:00
|
|
|
{
|
|
|
|
return bdrv_co_preadv_part(child, offset, bytes, qiov, 0, flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
int coroutine_fn bdrv_co_preadv_part(BdrvChild *child,
|
|
|
|
int64_t offset, unsigned int bytes,
|
|
|
|
QEMUIOVector *qiov, size_t qiov_offset,
|
|
|
|
BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2016-06-20 22:31:46 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
BdrvTrackedRequest req;
|
2019-06-04 19:15:05 +03:00
|
|
|
BdrvRequestPadding pad;
|
2015-04-28 16:27:52 +03:00
|
|
|
int ret;
|
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
trace_bdrv_co_preadv(bs, offset, bytes, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
ret = bdrv_check_byte_request(bs, offset, bytes);
|
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2020-02-06 19:42:45 +03:00
|
|
|
if (bytes == 0 && !QEMU_IS_ALIGNED(offset, bs->bl.request_alignment)) {
|
|
|
|
/*
|
|
|
|
* Aligning zero request is nonsense. Even if driver has special meaning
|
|
|
|
* of zero-length (like qcow2_co_pwritev_compressed_part), we can't pass
|
|
|
|
* it to driver due to request_alignment.
|
|
|
|
*
|
|
|
|
* Still, no reason to return an error if someone do unaligned
|
|
|
|
* zero-length read occasionally.
|
|
|
|
*/
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
|
|
|
|
2015-09-08 06:28:32 +03:00
|
|
|
/* Don't do copy-on-read if we read data before write operation */
|
2020-01-08 17:55:54 +03:00
|
|
|
if (atomic_read(&bs->copy_on_read)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
flags |= BDRV_REQ_COPY_ON_READ;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:11 +03:00
|
|
|
bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, &pad);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2015-11-09 13:16:46 +03:00
|
|
|
tracked_request_begin(&req, bs, offset, bytes, BDRV_TRACKED_READ);
|
2019-06-04 19:15:05 +03:00
|
|
|
ret = bdrv_aligned_preadv(child, &req, offset, bytes,
|
|
|
|
bs->bl.request_alignment,
|
2019-06-04 19:15:11 +03:00
|
|
|
qiov, qiov_offset, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
tracked_request_end(&req);
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
bdrv_padding_destroy(&pad);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-06-02 00:10:03 +03:00
|
|
|
static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs,
|
2017-06-09 13:18:08 +03:00
|
|
|
int64_t offset, int bytes, BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
|
|
|
QEMUIOVector qiov;
|
2019-02-18 17:09:11 +03:00
|
|
|
void *buf = NULL;
|
2015-04-28 16:27:52 +03:00
|
|
|
int ret = 0;
|
block: Honor BDRV_REQ_FUA during write_zeroes
The block layer has a couple of cases where it can lose
Force Unit Access semantics when writing a large block of
zeroes, such that the request returns before the zeroes
have been guaranteed to land on underlying media.
SCSI does not support FUA during WRITESAME(10/16); FUA is only
supported if it falls back to WRITE(10/16). But where the
underlying device is new enough to not need a fallback, it
means that any upper layer request with FUA semantics was
silently ignoring BDRV_REQ_FUA.
Conversely, NBD has situations where it can support FUA but not
ZERO_WRITE; when that happens, the generic block layer fallback
to bdrv_driver_pwritev() (or the older bdrv_co_writev() in qemu
2.6) was losing the FUA flag.
The problem of losing flags unrelated to ZERO_WRITE has been
latent in bdrv_co_do_write_zeroes() since commit aa7bfbff, but
back then, it did not matter because there was no FUA flag. It
became observable when commit 93f5e6d8 paved the way for flags
that can impact correctness, when we should have been using
bdrv_co_writev_flags() with modified flags. Compare to commit
9eeb6dd, which got flag manipulation right in
bdrv_co_do_zero_pwritev().
Symptoms: I tested with qemu-io with default writethrough cache
(which is supposed to use FUA semantics on every write), and
targetted an NBD client connected to a server that intentionally
did not advertise NBD_FLAG_SEND_FUA. When doing 'write 0 512',
the NBD client sent two operations (NBD_CMD_WRITE then
NBD_CMD_FLUSH) to get the fallback FUA semantics; but when doing
'write -z 0 512', the NBD client sent only NBD_CMD_WRITE.
The fix is do to a cleanup bdrv_co_flush() at the end of the
operation if any step in the middle relied on a BDS that does
not natively support FUA for that step (note that we don't
need to flush after every operation, if the operation is broken
into chunks based on bounce-buffer sizing). Each BDS gains a
new flag .supported_zero_flags, which parallels the use of
.supported_write_flags but only when accessing a zero write
operation (the flags MUST be different, because of SCSI having
different semantics based on WRITE vs. WRITESAME; and also
because BDRV_REQ_MAY_UNMAP only makes sense on zero writes).
Also fix some documentation to describe -ENOTSUP semantics,
particularly since iscsi depends on those semantics.
Down the road, we may want to add a driver where its
.bdrv_co_pwritev() honors all three of BDRV_REQ_FUA,
BDRV_REQ_ZERO_WRITE, and BDRV_REQ_MAY_UNMAP, and advertise
this via bs->supported_write_flags for blocks opened by that
driver; such a driver should NOT supply .bdrv_co_write_zeroes
nor .supported_zero_flags. But none of the drivers touched
in this patch want to do that (the act of writing zeroes is
different enough from normal writes to deserve a second
callback).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-05-04 01:39:07 +03:00
|
|
|
bool need_flush = false;
|
2016-05-26 06:48:45 +03:00
|
|
|
int head = 0;
|
|
|
|
int tail = 0;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-02 00:10:02 +03:00
|
|
|
int max_write_zeroes = MIN_NON_ZERO(bs->bl.max_pwrite_zeroes, INT_MAX);
|
2016-06-24 01:37:24 +03:00
|
|
|
int alignment = MAX(bs->bl.pwrite_zeroes_alignment,
|
|
|
|
bs->bl.request_alignment);
|
block: Perform copy-on-read in loop
Improve our braindead copy-on-read implementation. Pre-patch,
we have multiple issues:
- we create a bounce buffer and perform a write for the entire
request, even if the active image already has 99% of the
clusters occupied, and really only needs to copy-on-read the
remaining 1% of the clusters
- our bounce buffer was as large as the read request, and can
needlessly exhaust our memory by using double the memory of
the request size (the original request plus our bounce buffer),
rather than a capped maximum overhead beyond the original
- if a driver has a max_transfer limit, we are bypassing the
normal code in bdrv_aligned_preadv() that fragments to that
limit, and instead attempt to read the entire buffer from the
driver in one go, which some drivers may assert on
- a client can request a large request of nearly 2G such that
rounding the request out to cluster boundaries results in a
byte count larger than 2G. While this cannot exceed 32 bits,
it DOES have some follow-on problems:
-- the call to bdrv_driver_pread() can assert for exceeding
BDRV_REQUEST_MAX_BYTES, if the driver is old and lacks
.bdrv_co_preadv
-- if the buffer is all zeroes, the subsequent call to
bdrv_co_do_pwrite_zeroes is a no-op due to a negative size,
which means we did not actually copy on read
Fix all of these issues by breaking up the action into a loop,
where each iteration is capped to sane limits. Also, querying
the allocation status allows us to optimize: when data is
already present in the active layer, we don't need to bounce.
Note that the code has a telling comment that copy-on-read
should probably be a filter driver rather than a bolt-on hack
in io.c; but that remains a task for another day.
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-05 22:02:47 +03:00
|
|
|
int max_transfer = MIN_NON_ZERO(bs->bl.max_transfer, MAX_BOUNCE_BUFFER);
|
2016-06-02 00:10:03 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2019-03-22 15:38:43 +03:00
|
|
|
if ((flags & ~bs->supported_zero_flags) & BDRV_REQ_NO_FALLBACK) {
|
|
|
|
return -ENOTSUP;
|
|
|
|
}
|
|
|
|
|
2016-07-21 22:34:48 +03:00
|
|
|
assert(alignment % bs->bl.request_alignment == 0);
|
|
|
|
head = offset % alignment;
|
2017-06-09 13:18:08 +03:00
|
|
|
tail = (offset + bytes) % alignment;
|
2016-07-21 22:34:48 +03:00
|
|
|
max_write_zeroes = QEMU_ALIGN_DOWN(max_write_zeroes, alignment);
|
|
|
|
assert(max_write_zeroes >= bs->bl.request_alignment);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-06-09 13:18:08 +03:00
|
|
|
while (bytes > 0 && !ret) {
|
|
|
|
int num = bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
/* Align request. Block drivers can expect the "bulk" of the request
|
2016-05-26 06:48:45 +03:00
|
|
|
* to be aligned, and that unaligned requests do not cross cluster
|
|
|
|
* boundaries.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2016-05-26 06:48:45 +03:00
|
|
|
if (head) {
|
2016-11-17 23:13:56 +03:00
|
|
|
/* Make a small request up to the first aligned sector. For
|
|
|
|
* convenience, limit this request to max_transfer even if
|
|
|
|
* we don't need to fall back to writes. */
|
2017-06-09 13:18:08 +03:00
|
|
|
num = MIN(MIN(bytes, max_transfer), alignment - head);
|
2016-11-17 23:13:56 +03:00
|
|
|
head = (head + num) % alignment;
|
|
|
|
assert(num < max_write_zeroes);
|
2016-06-02 00:10:03 +03:00
|
|
|
} else if (tail && num > alignment) {
|
2016-05-26 06:48:45 +03:00
|
|
|
/* Shorten the request to the last aligned sector. */
|
|
|
|
num -= tail;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* limit request size */
|
|
|
|
if (num > max_write_zeroes) {
|
|
|
|
num = max_write_zeroes;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = -ENOTSUP;
|
|
|
|
/* First try the efficient write zeroes operation */
|
2016-06-02 00:10:03 +03:00
|
|
|
if (drv->bdrv_co_pwrite_zeroes) {
|
|
|
|
ret = drv->bdrv_co_pwrite_zeroes(bs, offset, num,
|
|
|
|
flags & bs->supported_zero_flags);
|
|
|
|
if (ret != -ENOTSUP && (flags & BDRV_REQ_FUA) &&
|
|
|
|
!(bs->supported_zero_flags & BDRV_REQ_FUA)) {
|
|
|
|
need_flush = true;
|
|
|
|
}
|
block: Honor BDRV_REQ_FUA during write_zeroes
The block layer has a couple of cases where it can lose
Force Unit Access semantics when writing a large block of
zeroes, such that the request returns before the zeroes
have been guaranteed to land on underlying media.
SCSI does not support FUA during WRITESAME(10/16); FUA is only
supported if it falls back to WRITE(10/16). But where the
underlying device is new enough to not need a fallback, it
means that any upper layer request with FUA semantics was
silently ignoring BDRV_REQ_FUA.
Conversely, NBD has situations where it can support FUA but not
ZERO_WRITE; when that happens, the generic block layer fallback
to bdrv_driver_pwritev() (or the older bdrv_co_writev() in qemu
2.6) was losing the FUA flag.
The problem of losing flags unrelated to ZERO_WRITE has been
latent in bdrv_co_do_write_zeroes() since commit aa7bfbff, but
back then, it did not matter because there was no FUA flag. It
became observable when commit 93f5e6d8 paved the way for flags
that can impact correctness, when we should have been using
bdrv_co_writev_flags() with modified flags. Compare to commit
9eeb6dd, which got flag manipulation right in
bdrv_co_do_zero_pwritev().
Symptoms: I tested with qemu-io with default writethrough cache
(which is supposed to use FUA semantics on every write), and
targetted an NBD client connected to a server that intentionally
did not advertise NBD_FLAG_SEND_FUA. When doing 'write 0 512',
the NBD client sent two operations (NBD_CMD_WRITE then
NBD_CMD_FLUSH) to get the fallback FUA semantics; but when doing
'write -z 0 512', the NBD client sent only NBD_CMD_WRITE.
The fix is do to a cleanup bdrv_co_flush() at the end of the
operation if any step in the middle relied on a BDS that does
not natively support FUA for that step (note that we don't
need to flush after every operation, if the operation is broken
into chunks based on bounce-buffer sizing). Each BDS gains a
new flag .supported_zero_flags, which parallels the use of
.supported_write_flags but only when accessing a zero write
operation (the flags MUST be different, because of SCSI having
different semantics based on WRITE vs. WRITESAME; and also
because BDRV_REQ_MAY_UNMAP only makes sense on zero writes).
Also fix some documentation to describe -ENOTSUP semantics,
particularly since iscsi depends on those semantics.
Down the road, we may want to add a driver where its
.bdrv_co_pwritev() honors all three of BDRV_REQ_FUA,
BDRV_REQ_ZERO_WRITE, and BDRV_REQ_MAY_UNMAP, and advertise
this via bs->supported_write_flags for blocks opened by that
driver; such a driver should NOT supply .bdrv_co_write_zeroes
nor .supported_zero_flags. But none of the drivers touched
in this patch want to do that (the act of writing zeroes is
different enough from normal writes to deserve a second
callback).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-05-04 01:39:07 +03:00
|
|
|
} else {
|
|
|
|
assert(!bs->supported_zero_flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2019-08-27 16:52:53 +03:00
|
|
|
if (ret == -ENOTSUP && !(flags & BDRV_REQ_NO_FALLBACK)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
/* Fall back to bounce buffer if write zeroes is unsupported */
|
block: Honor BDRV_REQ_FUA during write_zeroes
The block layer has a couple of cases where it can lose
Force Unit Access semantics when writing a large block of
zeroes, such that the request returns before the zeroes
have been guaranteed to land on underlying media.
SCSI does not support FUA during WRITESAME(10/16); FUA is only
supported if it falls back to WRITE(10/16). But where the
underlying device is new enough to not need a fallback, it
means that any upper layer request with FUA semantics was
silently ignoring BDRV_REQ_FUA.
Conversely, NBD has situations where it can support FUA but not
ZERO_WRITE; when that happens, the generic block layer fallback
to bdrv_driver_pwritev() (or the older bdrv_co_writev() in qemu
2.6) was losing the FUA flag.
The problem of losing flags unrelated to ZERO_WRITE has been
latent in bdrv_co_do_write_zeroes() since commit aa7bfbff, but
back then, it did not matter because there was no FUA flag. It
became observable when commit 93f5e6d8 paved the way for flags
that can impact correctness, when we should have been using
bdrv_co_writev_flags() with modified flags. Compare to commit
9eeb6dd, which got flag manipulation right in
bdrv_co_do_zero_pwritev().
Symptoms: I tested with qemu-io with default writethrough cache
(which is supposed to use FUA semantics on every write), and
targetted an NBD client connected to a server that intentionally
did not advertise NBD_FLAG_SEND_FUA. When doing 'write 0 512',
the NBD client sent two operations (NBD_CMD_WRITE then
NBD_CMD_FLUSH) to get the fallback FUA semantics; but when doing
'write -z 0 512', the NBD client sent only NBD_CMD_WRITE.
The fix is do to a cleanup bdrv_co_flush() at the end of the
operation if any step in the middle relied on a BDS that does
not natively support FUA for that step (note that we don't
need to flush after every operation, if the operation is broken
into chunks based on bounce-buffer sizing). Each BDS gains a
new flag .supported_zero_flags, which parallels the use of
.supported_write_flags but only when accessing a zero write
operation (the flags MUST be different, because of SCSI having
different semantics based on WRITE vs. WRITESAME; and also
because BDRV_REQ_MAY_UNMAP only makes sense on zero writes).
Also fix some documentation to describe -ENOTSUP semantics,
particularly since iscsi depends on those semantics.
Down the road, we may want to add a driver where its
.bdrv_co_pwritev() honors all three of BDRV_REQ_FUA,
BDRV_REQ_ZERO_WRITE, and BDRV_REQ_MAY_UNMAP, and advertise
this via bs->supported_write_flags for blocks opened by that
driver; such a driver should NOT supply .bdrv_co_write_zeroes
nor .supported_zero_flags. But none of the drivers touched
in this patch want to do that (the act of writing zeroes is
different enough from normal writes to deserve a second
callback).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-05-04 01:39:07 +03:00
|
|
|
BdrvRequestFlags write_flags = flags & ~BDRV_REQ_ZERO_WRITE;
|
|
|
|
|
|
|
|
if ((flags & BDRV_REQ_FUA) &&
|
|
|
|
!(bs->supported_write_flags & BDRV_REQ_FUA)) {
|
|
|
|
/* No need for bdrv_driver_pwrite() to do a fallback
|
|
|
|
* flush on each chunk; use just one at the end */
|
|
|
|
write_flags &= ~BDRV_REQ_FUA;
|
|
|
|
need_flush = true;
|
|
|
|
}
|
2016-06-24 01:37:19 +03:00
|
|
|
num = MIN(num, max_transfer);
|
2019-02-18 17:09:11 +03:00
|
|
|
if (buf == NULL) {
|
|
|
|
buf = qemu_try_blockalign0(bs, num);
|
|
|
|
if (buf == NULL) {
|
2015-04-28 16:27:52 +03:00
|
|
|
ret = -ENOMEM;
|
|
|
|
goto fail;
|
|
|
|
}
|
|
|
|
}
|
2019-02-18 17:09:11 +03:00
|
|
|
qemu_iovec_init_buf(&qiov, buf, num);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-06-04 19:15:06 +03:00
|
|
|
ret = bdrv_driver_pwritev(bs, offset, num, &qiov, 0, write_flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
/* Keep bounce buffer around if it is big enough for all
|
|
|
|
* all future requests.
|
|
|
|
*/
|
2016-06-24 01:37:19 +03:00
|
|
|
if (num < max_transfer) {
|
2019-02-18 17:09:11 +03:00
|
|
|
qemu_vfree(buf);
|
|
|
|
buf = NULL;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-06-02 00:10:03 +03:00
|
|
|
offset += num;
|
2017-06-09 13:18:08 +03:00
|
|
|
bytes -= num;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
fail:
|
block: Honor BDRV_REQ_FUA during write_zeroes
The block layer has a couple of cases where it can lose
Force Unit Access semantics when writing a large block of
zeroes, such that the request returns before the zeroes
have been guaranteed to land on underlying media.
SCSI does not support FUA during WRITESAME(10/16); FUA is only
supported if it falls back to WRITE(10/16). But where the
underlying device is new enough to not need a fallback, it
means that any upper layer request with FUA semantics was
silently ignoring BDRV_REQ_FUA.
Conversely, NBD has situations where it can support FUA but not
ZERO_WRITE; when that happens, the generic block layer fallback
to bdrv_driver_pwritev() (or the older bdrv_co_writev() in qemu
2.6) was losing the FUA flag.
The problem of losing flags unrelated to ZERO_WRITE has been
latent in bdrv_co_do_write_zeroes() since commit aa7bfbff, but
back then, it did not matter because there was no FUA flag. It
became observable when commit 93f5e6d8 paved the way for flags
that can impact correctness, when we should have been using
bdrv_co_writev_flags() with modified flags. Compare to commit
9eeb6dd, which got flag manipulation right in
bdrv_co_do_zero_pwritev().
Symptoms: I tested with qemu-io with default writethrough cache
(which is supposed to use FUA semantics on every write), and
targetted an NBD client connected to a server that intentionally
did not advertise NBD_FLAG_SEND_FUA. When doing 'write 0 512',
the NBD client sent two operations (NBD_CMD_WRITE then
NBD_CMD_FLUSH) to get the fallback FUA semantics; but when doing
'write -z 0 512', the NBD client sent only NBD_CMD_WRITE.
The fix is do to a cleanup bdrv_co_flush() at the end of the
operation if any step in the middle relied on a BDS that does
not natively support FUA for that step (note that we don't
need to flush after every operation, if the operation is broken
into chunks based on bounce-buffer sizing). Each BDS gains a
new flag .supported_zero_flags, which parallels the use of
.supported_write_flags but only when accessing a zero write
operation (the flags MUST be different, because of SCSI having
different semantics based on WRITE vs. WRITESAME; and also
because BDRV_REQ_MAY_UNMAP only makes sense on zero writes).
Also fix some documentation to describe -ENOTSUP semantics,
particularly since iscsi depends on those semantics.
Down the road, we may want to add a driver where its
.bdrv_co_pwritev() honors all three of BDRV_REQ_FUA,
BDRV_REQ_ZERO_WRITE, and BDRV_REQ_MAY_UNMAP, and advertise
this via bs->supported_write_flags for blocks opened by that
driver; such a driver should NOT supply .bdrv_co_write_zeroes
nor .supported_zero_flags. But none of the drivers touched
in this patch want to do that (the act of writing zeroes is
different enough from normal writes to deserve a second
callback).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-05-04 01:39:07 +03:00
|
|
|
if (ret == 0 && need_flush) {
|
|
|
|
ret = bdrv_co_flush(bs);
|
|
|
|
}
|
2019-02-18 17:09:11 +03:00
|
|
|
qemu_vfree(buf);
|
2015-04-28 16:27:52 +03:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-07-10 09:31:19 +03:00
|
|
|
static inline int coroutine_fn
|
|
|
|
bdrv_co_write_req_prepare(BdrvChild *child, int64_t offset, uint64_t bytes,
|
|
|
|
BdrvTrackedRequest *req, int flags)
|
|
|
|
{
|
|
|
|
BlockDriverState *bs = child->bs;
|
|
|
|
bool waited;
|
|
|
|
int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
|
|
|
|
|
|
|
|
if (bs->read_only) {
|
|
|
|
return -EPERM;
|
|
|
|
}
|
|
|
|
|
|
|
|
assert(!(bs->open_flags & BDRV_O_INACTIVE));
|
|
|
|
assert((bs->open_flags & BDRV_O_NO_IO) == 0);
|
|
|
|
assert(!(flags & ~BDRV_REQ_MASK));
|
|
|
|
|
|
|
|
if (flags & BDRV_REQ_SERIALISING) {
|
2020-01-08 17:55:55 +03:00
|
|
|
waited = bdrv_mark_request_serialising(req, bdrv_get_cluster_size(bs));
|
|
|
|
/*
|
|
|
|
* For a misaligned request we should have already waited earlier,
|
|
|
|
* because we come after bdrv_padding_rmw_read which must be called
|
|
|
|
* with the request already marked as serialising.
|
|
|
|
*/
|
|
|
|
assert(!waited ||
|
|
|
|
(req->offset == req->overlap_offset &&
|
|
|
|
req->bytes == req->overlap_bytes));
|
|
|
|
} else {
|
|
|
|
bdrv_wait_serialising_requests(req);
|
2018-07-10 09:31:19 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
assert(req->overlap_offset <= offset);
|
|
|
|
assert(offset + bytes <= req->overlap_offset + req->overlap_bytes);
|
2018-07-10 09:31:24 +03:00
|
|
|
assert(end_sector <= bs->total_sectors || child->perm & BLK_PERM_RESIZE);
|
2018-07-10 09:31:19 +03:00
|
|
|
|
2018-07-10 09:31:24 +03:00
|
|
|
switch (req->type) {
|
|
|
|
case BDRV_TRACKED_WRITE:
|
|
|
|
case BDRV_TRACKED_DISCARD:
|
|
|
|
if (flags & BDRV_REQ_WRITE_UNCHANGED) {
|
|
|
|
assert(child->perm & (BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE));
|
|
|
|
} else {
|
|
|
|
assert(child->perm & BLK_PERM_WRITE);
|
|
|
|
}
|
|
|
|
return notifier_with_return_list_notify(&bs->before_write_notifiers,
|
|
|
|
req);
|
|
|
|
case BDRV_TRACKED_TRUNCATE:
|
|
|
|
assert(child->perm & BLK_PERM_RESIZE);
|
|
|
|
return 0;
|
|
|
|
default:
|
|
|
|
abort();
|
2018-07-10 09:31:19 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void coroutine_fn
|
|
|
|
bdrv_co_write_req_finish(BdrvChild *child, int64_t offset, uint64_t bytes,
|
|
|
|
BdrvTrackedRequest *req, int ret)
|
|
|
|
{
|
|
|
|
int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
|
|
|
|
BlockDriverState *bs = child->bs;
|
|
|
|
|
|
|
|
atomic_inc(&bs->write_gen);
|
|
|
|
|
2018-07-10 09:31:21 +03:00
|
|
|
/*
|
|
|
|
* Discard cannot extend the image, but in error handling cases, such as
|
|
|
|
* when reverting a qcow2 cluster allocation, the discarded range can pass
|
|
|
|
* the end of image file, so we cannot assert about BDRV_TRACKED_DISCARD
|
|
|
|
* here. Instead, just skip it, since semantically a discard request
|
|
|
|
* beyond EOF cannot expand the image anyway.
|
|
|
|
*/
|
2018-07-10 09:31:20 +03:00
|
|
|
if (ret == 0 &&
|
2018-07-10 09:31:24 +03:00
|
|
|
(req->type == BDRV_TRACKED_TRUNCATE ||
|
|
|
|
end_sector > bs->total_sectors) &&
|
|
|
|
req->type != BDRV_TRACKED_DISCARD) {
|
2018-07-10 09:31:20 +03:00
|
|
|
bs->total_sectors = end_sector;
|
|
|
|
bdrv_parent_cb_resize(bs);
|
|
|
|
bdrv_dirty_bitmap_truncate(bs, end_sector << BDRV_SECTOR_BITS);
|
2018-07-10 09:31:19 +03:00
|
|
|
}
|
2018-07-10 09:31:21 +03:00
|
|
|
if (req->bytes) {
|
|
|
|
switch (req->type) {
|
|
|
|
case BDRV_TRACKED_WRITE:
|
|
|
|
stat64_max(&bs->wr_highest_offset, offset + bytes);
|
|
|
|
/* fall through, to set dirty bits */
|
|
|
|
case BDRV_TRACKED_DISCARD:
|
|
|
|
bdrv_set_dirty(bs, offset, bytes);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2018-07-10 09:31:19 +03:00
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
* Forwards an already correctly aligned write request to the BlockDriver,
|
|
|
|
* after possibly fragmenting it.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2017-02-09 17:58:43 +03:00
|
|
|
static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child,
|
2015-04-28 16:27:52 +03:00
|
|
|
BdrvTrackedRequest *req, int64_t offset, unsigned int bytes,
|
2019-06-04 19:15:10 +03:00
|
|
|
int64_t align, QEMUIOVector *qiov, size_t qiov_offset, int flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-02-09 17:58:43 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
BlockDriver *drv = bs->drv;
|
|
|
|
int ret;
|
|
|
|
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
uint64_t bytes_remaining = bytes;
|
|
|
|
int max_transfer;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2017-06-28 15:05:10 +03:00
|
|
|
if (bdrv_has_readonly_bitmaps(bs)) {
|
|
|
|
return -EPERM;
|
|
|
|
}
|
|
|
|
|
2016-06-24 01:37:05 +03:00
|
|
|
assert(is_power_of_2(align));
|
|
|
|
assert((offset & (align - 1)) == 0);
|
|
|
|
assert((bytes & (align - 1)) == 0);
|
2019-06-04 19:15:10 +03:00
|
|
|
assert(!qiov || qiov_offset + bytes <= qiov->size);
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
max_transfer = QEMU_ALIGN_DOWN(MIN_NON_ZERO(bs->bl.max_transfer, INT_MAX),
|
|
|
|
align);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2018-07-10 09:31:19 +03:00
|
|
|
ret = bdrv_co_write_req_prepare(child, offset, bytes, req, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
if (!ret && bs->detect_zeroes != BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF &&
|
2016-06-02 00:10:13 +03:00
|
|
|
!(flags & BDRV_REQ_ZERO_WRITE) && drv->bdrv_co_pwrite_zeroes &&
|
2019-06-04 19:15:10 +03:00
|
|
|
qemu_iovec_is_zero(qiov, qiov_offset, bytes)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
flags |= BDRV_REQ_ZERO_WRITE;
|
|
|
|
if (bs->detect_zeroes == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP) {
|
|
|
|
flags |= BDRV_REQ_MAY_UNMAP;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (ret < 0) {
|
|
|
|
/* Do nothing, write notifier decided to fail this request */
|
|
|
|
} else if (flags & BDRV_REQ_ZERO_WRITE) {
|
2015-06-16 15:19:22 +03:00
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_ZERO);
|
2016-06-03 19:42:51 +03:00
|
|
|
ret = bdrv_co_do_pwrite_zeroes(bs, offset, bytes, flags);
|
2016-07-22 11:17:49 +03:00
|
|
|
} else if (flags & BDRV_REQ_WRITE_COMPRESSED) {
|
2019-06-04 19:15:10 +03:00
|
|
|
ret = bdrv_driver_pwritev_compressed(bs, offset, bytes,
|
|
|
|
qiov, qiov_offset);
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
} else if (bytes <= max_transfer) {
|
2015-06-16 15:19:22 +03:00
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV);
|
2019-06-04 19:15:10 +03:00
|
|
|
ret = bdrv_driver_pwritev(bs, offset, bytes, qiov, qiov_offset, flags);
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
} else {
|
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV);
|
|
|
|
while (bytes_remaining) {
|
|
|
|
int num = MIN(bytes_remaining, max_transfer);
|
|
|
|
int local_flags = flags;
|
|
|
|
|
|
|
|
assert(num);
|
|
|
|
if (num < bytes_remaining && (flags & BDRV_REQ_FUA) &&
|
|
|
|
!(bs->supported_write_flags & BDRV_REQ_FUA)) {
|
|
|
|
/* If FUA is going to be emulated by flush, we only
|
|
|
|
* need to flush on the last iteration */
|
|
|
|
local_flags &= ~BDRV_REQ_FUA;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = bdrv_driver_pwritev(bs, offset + bytes - bytes_remaining,
|
2019-06-04 19:15:10 +03:00
|
|
|
num, qiov, bytes - bytes_remaining,
|
|
|
|
local_flags);
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
bytes_remaining -= num;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2015-06-16 15:19:22 +03:00
|
|
|
bdrv_debug_event(bs, BLKDBG_PWRITEV_DONE);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
if (ret >= 0) {
|
block: Fragment writes to max transfer length
Drivers should be able to rely on the block layer honoring the
max transfer length, rather than needing to return -EINVAL
(iscsi) or manually fragment things (nbd). We already fragment
write zeroes at the block layer; this patch adds the fragmentation
for normal writes, after requests have been aligned (fragmenting
before alignment would lead to multiple unaligned requests, rather
than just the head and tail).
When fragmenting a large request where FUA was requested, but
where we know that FUA is implemented by flushing all requests
rather than the given request, then we can still get by with
only one flush. Note, however, that we need a followup patch
to the raw format driver to avoid a regression in the number of
flushes actually issued.
The return value was previously nebulous on success (sometimes
zero, sometimes the length written); since we never have a short
write, and since fragmenting may store yet another positive
value in 'ret', change the function to always return 0 on success,
matching what we do in bdrv_aligned_preadv().
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1468607524-19021-4-git-send-email-eblake@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2016-07-15 21:32:01 +03:00
|
|
|
ret = 0;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2018-07-10 09:31:19 +03:00
|
|
|
bdrv_co_write_req_finish(child, offset, bytes, req, ret);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2017-02-09 17:58:43 +03:00
|
|
|
static int coroutine_fn bdrv_co_do_zero_pwritev(BdrvChild *child,
|
2015-05-13 16:12:00 +03:00
|
|
|
int64_t offset,
|
|
|
|
unsigned int bytes,
|
|
|
|
BdrvRequestFlags flags,
|
|
|
|
BdrvTrackedRequest *req)
|
|
|
|
{
|
2017-02-09 17:58:43 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-05-13 16:12:00 +03:00
|
|
|
QEMUIOVector local_qiov;
|
2016-06-24 01:37:24 +03:00
|
|
|
uint64_t align = bs->bl.request_alignment;
|
2015-05-13 16:12:00 +03:00
|
|
|
int ret = 0;
|
2019-06-04 19:15:05 +03:00
|
|
|
bool padding;
|
|
|
|
BdrvRequestPadding pad;
|
2015-05-13 16:12:00 +03:00
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
padding = bdrv_init_padding(bs, offset, bytes, &pad);
|
|
|
|
if (padding) {
|
2019-11-01 18:25:08 +03:00
|
|
|
bdrv_mark_request_serialising(req, align);
|
2015-05-13 16:12:00 +03:00
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
bdrv_padding_rmw_read(child, req, &pad, true);
|
|
|
|
|
|
|
|
if (pad.head || pad.merge_reads) {
|
|
|
|
int64_t aligned_offset = offset & ~(align - 1);
|
|
|
|
int64_t write_bytes = pad.merge_reads ? pad.buf_len : align;
|
|
|
|
|
|
|
|
qemu_iovec_init_buf(&local_qiov, pad.buf, write_bytes);
|
|
|
|
ret = bdrv_aligned_pwritev(child, req, aligned_offset, write_bytes,
|
2019-06-04 19:15:10 +03:00
|
|
|
align, &local_qiov, 0,
|
2019-06-04 19:15:05 +03:00
|
|
|
flags & ~BDRV_REQ_ZERO_WRITE);
|
|
|
|
if (ret < 0 || pad.merge_reads) {
|
|
|
|
/* Error or all work is done */
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
offset += write_bytes - pad.head;
|
|
|
|
bytes -= write_bytes - pad.head;
|
2015-05-13 16:12:00 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
assert(!bytes || (offset & (align - 1)) == 0);
|
|
|
|
if (bytes >= align) {
|
|
|
|
/* Write the aligned part in the middle. */
|
|
|
|
uint64_t aligned_bytes = bytes & ~(align - 1);
|
2017-02-09 17:58:43 +03:00
|
|
|
ret = bdrv_aligned_pwritev(child, req, offset, aligned_bytes, align,
|
2019-06-04 19:15:10 +03:00
|
|
|
NULL, 0, flags);
|
2015-05-13 16:12:00 +03:00
|
|
|
if (ret < 0) {
|
2019-06-04 19:15:05 +03:00
|
|
|
goto out;
|
2015-05-13 16:12:00 +03:00
|
|
|
}
|
|
|
|
bytes -= aligned_bytes;
|
|
|
|
offset += aligned_bytes;
|
|
|
|
}
|
|
|
|
|
|
|
|
assert(!bytes || (offset & (align - 1)) == 0);
|
|
|
|
if (bytes) {
|
2019-06-04 19:15:05 +03:00
|
|
|
assert(align == pad.tail + bytes);
|
2015-05-13 16:12:00 +03:00
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
qemu_iovec_init_buf(&local_qiov, pad.tail_buf, align);
|
2017-02-09 17:58:43 +03:00
|
|
|
ret = bdrv_aligned_pwritev(child, req, offset, align, align,
|
2019-06-04 19:15:10 +03:00
|
|
|
&local_qiov, 0,
|
|
|
|
flags & ~BDRV_REQ_ZERO_WRITE);
|
2015-05-13 16:12:00 +03:00
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
out:
|
|
|
|
bdrv_padding_destroy(&pad);
|
|
|
|
|
|
|
|
return ret;
|
2015-05-13 16:12:00 +03:00
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
|
|
|
* Handle a write request in coroutine context
|
|
|
|
*/
|
2016-06-20 22:31:46 +03:00
|
|
|
int coroutine_fn bdrv_co_pwritev(BdrvChild *child,
|
2015-04-28 16:27:52 +03:00
|
|
|
int64_t offset, unsigned int bytes, QEMUIOVector *qiov,
|
|
|
|
BdrvRequestFlags flags)
|
2019-06-04 19:15:11 +03:00
|
|
|
{
|
|
|
|
return bdrv_co_pwritev_part(child, offset, bytes, qiov, 0, flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child,
|
|
|
|
int64_t offset, unsigned int bytes, QEMUIOVector *qiov, size_t qiov_offset,
|
|
|
|
BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2016-06-20 22:31:46 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
BdrvTrackedRequest req;
|
2016-06-24 01:37:24 +03:00
|
|
|
uint64_t align = bs->bl.request_alignment;
|
2019-06-04 19:15:05 +03:00
|
|
|
BdrvRequestPadding pad;
|
2015-04-28 16:27:52 +03:00
|
|
|
int ret;
|
|
|
|
|
2017-08-04 13:50:36 +03:00
|
|
|
trace_bdrv_co_pwritev(child->bs, offset, bytes, flags);
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
if (!bs->drv) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = bdrv_check_byte_request(bs, offset, bytes);
|
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2019-10-14 11:15:45 +03:00
|
|
|
/* If the request is misaligned then we can't make it efficient */
|
|
|
|
if ((flags & BDRV_REQ_NO_FALLBACK) &&
|
|
|
|
!QEMU_IS_ALIGNED(offset | bytes, align))
|
|
|
|
{
|
|
|
|
return -ENOTSUP;
|
|
|
|
}
|
|
|
|
|
2020-02-06 19:42:45 +03:00
|
|
|
if (bytes == 0 && !QEMU_IS_ALIGNED(offset, bs->bl.request_alignment)) {
|
|
|
|
/*
|
|
|
|
* Aligning zero request is nonsense. Even if driver has special meaning
|
|
|
|
* of zero-length (like qcow2_co_pwritev_compressed_part), we can't pass
|
|
|
|
* it to driver due to request_alignment.
|
|
|
|
*
|
|
|
|
* Still, no reason to return an error if someone do unaligned
|
|
|
|
* zero-length write occasionally.
|
|
|
|
*/
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
|
|
|
* Align write if necessary by performing a read-modify-write cycle.
|
|
|
|
* Pad qiov with the read parts and be sure to have a tracked request not
|
|
|
|
* only for bdrv_aligned_pwritev, but also for the reads of the RMW cycle.
|
|
|
|
*/
|
2015-11-09 13:16:46 +03:00
|
|
|
tracked_request_begin(&req, bs, offset, bytes, BDRV_TRACKED_WRITE);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2018-02-14 19:09:20 +03:00
|
|
|
if (flags & BDRV_REQ_ZERO_WRITE) {
|
2017-02-09 17:58:43 +03:00
|
|
|
ret = bdrv_co_do_zero_pwritev(child, offset, bytes, flags, &req);
|
2015-05-13 16:12:00 +03:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2019-06-04 19:15:11 +03:00
|
|
|
if (bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, &pad)) {
|
2019-11-01 18:25:08 +03:00
|
|
|
bdrv_mark_request_serialising(&req, align);
|
2019-06-04 19:15:05 +03:00
|
|
|
bdrv_padding_rmw_read(child, &req, &pad, false);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-02-09 17:58:43 +03:00
|
|
|
ret = bdrv_aligned_pwritev(child, &req, offset, bytes, align,
|
2019-06-04 19:15:11 +03:00
|
|
|
qiov, qiov_offset, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-06-04 19:15:05 +03:00
|
|
|
bdrv_padding_destroy(&pad);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2015-05-13 16:12:00 +03:00
|
|
|
out:
|
|
|
|
tracked_request_end(&req);
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2019-06-04 19:15:05 +03:00
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-06-20 22:31:46 +03:00
|
|
|
int coroutine_fn bdrv_co_pwrite_zeroes(BdrvChild *child, int64_t offset,
|
2017-06-09 13:18:08 +03:00
|
|
|
int bytes, BdrvRequestFlags flags)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-06-09 13:18:08 +03:00
|
|
|
trace_bdrv_co_pwrite_zeroes(child->bs, offset, bytes, flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-20 22:31:46 +03:00
|
|
|
if (!(child->bs->open_flags & BDRV_O_UNMAP)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
flags &= ~BDRV_REQ_MAY_UNMAP;
|
|
|
|
}
|
|
|
|
|
2017-06-09 13:18:08 +03:00
|
|
|
return bdrv_co_pwritev(child, offset, bytes, NULL,
|
2016-06-02 00:10:04 +03:00
|
|
|
BDRV_REQ_ZERO_WRITE | flags);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-09-23 04:45:50 +03:00
|
|
|
/*
|
|
|
|
* Flush ALL BDSes regardless of if they are reachable via a BlkBackend or not.
|
|
|
|
*/
|
|
|
|
int bdrv_flush_all(void)
|
|
|
|
{
|
|
|
|
BdrvNextIterator it;
|
|
|
|
BlockDriverState *bs = NULL;
|
|
|
|
int result = 0;
|
|
|
|
|
2019-09-17 14:58:08 +03:00
|
|
|
/*
|
|
|
|
* bdrv queue is managed by record/replay,
|
|
|
|
* creating new flush request for stopping
|
|
|
|
* the VM may break the determinism
|
|
|
|
*/
|
|
|
|
if (replay_events_enabled()) {
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2016-09-23 04:45:50 +03:00
|
|
|
for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
|
|
|
|
AioContext *aio_context = bdrv_get_aio_context(bs);
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
aio_context_acquire(aio_context);
|
|
|
|
ret = bdrv_flush(bs);
|
|
|
|
if (ret < 0 && !result) {
|
|
|
|
result = ret;
|
|
|
|
}
|
|
|
|
aio_context_release(aio_context);
|
|
|
|
}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2017-10-12 06:47:05 +03:00
|
|
|
typedef struct BdrvCoBlockStatusData {
|
2015-04-28 16:27:52 +03:00
|
|
|
BlockDriverState *bs;
|
|
|
|
BlockDriverState *base;
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
bool want_zero;
|
2017-10-12 06:47:05 +03:00
|
|
|
int64_t offset;
|
|
|
|
int64_t bytes;
|
|
|
|
int64_t *pnum;
|
|
|
|
int64_t *map;
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
BlockDriverState **file;
|
2017-10-12 06:47:05 +03:00
|
|
|
int ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
bool done;
|
2017-10-12 06:47:05 +03:00
|
|
|
} BdrvCoBlockStatusData;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2018-02-13 23:26:43 +03:00
|
|
|
int coroutine_fn bdrv_co_block_status_from_file(BlockDriverState *bs,
|
|
|
|
bool want_zero,
|
|
|
|
int64_t offset,
|
|
|
|
int64_t bytes,
|
|
|
|
int64_t *pnum,
|
|
|
|
int64_t *map,
|
|
|
|
BlockDriverState **file)
|
2017-07-13 18:30:28 +03:00
|
|
|
{
|
|
|
|
assert(bs->file && bs->file->bs);
|
2018-02-13 23:26:43 +03:00
|
|
|
*pnum = bytes;
|
|
|
|
*map = offset;
|
2017-07-13 18:30:28 +03:00
|
|
|
*file = bs->file->bs;
|
2018-02-13 23:26:43 +03:00
|
|
|
return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID;
|
2017-07-13 18:30:28 +03:00
|
|
|
}
|
|
|
|
|
2018-02-13 23:26:43 +03:00
|
|
|
int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
|
|
|
|
bool want_zero,
|
|
|
|
int64_t offset,
|
|
|
|
int64_t bytes,
|
|
|
|
int64_t *pnum,
|
|
|
|
int64_t *map,
|
|
|
|
BlockDriverState **file)
|
2017-07-13 18:30:28 +03:00
|
|
|
{
|
|
|
|
assert(bs->backing && bs->backing->bs);
|
2018-02-13 23:26:43 +03:00
|
|
|
*pnum = bytes;
|
|
|
|
*map = offset;
|
2017-07-13 18:30:28 +03:00
|
|
|
*file = bs->backing->bs;
|
2018-02-13 23:26:43 +03:00
|
|
|
return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID;
|
2017-07-13 18:30:28 +03:00
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/*
|
|
|
|
* Returns the allocation status of the specified sectors.
|
|
|
|
* Drivers not implementing the functionality are assumed to not support
|
|
|
|
* backing files, hence all their sectors are reported as allocated.
|
|
|
|
*
|
block: Add .bdrv_co_block_status() callback
We are gradually moving away from sector-based interfaces, towards
byte-based. Now that the block layer exposes byte-based allocation,
it's time to tackle the drivers. Add a new callback that operates
on as small as byte boundaries. Subsequent patches will then update
individual drivers, then finally remove .bdrv_co_get_block_status().
The new code also passes through the 'want_zero' hint, which will
allow subsequent patches to further optimize callers that only care
about how much of the image is allocated (want_zero is false),
rather than full details about runs of zeroes and which offsets the
allocation actually maps to (want_zero is true). As part of this
effort, fix another part of the documentation: the claim in commit
4c41cb4 that BDRV_BLOCK_ALLOCATED is short for 'DATA || ZERO' is a
lie at the block layer (see commit e88ae2264), even though it is
how the bit is computed from the driver layer. After all, there
are intentionally cases where we return ZERO but not ALLOCATED at
the block layer, when we know that a read sees zero because the
backing file is too short. Note that the driver interface is thus
slightly different than the public interface with regards to which
bits will be set, and what guarantees are provided on input.
We also add an assertion that any driver using the new callback will
make progress (the only time pnum will be 0 is if the block layer
already handled an out-of-bounds request, or if there is an error);
the old driver interface did not provide this guarantee, which
could lead to some inf-loops in drastic corner-case failures.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2018-02-13 23:26:41 +03:00
|
|
|
* If 'want_zero' is true, the caller is querying for mapping
|
|
|
|
* purposes, with a focus on valid BDRV_BLOCK_OFFSET_VALID, _DATA, and
|
|
|
|
* _ZERO where possible; otherwise, the result favors larger 'pnum',
|
|
|
|
* with a focus on accurate BDRV_BLOCK_ALLOCATED.
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
*
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
* If 'offset' is beyond the end of the disk image the return value is
|
2017-05-05 05:14:59 +03:00
|
|
|
* BDRV_BLOCK_EOF and 'pnum' is set to 0.
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
* 'bytes' is the max value 'pnum' should be set to. If bytes goes
|
2017-05-05 05:14:59 +03:00
|
|
|
* beyond the end of the disk image it will be clamped; if 'pnum' is set to
|
|
|
|
* the end of the image, then the returned value will include BDRV_BLOCK_EOF.
|
2016-01-26 06:58:48 +03:00
|
|
|
*
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
* 'pnum' is set to the number of bytes (including and immediately
|
|
|
|
* following the specified offset) that are easily known to be in the
|
|
|
|
* same allocated/unallocated state. Note that a second call starting
|
|
|
|
* at the original offset plus returned pnum may have the same status.
|
|
|
|
* The returned value is non-zero on success except at end-of-file.
|
|
|
|
*
|
|
|
|
* Returns negative errno on failure. Otherwise, if the
|
|
|
|
* BDRV_BLOCK_OFFSET_VALID bit is set, 'map' and 'file' (if non-NULL) are
|
|
|
|
* set to the host mapping and BDS corresponding to the guest offset.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
|
|
|
|
bool want_zero,
|
|
|
|
int64_t offset, int64_t bytes,
|
|
|
|
int64_t *pnum, int64_t *map,
|
|
|
|
BlockDriverState **file)
|
|
|
|
{
|
|
|
|
int64_t total_size;
|
|
|
|
int64_t n; /* bytes */
|
2017-10-12 06:47:17 +03:00
|
|
|
int ret;
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
int64_t local_map = 0;
|
2017-10-12 06:46:57 +03:00
|
|
|
BlockDriverState *local_file = NULL;
|
2017-10-12 06:47:17 +03:00
|
|
|
int64_t aligned_offset, aligned_bytes;
|
|
|
|
uint32_t align;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-10-12 06:46:57 +03:00
|
|
|
assert(pnum);
|
|
|
|
*pnum = 0;
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
total_size = bdrv_getlength(bs);
|
|
|
|
if (total_size < 0) {
|
|
|
|
ret = total_size;
|
2017-10-12 06:46:57 +03:00
|
|
|
goto early_out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (offset >= total_size) {
|
2017-10-12 06:46:57 +03:00
|
|
|
ret = BDRV_BLOCK_EOF;
|
|
|
|
goto early_out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (!bytes) {
|
2017-10-12 06:46:57 +03:00
|
|
|
ret = 0;
|
|
|
|
goto early_out;
|
2017-10-05 22:02:44 +03:00
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
n = total_size - offset;
|
|
|
|
if (n < bytes) {
|
|
|
|
bytes = n;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
/* Must be non-NULL or bdrv_getlength() would have failed */
|
|
|
|
assert(bs->drv);
|
2018-02-13 23:27:01 +03:00
|
|
|
if (!bs->drv->bdrv_co_block_status) {
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
*pnum = bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
ret = BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED;
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (offset + bytes == total_size) {
|
2017-05-05 05:14:59 +03:00
|
|
|
ret |= BDRV_BLOCK_EOF;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
if (bs->drv->protocol_name) {
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
ret |= BDRV_BLOCK_OFFSET_VALID;
|
|
|
|
local_map = offset;
|
2017-10-12 06:46:57 +03:00
|
|
|
local_file = bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2017-10-12 06:46:57 +03:00
|
|
|
goto early_out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
2017-10-12 06:47:17 +03:00
|
|
|
|
|
|
|
/* Round out to request_alignment boundaries */
|
block: Add .bdrv_co_block_status() callback
We are gradually moving away from sector-based interfaces, towards
byte-based. Now that the block layer exposes byte-based allocation,
it's time to tackle the drivers. Add a new callback that operates
on as small as byte boundaries. Subsequent patches will then update
individual drivers, then finally remove .bdrv_co_get_block_status().
The new code also passes through the 'want_zero' hint, which will
allow subsequent patches to further optimize callers that only care
about how much of the image is allocated (want_zero is false),
rather than full details about runs of zeroes and which offsets the
allocation actually maps to (want_zero is true). As part of this
effort, fix another part of the documentation: the claim in commit
4c41cb4 that BDRV_BLOCK_ALLOCATED is short for 'DATA || ZERO' is a
lie at the block layer (see commit e88ae2264), even though it is
how the bit is computed from the driver layer. After all, there
are intentionally cases where we return ZERO but not ALLOCATED at
the block layer, when we know that a read sees zero because the
backing file is too short. Note that the driver interface is thus
slightly different than the public interface with regards to which
bits will be set, and what guarantees are provided on input.
We also add an assertion that any driver using the new callback will
make progress (the only time pnum will be 0 is if the block layer
already handled an out-of-bounds request, or if there is an error);
the old driver interface did not provide this guarantee, which
could lead to some inf-loops in drastic corner-case failures.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2018-02-13 23:26:41 +03:00
|
|
|
align = bs->bl.request_alignment;
|
2017-10-12 06:47:17 +03:00
|
|
|
aligned_offset = QEMU_ALIGN_DOWN(offset, align);
|
|
|
|
aligned_bytes = ROUND_UP(offset + bytes, align) - aligned_offset;
|
|
|
|
|
2018-02-13 23:27:01 +03:00
|
|
|
ret = bs->drv->bdrv_co_block_status(bs, want_zero, aligned_offset,
|
|
|
|
aligned_bytes, pnum, &local_map,
|
|
|
|
&local_file);
|
|
|
|
if (ret < 0) {
|
|
|
|
*pnum = 0;
|
|
|
|
goto out;
|
2017-10-12 06:47:17 +03:00
|
|
|
}
|
|
|
|
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
/*
|
2018-02-13 23:27:01 +03:00
|
|
|
* The driver's result must be a non-zero multiple of request_alignment.
|
2017-10-12 06:47:17 +03:00
|
|
|
* Clamp pnum and adjust map to original request.
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
*/
|
2018-02-13 23:27:01 +03:00
|
|
|
assert(*pnum && QEMU_IS_ALIGNED(*pnum, align) &&
|
|
|
|
align > offset - aligned_offset);
|
block: avoid recursive block_status call if possible
drv_co_block_status digs bs->file for additional, more accurate search
for hole inside region, reported as DATA by bs since 5daa74a6ebc.
This accuracy is not free: assume we have qcow2 disk. Actually, qcow2
knows, where are holes and where is data. But every block_status
request calls lseek additionally. Assume a big disk, full of
data, in any iterative copying block job (or img convert) we'll call
lseek(HOLE) on every iteration, and each of these lseeks will have to
iterate through all metadata up to the end of file. It's obviously
ineffective behavior. And for many scenarios we don't need this lseek
at all.
However, lseek is needed when we have metadata-preallocated image.
So, let's detect metadata-preallocation case and don't dig qcow2's
protocol file in other cases.
The idea is to compare allocation size in POV of filesystem with
allocations size in POV of Qcow2 (by refcounts). If allocation in fs is
significantly lower, consider it as metadata-preallocation case.
102 iotest changed, as our detector can't detect shrinked file as
metadata-preallocation, which don't seem to be wrong, as with metadata
preallocation we always have valid file length.
Two other iotests have a slight change in their QMP output sequence:
Active 'block-commit' returns earlier because the job coroutine yields
earlier on a blocking operation. This operation is loading the refcount
blocks in qcow2_detect_metadata_preallocation().
Suggested-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-08 19:26:17 +03:00
|
|
|
if (ret & BDRV_BLOCK_RECURSE) {
|
|
|
|
assert(ret & BDRV_BLOCK_DATA);
|
|
|
|
assert(ret & BDRV_BLOCK_OFFSET_VALID);
|
|
|
|
assert(!(ret & BDRV_BLOCK_ZERO));
|
|
|
|
}
|
|
|
|
|
2017-10-12 06:47:17 +03:00
|
|
|
*pnum -= offset - aligned_offset;
|
|
|
|
if (*pnum > bytes) {
|
|
|
|
*pnum = bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (ret & BDRV_BLOCK_OFFSET_VALID) {
|
2017-10-12 06:47:17 +03:00
|
|
|
local_map += offset - aligned_offset;
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
if (ret & BDRV_BLOCK_RAW) {
|
2017-10-12 06:46:57 +03:00
|
|
|
assert(ret & BDRV_BLOCK_OFFSET_VALID && local_file);
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
ret = bdrv_co_block_status(local_file, want_zero, local_map,
|
|
|
|
*pnum, pnum, &local_map, &local_file);
|
2016-10-27 13:48:52 +03:00
|
|
|
goto out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
|
|
|
|
ret |= BDRV_BLOCK_ALLOCATED;
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
} else if (want_zero) {
|
2015-04-28 16:27:52 +03:00
|
|
|
if (bdrv_unallocated_blocks_are_zero(bs)) {
|
|
|
|
ret |= BDRV_BLOCK_ZERO;
|
2015-06-17 15:55:21 +03:00
|
|
|
} else if (bs->backing) {
|
|
|
|
BlockDriverState *bs2 = bs->backing->bs;
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
int64_t size2 = bdrv_getlength(bs2);
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (size2 >= 0 && offset >= size2) {
|
2015-04-28 16:27:52 +03:00
|
|
|
ret |= BDRV_BLOCK_ZERO;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
block: avoid recursive block_status call if possible
drv_co_block_status digs bs->file for additional, more accurate search
for hole inside region, reported as DATA by bs since 5daa74a6ebc.
This accuracy is not free: assume we have qcow2 disk. Actually, qcow2
knows, where are holes and where is data. But every block_status
request calls lseek additionally. Assume a big disk, full of
data, in any iterative copying block job (or img convert) we'll call
lseek(HOLE) on every iteration, and each of these lseeks will have to
iterate through all metadata up to the end of file. It's obviously
ineffective behavior. And for many scenarios we don't need this lseek
at all.
However, lseek is needed when we have metadata-preallocated image.
So, let's detect metadata-preallocation case and don't dig qcow2's
protocol file in other cases.
The idea is to compare allocation size in POV of filesystem with
allocations size in POV of Qcow2 (by refcounts). If allocation in fs is
significantly lower, consider it as metadata-preallocation case.
102 iotest changed, as our detector can't detect shrinked file as
metadata-preallocation, which don't seem to be wrong, as with metadata
preallocation we always have valid file length.
Two other iotests have a slight change in their QMP output sequence:
Active 'block-commit' returns earlier because the job coroutine yields
earlier on a blocking operation. This operation is loading the refcount
blocks in qcow2_detect_metadata_preallocation().
Suggested-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2019-04-08 19:26:17 +03:00
|
|
|
if (want_zero && ret & BDRV_BLOCK_RECURSE &&
|
|
|
|
local_file && local_file != bs &&
|
2015-04-28 16:27:52 +03:00
|
|
|
(ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) &&
|
|
|
|
(ret & BDRV_BLOCK_OFFSET_VALID)) {
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
int64_t file_pnum;
|
|
|
|
int ret2;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
ret2 = bdrv_co_block_status(local_file, want_zero, local_map,
|
|
|
|
*pnum, &file_pnum, NULL, NULL);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret2 >= 0) {
|
|
|
|
/* Ignore errors. This is just providing extra information, it
|
|
|
|
* is useful but not necessary.
|
|
|
|
*/
|
block: Exploit BDRV_BLOCK_EOF for larger zero blocks
When we have a BDS with unallocated clusters, but asking the status
of its underlying bs->file or backing layer encounters an end-of-file
condition, we know that the rest of the unallocated area will read as
zeroes. However, pre-patch, this required two separate calls to
bdrv_get_block_status(), as the first call stops at the point where
the underlying file ends. Thanks to BDRV_BLOCK_EOF, we can now widen
the results of the primary status if the secondary status already
includes BDRV_BLOCK_ZERO.
In turn, this fixes a TODO mentioned in iotest 154, where we can now
see that all sectors in a partial cluster at the end of a file read
as zero when coupling the shorter backing file's status along with our
knowledge that the remaining sectors came from an unallocated cluster.
Also, note that the loop in bdrv_co_get_block_status_above() had an
inefficent exit: in cases where the active layer sets BDRV_BLOCK_ZERO
but does NOT set BDRV_BLOCK_ALLOCATED (namely, where we know we read
zeroes merely because our unallocated clusters lie beyond the backing
file's shorter length), we still ended up probing the backing layer
even though we already had a good answer.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20170505021500.19315-3-eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-05-05 05:15:00 +03:00
|
|
|
if (ret2 & BDRV_BLOCK_EOF &&
|
|
|
|
(!file_pnum || ret2 & BDRV_BLOCK_ZERO)) {
|
|
|
|
/*
|
|
|
|
* It is valid for the format block driver to read
|
|
|
|
* beyond the end of the underlying file's current
|
|
|
|
* size; such areas read as zero.
|
|
|
|
*/
|
2015-04-28 16:27:52 +03:00
|
|
|
ret |= BDRV_BLOCK_ZERO;
|
|
|
|
} else {
|
|
|
|
/* Limit request to the range reported by the protocol driver */
|
|
|
|
*pnum = file_pnum;
|
|
|
|
ret |= (ret2 & BDRV_BLOCK_ZERO);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
out:
|
|
|
|
bdrv_dec_in_flight(bs);
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (ret >= 0 && offset + *pnum == total_size) {
|
2017-05-05 05:14:59 +03:00
|
|
|
ret |= BDRV_BLOCK_EOF;
|
|
|
|
}
|
2017-10-12 06:46:57 +03:00
|
|
|
early_out:
|
|
|
|
if (file) {
|
|
|
|
*file = local_file;
|
|
|
|
}
|
block: Switch bdrv_co_get_block_status() to byte-based
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Convert another internal
function (no semantic change); and as with its public counterpart,
rename to bdrv_co_block_status() and split the offset return, to
make the compiler enforce that we catch all uses. For now, we
assert that callers and the return value still use aligned data,
but ultimately, this will be the function where we hand off to a
byte-based driver callback, and will eventually need to add logic
to ensure we round calls according to the driver's
request_alignment then touch up the result handed back to the
caller, to start permitting a caller to pass unaligned offsets.
Note that we are now prepared to accepts 'bytes' larger than INT_MAX;
this is okay as long as we clamp things internally before violating
any 32-bit limits, and makes no difference to how a client will
use the information (clients looping over the entire file must
already be prepared for consecutive calls to return the same status,
as drivers are already free to return shorter-than-maximal status
due to any other convenient split points, such as when the L2 table
crosses cluster boundaries in qcow2).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:04 +03:00
|
|
|
if (map) {
|
|
|
|
*map = local_map;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2017-10-12 06:47:07 +03:00
|
|
|
static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,
|
|
|
|
BlockDriverState *base,
|
|
|
|
bool want_zero,
|
|
|
|
int64_t offset,
|
|
|
|
int64_t bytes,
|
|
|
|
int64_t *pnum,
|
|
|
|
int64_t *map,
|
|
|
|
BlockDriverState **file)
|
2015-06-08 08:56:07 +03:00
|
|
|
{
|
|
|
|
BlockDriverState *p;
|
2017-10-12 06:47:07 +03:00
|
|
|
int ret = 0;
|
block: Exploit BDRV_BLOCK_EOF for larger zero blocks
When we have a BDS with unallocated clusters, but asking the status
of its underlying bs->file or backing layer encounters an end-of-file
condition, we know that the rest of the unallocated area will read as
zeroes. However, pre-patch, this required two separate calls to
bdrv_get_block_status(), as the first call stops at the point where
the underlying file ends. Thanks to BDRV_BLOCK_EOF, we can now widen
the results of the primary status if the secondary status already
includes BDRV_BLOCK_ZERO.
In turn, this fixes a TODO mentioned in iotest 154, where we can now
see that all sectors in a partial cluster at the end of a file read
as zero when coupling the shorter backing file's status along with our
knowledge that the remaining sectors came from an unallocated cluster.
Also, note that the loop in bdrv_co_get_block_status_above() had an
inefficent exit: in cases where the active layer sets BDRV_BLOCK_ZERO
but does NOT set BDRV_BLOCK_ALLOCATED (namely, where we know we read
zeroes merely because our unallocated clusters lie beyond the backing
file's shorter length), we still ended up probing the backing layer
even though we already had a good answer.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20170505021500.19315-3-eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-05-05 05:15:00 +03:00
|
|
|
bool first = true;
|
2015-06-08 08:56:07 +03:00
|
|
|
|
|
|
|
assert(bs != base);
|
2015-06-17 15:55:21 +03:00
|
|
|
for (p = bs; p != base; p = backing_bs(p)) {
|
2017-10-12 06:47:07 +03:00
|
|
|
ret = bdrv_co_block_status(p, want_zero, offset, bytes, pnum, map,
|
|
|
|
file);
|
block: Exploit BDRV_BLOCK_EOF for larger zero blocks
When we have a BDS with unallocated clusters, but asking the status
of its underlying bs->file or backing layer encounters an end-of-file
condition, we know that the rest of the unallocated area will read as
zeroes. However, pre-patch, this required two separate calls to
bdrv_get_block_status(), as the first call stops at the point where
the underlying file ends. Thanks to BDRV_BLOCK_EOF, we can now widen
the results of the primary status if the secondary status already
includes BDRV_BLOCK_ZERO.
In turn, this fixes a TODO mentioned in iotest 154, where we can now
see that all sectors in a partial cluster at the end of a file read
as zero when coupling the shorter backing file's status along with our
knowledge that the remaining sectors came from an unallocated cluster.
Also, note that the loop in bdrv_co_get_block_status_above() had an
inefficent exit: in cases where the active layer sets BDRV_BLOCK_ZERO
but does NOT set BDRV_BLOCK_ALLOCATED (namely, where we know we read
zeroes merely because our unallocated clusters lie beyond the backing
file's shorter length), we still ended up probing the backing layer
even though we already had a good answer.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20170505021500.19315-3-eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-05-05 05:15:00 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (ret & BDRV_BLOCK_ZERO && ret & BDRV_BLOCK_EOF && !first) {
|
|
|
|
/*
|
|
|
|
* Reading beyond the end of the file continues to read
|
|
|
|
* zeroes, but we can only widen the result to the
|
|
|
|
* unallocated length we learned from an earlier
|
|
|
|
* iteration.
|
|
|
|
*/
|
2017-10-12 06:47:07 +03:00
|
|
|
*pnum = bytes;
|
block: Exploit BDRV_BLOCK_EOF for larger zero blocks
When we have a BDS with unallocated clusters, but asking the status
of its underlying bs->file or backing layer encounters an end-of-file
condition, we know that the rest of the unallocated area will read as
zeroes. However, pre-patch, this required two separate calls to
bdrv_get_block_status(), as the first call stops at the point where
the underlying file ends. Thanks to BDRV_BLOCK_EOF, we can now widen
the results of the primary status if the secondary status already
includes BDRV_BLOCK_ZERO.
In turn, this fixes a TODO mentioned in iotest 154, where we can now
see that all sectors in a partial cluster at the end of a file read
as zero when coupling the shorter backing file's status along with our
knowledge that the remaining sectors came from an unallocated cluster.
Also, note that the loop in bdrv_co_get_block_status_above() had an
inefficent exit: in cases where the active layer sets BDRV_BLOCK_ZERO
but does NOT set BDRV_BLOCK_ALLOCATED (namely, where we know we read
zeroes merely because our unallocated clusters lie beyond the backing
file's shorter length), we still ended up probing the backing layer
even though we already had a good answer.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20170505021500.19315-3-eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-05-05 05:15:00 +03:00
|
|
|
}
|
|
|
|
if (ret & (BDRV_BLOCK_ZERO | BDRV_BLOCK_DATA)) {
|
2015-06-08 08:56:07 +03:00
|
|
|
break;
|
|
|
|
}
|
2017-10-12 06:47:07 +03:00
|
|
|
/* [offset, pnum] unallocated on this layer, which could be only
|
|
|
|
* the first part of [offset, bytes]. */
|
|
|
|
bytes = MIN(bytes, *pnum);
|
block: Exploit BDRV_BLOCK_EOF for larger zero blocks
When we have a BDS with unallocated clusters, but asking the status
of its underlying bs->file or backing layer encounters an end-of-file
condition, we know that the rest of the unallocated area will read as
zeroes. However, pre-patch, this required two separate calls to
bdrv_get_block_status(), as the first call stops at the point where
the underlying file ends. Thanks to BDRV_BLOCK_EOF, we can now widen
the results of the primary status if the secondary status already
includes BDRV_BLOCK_ZERO.
In turn, this fixes a TODO mentioned in iotest 154, where we can now
see that all sectors in a partial cluster at the end of a file read
as zero when coupling the shorter backing file's status along with our
knowledge that the remaining sectors came from an unallocated cluster.
Also, note that the loop in bdrv_co_get_block_status_above() had an
inefficent exit: in cases where the active layer sets BDRV_BLOCK_ZERO
but does NOT set BDRV_BLOCK_ALLOCATED (namely, where we know we read
zeroes merely because our unallocated clusters lie beyond the backing
file's shorter length), we still ended up probing the backing layer
even though we already had a good answer.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20170505021500.19315-3-eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-05-05 05:15:00 +03:00
|
|
|
first = false;
|
2015-06-08 08:56:07 +03:00
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
block: Convert bdrv_get_block_status_above() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status_above()
to bdrv_block_status_above() ensures that the compiler enforces that
all callers are updated. Likewise, since it a byte interface allows
an offset mapping that might not be sector aligned, split the mapping
out of the return value and into a pass-by-reference parameter. For
now, the io.c layer still assert()s that all uses are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status in the drivers.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), plus
updates for the new split return interface. But some code,
particularly bdrv_block_status(), gets a lot simpler because it no
longer has to mess with sectors. Likewise, mirror code no longer
computes s->granularity >> BDRV_SECTOR_BITS, and can therefore drop
an assertion about alignment because the loop no longer depends on
alignment (never mind that we don't really have a driver that
reports sub-sector alignments, so it's not really possible to test
the effect of sub-sector mirroring). Fix a neighboring assertion to
use is_power_of_2 while there.
For ease of review, bdrv_get_block_status() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:08 +03:00
|
|
|
/* Coroutine wrapper for bdrv_block_status_above() */
|
2017-10-12 06:47:07 +03:00
|
|
|
static void coroutine_fn bdrv_block_status_above_co_entry(void *opaque)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-10-12 06:47:05 +03:00
|
|
|
BdrvCoBlockStatusData *data = opaque;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-10-12 06:47:07 +03:00
|
|
|
data->ret = bdrv_co_block_status_above(data->bs, data->base,
|
|
|
|
data->want_zero,
|
|
|
|
data->offset, data->bytes,
|
|
|
|
data->pnum, data->map, data->file);
|
2015-04-28 16:27:52 +03:00
|
|
|
data->done = true;
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2017-10-12 06:47:07 +03:00
|
|
|
* Synchronous wrapper around bdrv_co_block_status_above().
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
2017-10-12 06:47:07 +03:00
|
|
|
* See bdrv_co_block_status_above() for details.
|
2015-04-28 16:27:52 +03:00
|
|
|
*/
|
2017-10-12 06:47:06 +03:00
|
|
|
static int bdrv_common_block_status_above(BlockDriverState *bs,
|
|
|
|
BlockDriverState *base,
|
|
|
|
bool want_zero, int64_t offset,
|
|
|
|
int64_t bytes, int64_t *pnum,
|
|
|
|
int64_t *map,
|
|
|
|
BlockDriverState **file)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
Coroutine *co;
|
2017-10-12 06:47:05 +03:00
|
|
|
BdrvCoBlockStatusData data = {
|
2015-04-28 16:27:52 +03:00
|
|
|
.bs = bs,
|
2015-06-08 08:56:07 +03:00
|
|
|
.base = base,
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
.want_zero = want_zero,
|
2017-10-12 06:47:06 +03:00
|
|
|
.offset = offset,
|
|
|
|
.bytes = bytes,
|
|
|
|
.pnum = pnum,
|
|
|
|
.map = map,
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
.file = file,
|
2015-04-28 16:27:52 +03:00
|
|
|
.done = false,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
/* Fast-path if already in coroutine context */
|
2017-10-12 06:47:07 +03:00
|
|
|
bdrv_block_status_above_co_entry(&data);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
2017-10-12 06:47:07 +03:00
|
|
|
co = qemu_coroutine_create(bdrv_block_status_above_co_entry, &data);
|
block: Use bdrv_coroutine_enter to start I/O coroutines
BDRV_POLL_WHILE waits for the started I/O by releasing bs's ctx then polling
the main context, which relies on the yielded coroutine continuing on bs->ctx
before notifying qemu_aio_context with bdrv_wakeup().
Thus, using qemu_coroutine_enter to start I/O is wrong because if the coroutine
is entered from main loop, co->ctx will be qemu_aio_context, as a result of the
"release, poll, acquire" loop of BDRV_POLL_WHILE, race conditions happen when
both main thread and the iothread access the same BDS:
main loop iothread
-----------------------------------------------------------------------
blockdev_snapshot
aio_context_acquire(bs->ctx)
virtio_scsi_data_plane_handle_cmd
bdrv_drained_begin(bs->ctx)
bdrv_flush(bs)
bdrv_co_flush(bs) aio_context_acquire(bs->ctx).enter
...
qemu_coroutine_yield(co)
BDRV_POLL_WHILE()
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx).return
...
aio_co_wake(co)
aio_poll(qemu_aio_context) ...
co_schedule_bh_cb() ...
qemu_coroutine_enter(co) ...
/* (A) bdrv_co_flush(bs) /* (B) I/O on bs */
continues... */
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx)
Note that in above case, bdrv_drained_begin() doesn't do the "release,
poll, acquire" in BDRV_POLL_WHILE, because bs->in_flight == 0.
Fix this by using bdrv_coroutine_enter and enter coroutine in the right
context.
iotests 109 output is updated because the coroutine reenter flow during
mirror job complete is different (now through co_queue_wakeup, instead
of the unconditional qemu_coroutine_switch before), making the end job
len different.
Signed-off-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2017-04-10 15:20:17 +03:00
|
|
|
bdrv_coroutine_enter(bs, co);
|
2016-10-27 13:48:55 +03:00
|
|
|
BDRV_POLL_WHILE(bs, !data.done);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2017-10-12 06:47:06 +03:00
|
|
|
return data.ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
block: Convert bdrv_get_block_status_above() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status_above()
to bdrv_block_status_above() ensures that the compiler enforces that
all callers are updated. Likewise, since it a byte interface allows
an offset mapping that might not be sector aligned, split the mapping
out of the return value and into a pass-by-reference parameter. For
now, the io.c layer still assert()s that all uses are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status in the drivers.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), plus
updates for the new split return interface. But some code,
particularly bdrv_block_status(), gets a lot simpler because it no
longer has to mess with sectors. Likewise, mirror code no longer
computes s->granularity >> BDRV_SECTOR_BITS, and can therefore drop
an assertion about alignment because the loop no longer depends on
alignment (never mind that we don't really have a driver that
reports sub-sector alignments, so it's not really possible to test
the effect of sub-sector mirroring). Fix a neighboring assertion to
use is_power_of_2 while there.
For ease of review, bdrv_get_block_status() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:08 +03:00
|
|
|
int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,
|
|
|
|
int64_t offset, int64_t bytes, int64_t *pnum,
|
|
|
|
int64_t *map, BlockDriverState **file)
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
{
|
block: Convert bdrv_get_block_status_above() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status_above()
to bdrv_block_status_above() ensures that the compiler enforces that
all callers are updated. Likewise, since it a byte interface allows
an offset mapping that might not be sector aligned, split the mapping
out of the return value and into a pass-by-reference parameter. For
now, the io.c layer still assert()s that all uses are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status in the drivers.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), plus
updates for the new split return interface. But some code,
particularly bdrv_block_status(), gets a lot simpler because it no
longer has to mess with sectors. Likewise, mirror code no longer
computes s->granularity >> BDRV_SECTOR_BITS, and can therefore drop
an assertion about alignment because the loop no longer depends on
alignment (never mind that we don't really have a driver that
reports sub-sector alignments, so it's not really possible to test
the effect of sub-sector mirroring). Fix a neighboring assertion to
use is_power_of_2 while there.
For ease of review, bdrv_get_block_status() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:08 +03:00
|
|
|
return bdrv_common_block_status_above(bs, base, true, offset, bytes,
|
|
|
|
pnum, map, file);
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
}
|
|
|
|
|
block: Convert bdrv_get_block_status() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status() to
bdrv_block_status() ensures that the compiler enforces that all
callers are updated. For now, the io.c layer still assert()s that
all callers are sector-aligned, but that can be relaxed when a later
patch implements byte-based block status in the drivers.
There was an inherent limitation in returning the offset via the
return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which
means an offset can only be mapped for sector-aligned queries (or,
if we declare that non-aligned input is at the same relative position
modulo 512 of the answer), so the new interface also changes things to
return the offset via output through a parameter by reference rather
than mashed into the return value. We'll have some glue code that
munges between the two styles until we finish converting all uses.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), coupled
with the tweak in calling convention. But some code, particularly
bdrv_is_allocated(), gets a lot simpler because it no longer has to
mess with sectors.
For ease of review, bdrv_get_block_status_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:03 +03:00
|
|
|
int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes,
|
|
|
|
int64_t *pnum, int64_t *map, BlockDriverState **file)
|
2015-06-08 08:56:07 +03:00
|
|
|
{
|
block: Convert bdrv_get_block_status_above() to bytes
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the name of the function from bdrv_get_block_status_above()
to bdrv_block_status_above() ensures that the compiler enforces that
all callers are updated. Likewise, since it a byte interface allows
an offset mapping that might not be sector aligned, split the mapping
out of the return value and into a pass-by-reference parameter. For
now, the io.c layer still assert()s that all uses are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status in the drivers.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_block_status(), plus
updates for the new split return interface. But some code,
particularly bdrv_block_status(), gets a lot simpler because it no
longer has to mess with sectors. Likewise, mirror code no longer
computes s->granularity >> BDRV_SECTOR_BITS, and can therefore drop
an assertion about alignment because the loop no longer depends on
alignment (never mind that we don't really have a driver that
reports sub-sector alignments, so it's not really possible to test
the effect of sub-sector mirroring). Fix a neighboring assertion to
use is_power_of_2 while there.
For ease of review, bdrv_get_block_status() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:47:08 +03:00
|
|
|
return bdrv_block_status_above(bs, backing_bs(bs),
|
|
|
|
offset, bytes, pnum, map, file);
|
2015-06-08 08:56:07 +03:00
|
|
|
}
|
|
|
|
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
|
|
|
|
int64_t bytes, int64_t *pnum)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2017-10-12 06:47:06 +03:00
|
|
|
int ret;
|
|
|
|
int64_t dummy;
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
|
2017-10-12 06:47:06 +03:00
|
|
|
ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, offset,
|
|
|
|
bytes, pnum ? pnum : &dummy, NULL,
|
block: Add flag to avoid wasted work in bdrv_is_allocated()
Not all callers care about which BDS owns the mapping for a given
range of the file, or where the zeroes lie within that mapping. In
particular, bdrv_is_allocated() cares more about finding the
largest run of allocated data from the guest perspective, whether
or not that data is consecutive from the host perspective, and
whether or not the data reads as zero. Therefore, doing subsequent
refinements such as checking how much of the format-layer
allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is
wasted work - in the best case, it just costs extra CPU cycles
during a single bdrv_is_allocated(), but in the worst case, it
results in a smaller *pnum, and forces callers to iterate through
more status probes when visiting the entire file for even more
extra CPU cycles.
This patch only optimizes the block layer (no behavior change when
want_zero is true, but skip unnecessary effort when it is false).
Then when subsequent patches tweak the driver callback to be
byte-based, we can also pass this hint through to the driver.
Tweak BdrvCoGetBlockStatusData to declare arguments in parameter
order, rather than mixing things up (minimizing padding is not
necessary here).
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-10-12 06:46:58 +03:00
|
|
|
NULL);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
return !!(ret & BDRV_BLOCK_ALLOCATED);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Given an image chain: ... -> [BASE] -> [INTER1] -> [INTER2] -> [TOP]
|
|
|
|
*
|
2019-05-29 20:56:14 +03:00
|
|
|
* Return 1 if (a prefix of) the given range is allocated in any image
|
|
|
|
* between BASE and TOP (BASE is only included if include_base is set).
|
|
|
|
* BASE can be NULL to check if the given offset is allocated in any
|
|
|
|
* image of the chain. Return 0 otherwise, or negative errno on
|
|
|
|
* failure.
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
* 'pnum' is set to the number of bytes (including and immediately
|
|
|
|
* following the specified offset) that are known to be in the same
|
|
|
|
* allocated/unallocated state. Note that a subsequent call starting
|
|
|
|
* at 'offset + *pnum' may return the same allocation status (in other
|
|
|
|
* words, the result is not necessarily the maximum possible range);
|
|
|
|
* but 'pnum' will only be 0 when end of file is reached.
|
2015-04-28 16:27:52 +03:00
|
|
|
*
|
|
|
|
*/
|
|
|
|
int bdrv_is_allocated_above(BlockDriverState *top,
|
|
|
|
BlockDriverState *base,
|
2019-05-29 20:56:14 +03:00
|
|
|
bool include_base, int64_t offset,
|
|
|
|
int64_t bytes, int64_t *pnum)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
BlockDriverState *intermediate;
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
int ret;
|
|
|
|
int64_t n = bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-05-29 20:56:14 +03:00
|
|
|
assert(base || !include_base);
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
intermediate = top;
|
2019-05-29 20:56:14 +03:00
|
|
|
while (include_base || intermediate != base) {
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
int64_t pnum_inter;
|
2017-07-07 15:44:58 +03:00
|
|
|
int64_t size_inter;
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
|
2019-05-29 20:56:14 +03:00
|
|
|
assert(intermediate);
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
ret = bdrv_is_allocated(intermediate, offset, bytes, &pnum_inter);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
block: Make bdrv_is_allocated() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned
on input and that *pnum is sector-aligned on return to the caller,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, this code adds usages like
DIV_ROUND_UP(,BDRV_SECTOR_SIZE) to callers that still want aligned
values, where the call might reasonbly give non-aligned results
in the future; on the other hand, no rounding is needed for callers
that should just continue to work with byte alignment.
For the most part this patch is just the addition of scaling at the
callers followed by inverse scaling at bdrv_is_allocated(). But
some code, particularly bdrv_commit(), gets a lot simpler because it
no longer has to mess with sectors; also, it is now possible to pass
NULL if the caller does not care how much of the image is allocated
beyond the initial offset. Leave comments where we can further
simplify once a later patch eliminates the need for sector-aligned
requests through bdrv_is_allocated().
For ease of review, bdrv_is_allocated_above() will be tackled
separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:57 +03:00
|
|
|
}
|
|
|
|
if (ret) {
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
*pnum = pnum_inter;
|
2015-04-28 16:27:52 +03:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
size_inter = bdrv_getlength(intermediate);
|
2017-07-07 15:44:58 +03:00
|
|
|
if (size_inter < 0) {
|
|
|
|
return size_inter;
|
|
|
|
}
|
block: Make bdrv_is_allocated_above() byte-based
We are gradually moving away from sector-based interfaces, towards
byte-based. In the common case, allocation is unlikely to ever use
values that are not naturally sector-aligned, but it is possible
that byte-based values will let us be more precise about allocation
at the end of an unaligned file that can do byte-based access.
Changing the signature of the function to use int64_t *pnum ensures
that the compiler enforces that all callers are updated. For now,
the io.c layer still assert()s that all callers are sector-aligned,
but that can be relaxed when a later patch implements byte-based
block status. Therefore, for the most part this patch is just the
addition of scaling at the callers followed by inverse scaling at
bdrv_is_allocated(). But some code, particularly stream_run(),
gets a lot simpler because it no longer has to mess with sectors.
Leave comments where we can further simplify by switching to
byte-based iterations, once later patches eliminate the need for
sector-aligned operations.
For ease of review, bdrv_is_allocated() was tackled separately.
Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2017-07-07 15:44:59 +03:00
|
|
|
if (n > pnum_inter &&
|
|
|
|
(intermediate == top || offset + pnum_inter < size_inter)) {
|
|
|
|
n = pnum_inter;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2019-05-29 20:56:14 +03:00
|
|
|
if (intermediate == base) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2015-06-17 15:55:21 +03:00
|
|
|
intermediate = backing_bs(intermediate);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
*pnum = n;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-06-09 17:24:44 +03:00
|
|
|
typedef struct BdrvVmstateCo {
|
|
|
|
BlockDriverState *bs;
|
|
|
|
QEMUIOVector *qiov;
|
|
|
|
int64_t pos;
|
|
|
|
bool is_read;
|
|
|
|
int ret;
|
|
|
|
} BdrvVmstateCo;
|
|
|
|
|
|
|
|
static int coroutine_fn
|
|
|
|
bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
|
|
|
|
bool is_read)
|
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
2017-05-22 16:57:01 +03:00
|
|
|
int ret = -ENOTSUP;
|
|
|
|
|
|
|
|
bdrv_inc_in_flight(bs);
|
2016-06-09 17:24:44 +03:00
|
|
|
|
|
|
|
if (!drv) {
|
2017-05-22 16:57:01 +03:00
|
|
|
ret = -ENOMEDIUM;
|
2016-06-09 17:24:44 +03:00
|
|
|
} else if (drv->bdrv_load_vmstate) {
|
2017-05-22 16:57:01 +03:00
|
|
|
if (is_read) {
|
|
|
|
ret = drv->bdrv_load_vmstate(bs, qiov, pos);
|
|
|
|
} else {
|
|
|
|
ret = drv->bdrv_save_vmstate(bs, qiov, pos);
|
|
|
|
}
|
2016-06-09 17:24:44 +03:00
|
|
|
} else if (bs->file) {
|
2017-05-22 16:57:01 +03:00
|
|
|
ret = bdrv_co_rw_vmstate(bs->file->bs, qiov, pos, is_read);
|
2016-06-09 17:24:44 +03:00
|
|
|
}
|
|
|
|
|
2017-05-22 16:57:01 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
|
|
|
return ret;
|
2016-06-09 17:24:44 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
static void coroutine_fn bdrv_co_rw_vmstate_entry(void *opaque)
|
|
|
|
{
|
|
|
|
BdrvVmstateCo *co = opaque;
|
|
|
|
co->ret = bdrv_co_rw_vmstate(co->bs, co->qiov, co->pos, co->is_read);
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2016-06-09 17:24:44 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline int
|
|
|
|
bdrv_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
|
|
|
|
bool is_read)
|
|
|
|
{
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
return bdrv_co_rw_vmstate(bs, qiov, pos, is_read);
|
|
|
|
} else {
|
|
|
|
BdrvVmstateCo data = {
|
|
|
|
.bs = bs,
|
|
|
|
.qiov = qiov,
|
|
|
|
.pos = pos,
|
|
|
|
.is_read = is_read,
|
|
|
|
.ret = -EINPROGRESS,
|
|
|
|
};
|
coroutine: move entry argument to qemu_coroutine_create
In practice the entry argument is always known at creation time, and
it is confusing that sometimes qemu_coroutine_enter is used with a
non-NULL argument to re-enter a coroutine (this happens in
block/sheepdog.c and tests/test-coroutine.c). So pass the opaque value
at creation time, for consistency with e.g. aio_bh_new.
Mostly done with the following semantic patch:
@ entry1 @
expression entry, arg, co;
@@
- co = qemu_coroutine_create(entry);
+ co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry2 @
expression entry, arg;
identifier co;
@@
- Coroutine *co = qemu_coroutine_create(entry);
+ Coroutine *co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry3 @
expression entry, arg;
@@
- qemu_coroutine_enter(qemu_coroutine_create(entry), arg);
+ qemu_coroutine_enter(qemu_coroutine_create(entry, arg));
@ reentry @
expression co;
@@
- qemu_coroutine_enter(co, NULL);
+ qemu_coroutine_enter(co);
except for the aforementioned few places where the semantic patch
stumbled (as expected) and for test_co_queue, which would otherwise
produce an uninitialized variable warning.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-07-04 20:10:01 +03:00
|
|
|
Coroutine *co = qemu_coroutine_create(bdrv_co_rw_vmstate_entry, &data);
|
2016-06-09 17:24:44 +03:00
|
|
|
|
block: Use bdrv_coroutine_enter to start I/O coroutines
BDRV_POLL_WHILE waits for the started I/O by releasing bs's ctx then polling
the main context, which relies on the yielded coroutine continuing on bs->ctx
before notifying qemu_aio_context with bdrv_wakeup().
Thus, using qemu_coroutine_enter to start I/O is wrong because if the coroutine
is entered from main loop, co->ctx will be qemu_aio_context, as a result of the
"release, poll, acquire" loop of BDRV_POLL_WHILE, race conditions happen when
both main thread and the iothread access the same BDS:
main loop iothread
-----------------------------------------------------------------------
blockdev_snapshot
aio_context_acquire(bs->ctx)
virtio_scsi_data_plane_handle_cmd
bdrv_drained_begin(bs->ctx)
bdrv_flush(bs)
bdrv_co_flush(bs) aio_context_acquire(bs->ctx).enter
...
qemu_coroutine_yield(co)
BDRV_POLL_WHILE()
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx).return
...
aio_co_wake(co)
aio_poll(qemu_aio_context) ...
co_schedule_bh_cb() ...
qemu_coroutine_enter(co) ...
/* (A) bdrv_co_flush(bs) /* (B) I/O on bs */
continues... */
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx)
Note that in above case, bdrv_drained_begin() doesn't do the "release,
poll, acquire" in BDRV_POLL_WHILE, because bs->in_flight == 0.
Fix this by using bdrv_coroutine_enter and enter coroutine in the right
context.
iotests 109 output is updated because the coroutine reenter flow during
mirror job complete is different (now through co_queue_wakeup, instead
of the unconditional qemu_coroutine_switch before), making the end job
len different.
Signed-off-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2017-04-10 15:20:17 +03:00
|
|
|
bdrv_coroutine_enter(bs, co);
|
2017-05-22 16:57:02 +03:00
|
|
|
BDRV_POLL_WHILE(bs, data.ret == -EINPROGRESS);
|
2016-06-09 17:24:44 +03:00
|
|
|
return data.ret;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
int bdrv_save_vmstate(BlockDriverState *bs, const uint8_t *buf,
|
|
|
|
int64_t pos, int size)
|
|
|
|
{
|
2019-02-18 17:09:11 +03:00
|
|
|
QEMUIOVector qiov = QEMU_IOVEC_INIT_BUF(qiov, buf, size);
|
2016-06-10 18:57:26 +03:00
|
|
|
int ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-06-10 18:57:26 +03:00
|
|
|
ret = bdrv_writev_vmstate(bs, &qiov, pos);
|
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return size;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int bdrv_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos)
|
|
|
|
{
|
2016-06-09 17:24:44 +03:00
|
|
|
return bdrv_rw_vmstate(bs, qiov, pos, false);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int bdrv_load_vmstate(BlockDriverState *bs, uint8_t *buf,
|
|
|
|
int64_t pos, int size)
|
2016-06-09 17:50:16 +03:00
|
|
|
{
|
2019-02-18 17:09:11 +03:00
|
|
|
QEMUIOVector qiov = QEMU_IOVEC_INIT_BUF(qiov, buf, size);
|
2016-06-10 18:57:26 +03:00
|
|
|
int ret;
|
2016-06-09 17:50:16 +03:00
|
|
|
|
2016-06-10 18:57:26 +03:00
|
|
|
ret = bdrv_readv_vmstate(bs, &qiov, pos);
|
|
|
|
if (ret < 0) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return size;
|
2016-06-09 17:50:16 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int bdrv_readv_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2016-06-09 17:24:44 +03:00
|
|
|
return bdrv_rw_vmstate(bs, qiov, pos, true);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/**************************************************************/
|
|
|
|
/* async I/Os */
|
|
|
|
|
|
|
|
void bdrv_aio_cancel(BlockAIOCB *acb)
|
|
|
|
{
|
|
|
|
qemu_aio_ref(acb);
|
|
|
|
bdrv_aio_cancel_async(acb);
|
|
|
|
while (acb->refcnt > 1) {
|
|
|
|
if (acb->aiocb_info->get_aio_context) {
|
|
|
|
aio_poll(acb->aiocb_info->get_aio_context(acb), true);
|
|
|
|
} else if (acb->bs) {
|
2017-02-13 16:52:29 +03:00
|
|
|
/* qemu_aio_ref and qemu_aio_unref are not thread-safe, so
|
|
|
|
* assert that we're not using an I/O thread. Thread-safe
|
|
|
|
* code should use bdrv_aio_cancel_async exclusively.
|
|
|
|
*/
|
|
|
|
assert(bdrv_get_aio_context(acb->bs) == qemu_get_aio_context());
|
2015-04-28 16:27:52 +03:00
|
|
|
aio_poll(bdrv_get_aio_context(acb->bs), true);
|
|
|
|
} else {
|
|
|
|
abort();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
qemu_aio_unref(acb);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Async version of aio cancel. The caller is not blocked if the acb implements
|
|
|
|
* cancel_async, otherwise we do nothing and let the request normally complete.
|
|
|
|
* In either case the completion callback must be called. */
|
|
|
|
void bdrv_aio_cancel_async(BlockAIOCB *acb)
|
|
|
|
{
|
|
|
|
if (acb->aiocb_info->cancel_async) {
|
|
|
|
acb->aiocb_info->cancel_async(acb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/**************************************************************/
|
|
|
|
/* Coroutine block device emulation */
|
|
|
|
|
2016-05-30 16:00:40 +03:00
|
|
|
typedef struct FlushCo {
|
|
|
|
BlockDriverState *bs;
|
|
|
|
int ret;
|
|
|
|
} FlushCo;
|
|
|
|
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
static void coroutine_fn bdrv_flush_co_entry(void *opaque)
|
|
|
|
{
|
2016-05-30 16:00:40 +03:00
|
|
|
FlushCo *rwco = opaque;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
rwco->ret = bdrv_co_flush(rwco->bs);
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
|
|
|
|
{
|
2017-04-10 16:00:50 +03:00
|
|
|
int current_gen;
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
bdrv_inc_in_flight(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-04-26 05:03:49 +03:00
|
|
|
if (!bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
|
2015-06-23 13:44:57 +03:00
|
|
|
bdrv_is_sg(bs)) {
|
2017-04-10 16:00:50 +03:00
|
|
|
goto early_exit;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_lock(&bs->reqs_lock);
|
2017-06-05 15:39:01 +03:00
|
|
|
current_gen = atomic_read(&bs->write_gen);
|
2016-07-18 22:39:52 +03:00
|
|
|
|
|
|
|
/* Wait until any previous flushes are completed */
|
2016-10-27 13:48:52 +03:00
|
|
|
while (bs->active_flush_req) {
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_queue_wait(&bs->flush_queue, &bs->reqs_lock);
|
2016-07-18 22:39:52 +03:00
|
|
|
}
|
|
|
|
|
2017-06-05 15:39:02 +03:00
|
|
|
/* Flushes reach this point in nondecreasing current_gen order. */
|
2016-10-27 13:48:52 +03:00
|
|
|
bs->active_flush_req = true;
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_unlock(&bs->reqs_lock);
|
2016-07-18 22:39:52 +03:00
|
|
|
|
2016-03-14 10:44:53 +03:00
|
|
|
/* Write back all layers by calling one driver function */
|
|
|
|
if (bs->drv->bdrv_co_flush) {
|
|
|
|
ret = bs->drv->bdrv_co_flush(bs);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
/* Write back cached data to the OS even with cache=unsafe */
|
|
|
|
BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
|
|
|
|
if (bs->drv->bdrv_co_flush_to_os) {
|
|
|
|
ret = bs->drv->bdrv_co_flush_to_os(bs);
|
|
|
|
if (ret < 0) {
|
2015-11-09 13:16:47 +03:00
|
|
|
goto out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* But don't actually force it to the disk with cache=unsafe */
|
|
|
|
if (bs->open_flags & BDRV_O_NO_FLUSH) {
|
|
|
|
goto flush_parent;
|
|
|
|
}
|
|
|
|
|
2016-07-18 22:39:52 +03:00
|
|
|
/* Check if we really need to flush anything */
|
|
|
|
if (bs->flushed_gen == current_gen) {
|
|
|
|
goto flush_parent;
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!bs->drv) {
|
|
|
|
/* bs->drv->bdrv_co_flush() might have ejected the BDS
|
|
|
|
* (even in case of apparent success) */
|
|
|
|
ret = -ENOMEDIUM;
|
|
|
|
goto out;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
if (bs->drv->bdrv_co_flush_to_disk) {
|
|
|
|
ret = bs->drv->bdrv_co_flush_to_disk(bs);
|
|
|
|
} else if (bs->drv->bdrv_aio_flush) {
|
|
|
|
BlockAIOCB *acb;
|
|
|
|
CoroutineIOCompletion co = {
|
|
|
|
.coroutine = qemu_coroutine_self(),
|
|
|
|
};
|
|
|
|
|
|
|
|
acb = bs->drv->bdrv_aio_flush(bs, bdrv_co_io_em_complete, &co);
|
|
|
|
if (acb == NULL) {
|
|
|
|
ret = -EIO;
|
|
|
|
} else {
|
|
|
|
qemu_coroutine_yield();
|
|
|
|
ret = co.ret;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Some block drivers always operate in either writethrough or unsafe
|
|
|
|
* mode and don't support bdrv_flush therefore. Usually qemu doesn't
|
|
|
|
* know how the server works (because the behaviour is hardcoded or
|
|
|
|
* depends on server-side configuration), so we can't ensure that
|
|
|
|
* everything is safe on disk. Returning an error doesn't work because
|
|
|
|
* that would break guests even if the server operates in writethrough
|
|
|
|
* mode.
|
|
|
|
*
|
|
|
|
* Let's hope the user knows what he's doing.
|
|
|
|
*/
|
|
|
|
ret = 0;
|
|
|
|
}
|
2016-07-18 22:39:52 +03:00
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
if (ret < 0) {
|
2015-11-09 13:16:47 +03:00
|
|
|
goto out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH
|
|
|
|
* in the case of cache=unsafe, so there are no useless flushes.
|
|
|
|
*/
|
|
|
|
flush_parent:
|
2015-11-09 13:16:47 +03:00
|
|
|
ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
|
|
|
|
out:
|
2016-07-18 22:39:52 +03:00
|
|
|
/* Notify any pending flushes that we have completed */
|
2016-11-05 02:03:15 +03:00
|
|
|
if (ret == 0) {
|
|
|
|
bs->flushed_gen = current_gen;
|
|
|
|
}
|
2017-06-05 15:39:02 +03:00
|
|
|
|
|
|
|
qemu_co_mutex_lock(&bs->reqs_lock);
|
2016-10-27 13:48:52 +03:00
|
|
|
bs->active_flush_req = false;
|
2016-08-17 21:06:54 +03:00
|
|
|
/* Return value is ignored - it's ok if wait queue is empty */
|
|
|
|
qemu_co_queue_next(&bs->flush_queue);
|
2017-06-05 15:39:02 +03:00
|
|
|
qemu_co_mutex_unlock(&bs->reqs_lock);
|
2016-07-18 22:39:52 +03:00
|
|
|
|
2017-04-10 16:00:50 +03:00
|
|
|
early_exit:
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2015-11-09 13:16:47 +03:00
|
|
|
return ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int bdrv_flush(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
Coroutine *co;
|
2016-05-30 16:00:40 +03:00
|
|
|
FlushCo flush_co = {
|
2015-04-28 16:27:52 +03:00
|
|
|
.bs = bs,
|
|
|
|
.ret = NOT_DONE,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
/* Fast-path if already in coroutine context */
|
2016-05-30 16:00:40 +03:00
|
|
|
bdrv_flush_co_entry(&flush_co);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
coroutine: move entry argument to qemu_coroutine_create
In practice the entry argument is always known at creation time, and
it is confusing that sometimes qemu_coroutine_enter is used with a
non-NULL argument to re-enter a coroutine (this happens in
block/sheepdog.c and tests/test-coroutine.c). So pass the opaque value
at creation time, for consistency with e.g. aio_bh_new.
Mostly done with the following semantic patch:
@ entry1 @
expression entry, arg, co;
@@
- co = qemu_coroutine_create(entry);
+ co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry2 @
expression entry, arg;
identifier co;
@@
- Coroutine *co = qemu_coroutine_create(entry);
+ Coroutine *co = qemu_coroutine_create(entry, arg);
...
- qemu_coroutine_enter(co, arg);
+ qemu_coroutine_enter(co);
@ entry3 @
expression entry, arg;
@@
- qemu_coroutine_enter(qemu_coroutine_create(entry), arg);
+ qemu_coroutine_enter(qemu_coroutine_create(entry, arg));
@ reentry @
expression co;
@@
- qemu_coroutine_enter(co, NULL);
+ qemu_coroutine_enter(co);
except for the aforementioned few places where the semantic patch
stumbled (as expected) and for test_co_queue, which would otherwise
produce an uninitialized variable warning.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-07-04 20:10:01 +03:00
|
|
|
co = qemu_coroutine_create(bdrv_flush_co_entry, &flush_co);
|
block: Use bdrv_coroutine_enter to start I/O coroutines
BDRV_POLL_WHILE waits for the started I/O by releasing bs's ctx then polling
the main context, which relies on the yielded coroutine continuing on bs->ctx
before notifying qemu_aio_context with bdrv_wakeup().
Thus, using qemu_coroutine_enter to start I/O is wrong because if the coroutine
is entered from main loop, co->ctx will be qemu_aio_context, as a result of the
"release, poll, acquire" loop of BDRV_POLL_WHILE, race conditions happen when
both main thread and the iothread access the same BDS:
main loop iothread
-----------------------------------------------------------------------
blockdev_snapshot
aio_context_acquire(bs->ctx)
virtio_scsi_data_plane_handle_cmd
bdrv_drained_begin(bs->ctx)
bdrv_flush(bs)
bdrv_co_flush(bs) aio_context_acquire(bs->ctx).enter
...
qemu_coroutine_yield(co)
BDRV_POLL_WHILE()
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx).return
...
aio_co_wake(co)
aio_poll(qemu_aio_context) ...
co_schedule_bh_cb() ...
qemu_coroutine_enter(co) ...
/* (A) bdrv_co_flush(bs) /* (B) I/O on bs */
continues... */
aio_context_release(bs->ctx)
aio_context_acquire(bs->ctx)
Note that in above case, bdrv_drained_begin() doesn't do the "release,
poll, acquire" in BDRV_POLL_WHILE, because bs->in_flight == 0.
Fix this by using bdrv_coroutine_enter and enter coroutine in the right
context.
iotests 109 output is updated because the coroutine reenter flow during
mirror job complete is different (now through co_queue_wakeup, instead
of the unconditional qemu_coroutine_switch before), making the end job
len different.
Signed-off-by: Fam Zheng <famz@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
2017-04-10 15:20:17 +03:00
|
|
|
bdrv_coroutine_enter(bs, co);
|
2016-10-27 13:48:55 +03:00
|
|
|
BDRV_POLL_WHILE(bs, flush_co.ret == NOT_DONE);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-05-30 16:00:40 +03:00
|
|
|
return flush_co.ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
typedef struct DiscardCo {
|
2018-07-10 09:31:17 +03:00
|
|
|
BdrvChild *child;
|
2016-07-16 02:22:51 +03:00
|
|
|
int64_t offset;
|
2019-04-23 15:57:05 +03:00
|
|
|
int64_t bytes;
|
2015-04-28 16:27:52 +03:00
|
|
|
int ret;
|
|
|
|
} DiscardCo;
|
2016-07-16 02:22:51 +03:00
|
|
|
static void coroutine_fn bdrv_pdiscard_co_entry(void *opaque)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
DiscardCo *rwco = opaque;
|
|
|
|
|
2018-07-10 09:31:17 +03:00
|
|
|
rwco->ret = bdrv_co_pdiscard(rwco->child, rwco->offset, rwco->bytes);
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2019-04-23 15:57:05 +03:00
|
|
|
int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset,
|
|
|
|
int64_t bytes)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
2015-11-09 13:16:48 +03:00
|
|
|
BdrvTrackedRequest req;
|
2016-07-16 02:22:50 +03:00
|
|
|
int max_pdiscard, ret;
|
block: Pass unaligned discard requests to drivers
Discard is advisory, so rounding the requests to alignment
boundaries is never semantically wrong from the data that
the guest sees. But at least the Dell Equallogic iSCSI SANs
has an interesting property that its advertised discard
alignment is 15M, yet documents that discarding a sequence
of 1M slices will eventually result in the 15M page being
marked as discarded, and it is possible to observe which
pages have been discarded.
Between commits 9f1963b and b8d0a980, we converted the block
layer to a byte-based interface that ultimately ignores any
unaligned head or tail based on the driver's advertised
discard granularity, which means that qemu 2.7 refuses to
pass any discard request smaller than 15M down to the Dell
Equallogic hardware. This is a slight regression in behavior
compared to earlier qemu, where a guest executing discards
in power-of-2 chunks used to be able to get every page
discarded, but is now left with various pages still allocated
because the guest requests did not align with the hardware's
15M pages.
Since the SCSI specification says nothing about a minimum
discard granularity, and only documents the preferred
alignment, it is best if the block layer gives the driver
every bit of information about discard requests, rather than
rounding it to alignment boundaries early.
Rework the block layer discard algorithm to mirror the write
zero algorithm: always peel off any unaligned head or tail
and manage that in isolation, then do the bulk of the request
on an aligned boundary. The fallback when the driver returns
-ENOTSUP for an unaligned request is to silently ignore that
portion of the discard request; but for devices that can pass
the partial request all the way down to hardware, this can
result in the hardware coalescing requests and discarding
aligned pages after all.
Reported by: Peter Lieven <pl@kamp.de>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-11-17 23:13:58 +03:00
|
|
|
int head, tail, align;
|
2018-07-10 09:31:17 +03:00
|
|
|
BlockDriverState *bs = child->bs;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2019-04-23 15:57:05 +03:00
|
|
|
if (!bs || !bs->drv || !bdrv_is_inserted(bs)) {
|
2015-04-28 16:27:52 +03:00
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
|
2017-06-28 15:05:10 +03:00
|
|
|
if (bdrv_has_readonly_bitmaps(bs)) {
|
|
|
|
return -EPERM;
|
|
|
|
}
|
|
|
|
|
2019-04-23 15:57:05 +03:00
|
|
|
if (offset < 0 || bytes < 0 || bytes > INT64_MAX - offset) {
|
|
|
|
return -EIO;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Do nothing if disabled. */
|
|
|
|
if (!(bs->open_flags & BDRV_O_UNMAP)) {
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-07-16 02:23:06 +03:00
|
|
|
if (!bs->drv->bdrv_co_pdiscard && !bs->drv->bdrv_aio_pdiscard) {
|
2015-04-28 16:27:52 +03:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
block: Pass unaligned discard requests to drivers
Discard is advisory, so rounding the requests to alignment
boundaries is never semantically wrong from the data that
the guest sees. But at least the Dell Equallogic iSCSI SANs
has an interesting property that its advertised discard
alignment is 15M, yet documents that discarding a sequence
of 1M slices will eventually result in the 15M page being
marked as discarded, and it is possible to observe which
pages have been discarded.
Between commits 9f1963b and b8d0a980, we converted the block
layer to a byte-based interface that ultimately ignores any
unaligned head or tail based on the driver's advertised
discard granularity, which means that qemu 2.7 refuses to
pass any discard request smaller than 15M down to the Dell
Equallogic hardware. This is a slight regression in behavior
compared to earlier qemu, where a guest executing discards
in power-of-2 chunks used to be able to get every page
discarded, but is now left with various pages still allocated
because the guest requests did not align with the hardware's
15M pages.
Since the SCSI specification says nothing about a minimum
discard granularity, and only documents the preferred
alignment, it is best if the block layer gives the driver
every bit of information about discard requests, rather than
rounding it to alignment boundaries early.
Rework the block layer discard algorithm to mirror the write
zero algorithm: always peel off any unaligned head or tail
and manage that in isolation, then do the bulk of the request
on an aligned boundary. The fallback when the driver returns
-ENOTSUP for an unaligned request is to silently ignore that
portion of the discard request; but for devices that can pass
the partial request all the way down to hardware, this can
result in the hardware coalescing requests and discarding
aligned pages after all.
Reported by: Peter Lieven <pl@kamp.de>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-11-17 23:13:58 +03:00
|
|
|
/* Discard is advisory, but some devices track and coalesce
|
|
|
|
* unaligned requests, so we must pass everything down rather than
|
|
|
|
* round here. Still, most devices will just silently ignore
|
|
|
|
* unaligned requests (by returning -ENOTSUP), so we must fragment
|
|
|
|
* the request accordingly. */
|
2016-07-16 02:23:06 +03:00
|
|
|
align = MAX(bs->bl.pdiscard_alignment, bs->bl.request_alignment);
|
2016-07-21 22:34:48 +03:00
|
|
|
assert(align % bs->bl.request_alignment == 0);
|
|
|
|
head = offset % align;
|
2017-06-09 13:18:08 +03:00
|
|
|
tail = (offset + bytes) % align;
|
2016-07-16 02:22:50 +03:00
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
2017-06-09 13:18:08 +03:00
|
|
|
tracked_request_begin(&req, bs, offset, bytes, BDRV_TRACKED_DISCARD);
|
2015-06-08 08:56:10 +03:00
|
|
|
|
2018-07-10 09:31:21 +03:00
|
|
|
ret = bdrv_co_write_req_prepare(child, offset, bytes, &req, 0);
|
2016-06-16 19:09:41 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2016-07-16 02:22:50 +03:00
|
|
|
max_pdiscard = QEMU_ALIGN_DOWN(MIN_NON_ZERO(bs->bl.max_pdiscard, INT_MAX),
|
|
|
|
align);
|
block: Pass unaligned discard requests to drivers
Discard is advisory, so rounding the requests to alignment
boundaries is never semantically wrong from the data that
the guest sees. But at least the Dell Equallogic iSCSI SANs
has an interesting property that its advertised discard
alignment is 15M, yet documents that discarding a sequence
of 1M slices will eventually result in the 15M page being
marked as discarded, and it is possible to observe which
pages have been discarded.
Between commits 9f1963b and b8d0a980, we converted the block
layer to a byte-based interface that ultimately ignores any
unaligned head or tail based on the driver's advertised
discard granularity, which means that qemu 2.7 refuses to
pass any discard request smaller than 15M down to the Dell
Equallogic hardware. This is a slight regression in behavior
compared to earlier qemu, where a guest executing discards
in power-of-2 chunks used to be able to get every page
discarded, but is now left with various pages still allocated
because the guest requests did not align with the hardware's
15M pages.
Since the SCSI specification says nothing about a minimum
discard granularity, and only documents the preferred
alignment, it is best if the block layer gives the driver
every bit of information about discard requests, rather than
rounding it to alignment boundaries early.
Rework the block layer discard algorithm to mirror the write
zero algorithm: always peel off any unaligned head or tail
and manage that in isolation, then do the bulk of the request
on an aligned boundary. The fallback when the driver returns
-ENOTSUP for an unaligned request is to silently ignore that
portion of the discard request; but for devices that can pass
the partial request all the way down to hardware, this can
result in the hardware coalescing requests and discarding
aligned pages after all.
Reported by: Peter Lieven <pl@kamp.de>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-11-17 23:13:58 +03:00
|
|
|
assert(max_pdiscard >= bs->bl.request_alignment);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-06-09 13:18:08 +03:00
|
|
|
while (bytes > 0) {
|
2019-04-23 15:57:05 +03:00
|
|
|
int64_t num = bytes;
|
block: Pass unaligned discard requests to drivers
Discard is advisory, so rounding the requests to alignment
boundaries is never semantically wrong from the data that
the guest sees. But at least the Dell Equallogic iSCSI SANs
has an interesting property that its advertised discard
alignment is 15M, yet documents that discarding a sequence
of 1M slices will eventually result in the 15M page being
marked as discarded, and it is possible to observe which
pages have been discarded.
Between commits 9f1963b and b8d0a980, we converted the block
layer to a byte-based interface that ultimately ignores any
unaligned head or tail based on the driver's advertised
discard granularity, which means that qemu 2.7 refuses to
pass any discard request smaller than 15M down to the Dell
Equallogic hardware. This is a slight regression in behavior
compared to earlier qemu, where a guest executing discards
in power-of-2 chunks used to be able to get every page
discarded, but is now left with various pages still allocated
because the guest requests did not align with the hardware's
15M pages.
Since the SCSI specification says nothing about a minimum
discard granularity, and only documents the preferred
alignment, it is best if the block layer gives the driver
every bit of information about discard requests, rather than
rounding it to alignment boundaries early.
Rework the block layer discard algorithm to mirror the write
zero algorithm: always peel off any unaligned head or tail
and manage that in isolation, then do the bulk of the request
on an aligned boundary. The fallback when the driver returns
-ENOTSUP for an unaligned request is to silently ignore that
portion of the discard request; but for devices that can pass
the partial request all the way down to hardware, this can
result in the hardware coalescing requests and discarding
aligned pages after all.
Reported by: Peter Lieven <pl@kamp.de>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-11-17 23:13:58 +03:00
|
|
|
|
|
|
|
if (head) {
|
|
|
|
/* Make small requests to get to alignment boundaries. */
|
2017-06-09 13:18:08 +03:00
|
|
|
num = MIN(bytes, align - head);
|
block: Pass unaligned discard requests to drivers
Discard is advisory, so rounding the requests to alignment
boundaries is never semantically wrong from the data that
the guest sees. But at least the Dell Equallogic iSCSI SANs
has an interesting property that its advertised discard
alignment is 15M, yet documents that discarding a sequence
of 1M slices will eventually result in the 15M page being
marked as discarded, and it is possible to observe which
pages have been discarded.
Between commits 9f1963b and b8d0a980, we converted the block
layer to a byte-based interface that ultimately ignores any
unaligned head or tail based on the driver's advertised
discard granularity, which means that qemu 2.7 refuses to
pass any discard request smaller than 15M down to the Dell
Equallogic hardware. This is a slight regression in behavior
compared to earlier qemu, where a guest executing discards
in power-of-2 chunks used to be able to get every page
discarded, but is now left with various pages still allocated
because the guest requests did not align with the hardware's
15M pages.
Since the SCSI specification says nothing about a minimum
discard granularity, and only documents the preferred
alignment, it is best if the block layer gives the driver
every bit of information about discard requests, rather than
rounding it to alignment boundaries early.
Rework the block layer discard algorithm to mirror the write
zero algorithm: always peel off any unaligned head or tail
and manage that in isolation, then do the bulk of the request
on an aligned boundary. The fallback when the driver returns
-ENOTSUP for an unaligned request is to silently ignore that
portion of the discard request; but for devices that can pass
the partial request all the way down to hardware, this can
result in the hardware coalescing requests and discarding
aligned pages after all.
Reported by: Peter Lieven <pl@kamp.de>
CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2016-11-17 23:13:58 +03:00
|
|
|
if (!QEMU_IS_ALIGNED(num, bs->bl.request_alignment)) {
|
|
|
|
num %= bs->bl.request_alignment;
|
|
|
|
}
|
|
|
|
head = (head + num) % align;
|
|
|
|
assert(num < max_pdiscard);
|
|
|
|
} else if (tail) {
|
|
|
|
if (num > align) {
|
|
|
|
/* Shorten the request to the last aligned cluster. */
|
|
|
|
num -= tail;
|
|
|
|
} else if (!QEMU_IS_ALIGNED(tail, bs->bl.request_alignment) &&
|
|
|
|
tail > bs->bl.request_alignment) {
|
|
|
|
tail %= bs->bl.request_alignment;
|
|
|
|
num -= tail;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* limit request size */
|
|
|
|
if (num > max_pdiscard) {
|
|
|
|
num = max_pdiscard;
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2017-11-10 23:31:09 +03:00
|
|
|
if (!bs->drv) {
|
|
|
|
ret = -ENOMEDIUM;
|
|
|
|
goto out;
|
|
|
|
}
|
2016-07-16 02:22:58 +03:00
|
|
|
if (bs->drv->bdrv_co_pdiscard) {
|
|
|
|
ret = bs->drv->bdrv_co_pdiscard(bs, offset, num);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
|
|
|
BlockAIOCB *acb;
|
|
|
|
CoroutineIOCompletion co = {
|
|
|
|
.coroutine = qemu_coroutine_self(),
|
|
|
|
};
|
|
|
|
|
2016-07-16 02:22:57 +03:00
|
|
|
acb = bs->drv->bdrv_aio_pdiscard(bs, offset, num,
|
|
|
|
bdrv_co_io_em_complete, &co);
|
2015-04-28 16:27:52 +03:00
|
|
|
if (acb == NULL) {
|
2015-11-09 13:16:48 +03:00
|
|
|
ret = -EIO;
|
|
|
|
goto out;
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
|
|
|
qemu_coroutine_yield();
|
|
|
|
ret = co.ret;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (ret && ret != -ENOTSUP) {
|
2015-11-09 13:16:48 +03:00
|
|
|
goto out;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2016-07-16 02:22:50 +03:00
|
|
|
offset += num;
|
2017-06-09 13:18:08 +03:00
|
|
|
bytes -= num;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
2015-11-09 13:16:48 +03:00
|
|
|
ret = 0;
|
|
|
|
out:
|
2018-07-10 09:31:21 +03:00
|
|
|
bdrv_co_write_req_finish(child, req.offset, req.bytes, &req, ret);
|
2015-11-09 13:16:48 +03:00
|
|
|
tracked_request_end(&req);
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2015-11-09 13:16:48 +03:00
|
|
|
return ret;
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
2019-04-23 15:57:05 +03:00
|
|
|
int bdrv_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
Coroutine *co;
|
|
|
|
DiscardCo rwco = {
|
2018-07-10 09:31:17 +03:00
|
|
|
.child = child,
|
2016-07-16 02:22:51 +03:00
|
|
|
.offset = offset,
|
2017-06-09 13:18:08 +03:00
|
|
|
.bytes = bytes,
|
2015-04-28 16:27:52 +03:00
|
|
|
.ret = NOT_DONE,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
/* Fast-path if already in coroutine context */
|
2016-07-16 02:22:51 +03:00
|
|
|
bdrv_pdiscard_co_entry(&rwco);
|
2015-04-28 16:27:52 +03:00
|
|
|
} else {
|
2016-07-16 02:22:51 +03:00
|
|
|
co = qemu_coroutine_create(bdrv_pdiscard_co_entry, &rwco);
|
2018-07-10 09:31:17 +03:00
|
|
|
bdrv_coroutine_enter(child->bs, co);
|
|
|
|
BDRV_POLL_WHILE(child->bs, rwco.ret == NOT_DONE);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
return rwco.ret;
|
|
|
|
}
|
|
|
|
|
2016-10-20 13:56:14 +03:00
|
|
|
int bdrv_co_ioctl(BlockDriverState *bs, int req, void *buf)
|
2015-04-28 16:27:52 +03:00
|
|
|
{
|
|
|
|
BlockDriver *drv = bs->drv;
|
2015-11-09 13:16:51 +03:00
|
|
|
CoroutineIOCompletion co = {
|
|
|
|
.coroutine = qemu_coroutine_self(),
|
|
|
|
};
|
|
|
|
BlockAIOCB *acb;
|
2015-04-28 16:27:52 +03:00
|
|
|
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
2016-10-20 16:07:27 +03:00
|
|
|
if (!drv || (!drv->bdrv_aio_ioctl && !drv->bdrv_co_ioctl)) {
|
2015-11-09 13:16:51 +03:00
|
|
|
co.ret = -ENOTSUP;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2016-10-20 16:07:27 +03:00
|
|
|
if (drv->bdrv_co_ioctl) {
|
|
|
|
co.ret = drv->bdrv_co_ioctl(bs, req, buf);
|
|
|
|
} else {
|
|
|
|
acb = drv->bdrv_aio_ioctl(bs, req, buf, bdrv_co_io_em_complete, &co);
|
|
|
|
if (!acb) {
|
|
|
|
co.ret = -ENOTSUP;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
qemu_coroutine_yield();
|
2015-11-09 13:16:51 +03:00
|
|
|
}
|
|
|
|
out:
|
2016-10-27 13:48:52 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2015-11-09 13:16:51 +03:00
|
|
|
return co.ret;
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:27:52 +03:00
|
|
|
void *qemu_blockalign(BlockDriverState *bs, size_t size)
|
|
|
|
{
|
|
|
|
return qemu_memalign(bdrv_opt_mem_align(bs), size);
|
|
|
|
}
|
|
|
|
|
|
|
|
void *qemu_blockalign0(BlockDriverState *bs, size_t size)
|
|
|
|
{
|
|
|
|
return memset(qemu_blockalign(bs, size), 0, size);
|
|
|
|
}
|
|
|
|
|
|
|
|
void *qemu_try_blockalign(BlockDriverState *bs, size_t size)
|
|
|
|
{
|
|
|
|
size_t align = bdrv_opt_mem_align(bs);
|
|
|
|
|
|
|
|
/* Ensure that NULL is never returned on success */
|
|
|
|
assert(align > 0);
|
|
|
|
if (size == 0) {
|
|
|
|
size = align;
|
|
|
|
}
|
|
|
|
|
|
|
|
return qemu_try_memalign(align, size);
|
|
|
|
}
|
|
|
|
|
|
|
|
void *qemu_try_blockalign0(BlockDriverState *bs, size_t size)
|
|
|
|
{
|
|
|
|
void *mem = qemu_try_blockalign(bs, size);
|
|
|
|
|
|
|
|
if (mem) {
|
|
|
|
memset(mem, 0, size);
|
|
|
|
}
|
|
|
|
|
|
|
|
return mem;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if all memory in this vector is sector aligned.
|
|
|
|
*/
|
|
|
|
bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov)
|
|
|
|
{
|
|
|
|
int i;
|
2015-05-12 17:30:55 +03:00
|
|
|
size_t alignment = bdrv_min_mem_align(bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
|
|
|
|
for (i = 0; i < qiov->niov; i++) {
|
|
|
|
if ((uintptr_t) qiov->iov[i].iov_base % alignment) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
if (qiov->iov[i].iov_len % alignment) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_add_before_write_notifier(BlockDriverState *bs,
|
|
|
|
NotifierWithReturn *notifier)
|
|
|
|
{
|
|
|
|
notifier_with_return_list_add(&bs->before_write_notifiers, notifier);
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_io_plug(BlockDriverState *bs)
|
|
|
|
{
|
2016-04-07 19:33:34 +03:00
|
|
|
BdrvChild *child;
|
|
|
|
|
|
|
|
QLIST_FOREACH(child, &bs->children, next) {
|
|
|
|
bdrv_io_plug(child->bs);
|
|
|
|
}
|
|
|
|
|
2017-06-05 15:38:55 +03:00
|
|
|
if (atomic_fetch_inc(&bs->io_plugged) == 0) {
|
2016-04-07 19:33:34 +03:00
|
|
|
BlockDriver *drv = bs->drv;
|
|
|
|
if (drv && drv->bdrv_io_plug) {
|
|
|
|
drv->bdrv_io_plug(bs);
|
|
|
|
}
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_io_unplug(BlockDriverState *bs)
|
|
|
|
{
|
2016-04-07 19:33:34 +03:00
|
|
|
BdrvChild *child;
|
|
|
|
|
|
|
|
assert(bs->io_plugged);
|
2017-06-05 15:38:55 +03:00
|
|
|
if (atomic_fetch_dec(&bs->io_plugged) == 1) {
|
2016-04-07 19:33:34 +03:00
|
|
|
BlockDriver *drv = bs->drv;
|
|
|
|
if (drv && drv->bdrv_io_unplug) {
|
|
|
|
drv->bdrv_io_unplug(bs);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
QLIST_FOREACH(child, &bs->children, next) {
|
|
|
|
bdrv_io_unplug(child->bs);
|
2015-04-28 16:27:52 +03:00
|
|
|
}
|
|
|
|
}
|
2018-01-16 09:08:56 +03:00
|
|
|
|
|
|
|
void bdrv_register_buf(BlockDriverState *bs, void *host, size_t size)
|
|
|
|
{
|
|
|
|
BdrvChild *child;
|
|
|
|
|
|
|
|
if (bs->drv && bs->drv->bdrv_register_buf) {
|
|
|
|
bs->drv->bdrv_register_buf(bs, host, size);
|
|
|
|
}
|
|
|
|
QLIST_FOREACH(child, &bs->children, next) {
|
|
|
|
bdrv_register_buf(child->bs, host, size);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void bdrv_unregister_buf(BlockDriverState *bs, void *host)
|
|
|
|
{
|
|
|
|
BdrvChild *child;
|
|
|
|
|
|
|
|
if (bs->drv && bs->drv->bdrv_unregister_buf) {
|
|
|
|
bs->drv->bdrv_unregister_buf(bs, host);
|
|
|
|
}
|
|
|
|
QLIST_FOREACH(child, &bs->children, next) {
|
|
|
|
bdrv_unregister_buf(child->bs, host);
|
|
|
|
}
|
|
|
|
}
|
2018-06-01 12:26:39 +03:00
|
|
|
|
2018-07-09 19:37:17 +03:00
|
|
|
static int coroutine_fn bdrv_co_copy_range_internal(
|
|
|
|
BdrvChild *src, uint64_t src_offset, BdrvChild *dst,
|
|
|
|
uint64_t dst_offset, uint64_t bytes,
|
|
|
|
BdrvRequestFlags read_flags, BdrvRequestFlags write_flags,
|
|
|
|
bool recurse_src)
|
2018-06-01 12:26:39 +03:00
|
|
|
{
|
2018-07-09 19:37:16 +03:00
|
|
|
BdrvTrackedRequest req;
|
2018-06-01 12:26:39 +03:00
|
|
|
int ret;
|
|
|
|
|
2019-03-22 15:38:43 +03:00
|
|
|
/* TODO We can support BDRV_REQ_NO_FALLBACK here */
|
|
|
|
assert(!(read_flags & BDRV_REQ_NO_FALLBACK));
|
|
|
|
assert(!(write_flags & BDRV_REQ_NO_FALLBACK));
|
|
|
|
|
2018-07-03 05:37:56 +03:00
|
|
|
if (!dst || !dst->bs) {
|
2018-06-01 12:26:39 +03:00
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
ret = bdrv_check_byte_request(dst->bs, dst_offset, bytes);
|
|
|
|
if (ret) {
|
|
|
|
return ret;
|
|
|
|
}
|
2018-07-09 19:37:17 +03:00
|
|
|
if (write_flags & BDRV_REQ_ZERO_WRITE) {
|
|
|
|
return bdrv_co_pwrite_zeroes(dst, dst_offset, bytes, write_flags);
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
|
|
|
|
2018-07-03 05:37:56 +03:00
|
|
|
if (!src || !src->bs) {
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
ret = bdrv_check_byte_request(src->bs, src_offset, bytes);
|
|
|
|
if (ret) {
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-06-01 12:26:39 +03:00
|
|
|
if (!src->bs->drv->bdrv_co_copy_range_from
|
|
|
|
|| !dst->bs->drv->bdrv_co_copy_range_to
|
|
|
|
|| src->bs->encrypted || dst->bs->encrypted) {
|
|
|
|
return -ENOTSUP;
|
|
|
|
}
|
2018-06-27 06:57:52 +03:00
|
|
|
|
2018-06-01 12:26:39 +03:00
|
|
|
if (recurse_src) {
|
2018-07-09 19:37:16 +03:00
|
|
|
bdrv_inc_in_flight(src->bs);
|
|
|
|
tracked_request_begin(&req, src->bs, src_offset, bytes,
|
|
|
|
BDRV_TRACKED_READ);
|
|
|
|
|
2018-07-09 19:37:18 +03:00
|
|
|
/* BDRV_REQ_SERIALISING is only for write operation */
|
|
|
|
assert(!(read_flags & BDRV_REQ_SERIALISING));
|
2020-01-08 17:55:54 +03:00
|
|
|
bdrv_wait_serialising_requests(&req);
|
2018-07-09 19:37:16 +03:00
|
|
|
|
2018-06-27 06:57:52 +03:00
|
|
|
ret = src->bs->drv->bdrv_co_copy_range_from(src->bs,
|
|
|
|
src, src_offset,
|
|
|
|
dst, dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
bytes,
|
|
|
|
read_flags, write_flags);
|
2018-07-09 19:37:16 +03:00
|
|
|
|
|
|
|
tracked_request_end(&req);
|
|
|
|
bdrv_dec_in_flight(src->bs);
|
2018-06-01 12:26:39 +03:00
|
|
|
} else {
|
2018-07-09 19:37:16 +03:00
|
|
|
bdrv_inc_in_flight(dst->bs);
|
|
|
|
tracked_request_begin(&req, dst->bs, dst_offset, bytes,
|
|
|
|
BDRV_TRACKED_WRITE);
|
2018-07-10 09:31:22 +03:00
|
|
|
ret = bdrv_co_write_req_prepare(dst, dst_offset, bytes, &req,
|
|
|
|
write_flags);
|
|
|
|
if (!ret) {
|
|
|
|
ret = dst->bs->drv->bdrv_co_copy_range_to(dst->bs,
|
|
|
|
src, src_offset,
|
|
|
|
dst, dst_offset,
|
|
|
|
bytes,
|
|
|
|
read_flags, write_flags);
|
|
|
|
}
|
|
|
|
bdrv_co_write_req_finish(dst, dst_offset, bytes, &req, ret);
|
2018-07-09 19:37:16 +03:00
|
|
|
tracked_request_end(&req);
|
|
|
|
bdrv_dec_in_flight(dst->bs);
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
2018-07-09 19:37:16 +03:00
|
|
|
|
2018-06-27 06:57:52 +03:00
|
|
|
return ret;
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Copy range from @src to @dst.
|
|
|
|
*
|
|
|
|
* See the comment of bdrv_co_copy_range for the parameter and return value
|
|
|
|
* semantics. */
|
|
|
|
int coroutine_fn bdrv_co_copy_range_from(BdrvChild *src, uint64_t src_offset,
|
|
|
|
BdrvChild *dst, uint64_t dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
uint64_t bytes,
|
|
|
|
BdrvRequestFlags read_flags,
|
|
|
|
BdrvRequestFlags write_flags)
|
2018-06-01 12:26:39 +03:00
|
|
|
{
|
2018-07-10 09:31:16 +03:00
|
|
|
trace_bdrv_co_copy_range_from(src, src_offset, dst, dst_offset, bytes,
|
|
|
|
read_flags, write_flags);
|
2018-06-01 12:26:39 +03:00
|
|
|
return bdrv_co_copy_range_internal(src, src_offset, dst, dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
bytes, read_flags, write_flags, true);
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Copy range from @src to @dst.
|
|
|
|
*
|
|
|
|
* See the comment of bdrv_co_copy_range for the parameter and return value
|
|
|
|
* semantics. */
|
|
|
|
int coroutine_fn bdrv_co_copy_range_to(BdrvChild *src, uint64_t src_offset,
|
|
|
|
BdrvChild *dst, uint64_t dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
uint64_t bytes,
|
|
|
|
BdrvRequestFlags read_flags,
|
|
|
|
BdrvRequestFlags write_flags)
|
2018-06-01 12:26:39 +03:00
|
|
|
{
|
2018-07-10 09:31:16 +03:00
|
|
|
trace_bdrv_co_copy_range_to(src, src_offset, dst, dst_offset, bytes,
|
|
|
|
read_flags, write_flags);
|
2018-06-01 12:26:39 +03:00
|
|
|
return bdrv_co_copy_range_internal(src, src_offset, dst, dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
bytes, read_flags, write_flags, false);
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
int coroutine_fn bdrv_co_copy_range(BdrvChild *src, uint64_t src_offset,
|
|
|
|
BdrvChild *dst, uint64_t dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
uint64_t bytes, BdrvRequestFlags read_flags,
|
|
|
|
BdrvRequestFlags write_flags)
|
2018-06-01 12:26:39 +03:00
|
|
|
{
|
2018-06-27 06:57:52 +03:00
|
|
|
return bdrv_co_copy_range_from(src, src_offset,
|
|
|
|
dst, dst_offset,
|
2018-07-09 19:37:17 +03:00
|
|
|
bytes, read_flags, write_flags);
|
2018-06-01 12:26:39 +03:00
|
|
|
}
|
2018-06-26 14:55:20 +03:00
|
|
|
|
|
|
|
static void bdrv_parent_cb_resize(BlockDriverState *bs)
|
|
|
|
{
|
|
|
|
BdrvChild *c;
|
|
|
|
QLIST_FOREACH(c, &bs->parents, next_parent) {
|
|
|
|
if (c->role->resize) {
|
|
|
|
c->role->resize(c);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Truncate file to 'offset' bytes (needed only for file protocols)
|
2019-09-18 12:51:40 +03:00
|
|
|
*
|
|
|
|
* If 'exact' is true, the file must be resized to exactly the given
|
|
|
|
* 'offset'. Otherwise, it is sufficient for the node to be at least
|
|
|
|
* 'offset' bytes in length.
|
2018-06-26 14:55:20 +03:00
|
|
|
*/
|
2019-09-18 12:51:40 +03:00
|
|
|
int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
|
2018-06-26 14:55:20 +03:00
|
|
|
PreallocMode prealloc, Error **errp)
|
|
|
|
{
|
|
|
|
BlockDriverState *bs = child->bs;
|
|
|
|
BlockDriver *drv = bs->drv;
|
2018-06-26 15:23:23 +03:00
|
|
|
BdrvTrackedRequest req;
|
|
|
|
int64_t old_size, new_bytes;
|
2018-06-26 14:55:20 +03:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
|
|
|
|
/* if bs->drv == NULL, bs is closed, so there's nothing to do here */
|
|
|
|
if (!drv) {
|
|
|
|
error_setg(errp, "No medium inserted");
|
|
|
|
return -ENOMEDIUM;
|
|
|
|
}
|
|
|
|
if (offset < 0) {
|
|
|
|
error_setg(errp, "Image size cannot be negative");
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2018-06-26 15:23:23 +03:00
|
|
|
old_size = bdrv_getlength(bs);
|
|
|
|
if (old_size < 0) {
|
|
|
|
error_setg_errno(errp, -old_size, "Failed to get old image size");
|
|
|
|
return old_size;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (offset > old_size) {
|
|
|
|
new_bytes = offset - old_size;
|
|
|
|
} else {
|
|
|
|
new_bytes = 0;
|
|
|
|
}
|
|
|
|
|
2018-06-26 14:55:20 +03:00
|
|
|
bdrv_inc_in_flight(bs);
|
2018-07-10 09:31:23 +03:00
|
|
|
tracked_request_begin(&req, bs, offset - new_bytes, new_bytes,
|
|
|
|
BDRV_TRACKED_TRUNCATE);
|
2018-06-26 15:23:23 +03:00
|
|
|
|
|
|
|
/* If we are growing the image and potentially using preallocation for the
|
|
|
|
* new area, we need to make sure that no write requests are made to it
|
|
|
|
* concurrently or they might be overwritten by preallocation. */
|
|
|
|
if (new_bytes) {
|
2019-11-01 18:25:08 +03:00
|
|
|
bdrv_mark_request_serialising(&req, 1);
|
2018-07-10 09:31:24 +03:00
|
|
|
}
|
|
|
|
if (bs->read_only) {
|
|
|
|
error_setg(errp, "Image is read-only");
|
|
|
|
ret = -EACCES;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
ret = bdrv_co_write_req_prepare(child, offset - new_bytes, new_bytes, &req,
|
|
|
|
0);
|
|
|
|
if (ret < 0) {
|
|
|
|
error_setg_errno(errp, -ret,
|
|
|
|
"Failed to prepare request for truncation");
|
|
|
|
goto out;
|
2018-06-26 15:23:23 +03:00
|
|
|
}
|
2018-06-26 14:55:20 +03:00
|
|
|
|
2019-09-18 12:51:37 +03:00
|
|
|
if (drv->bdrv_co_truncate) {
|
2019-09-18 12:51:40 +03:00
|
|
|
ret = drv->bdrv_co_truncate(bs, offset, exact, prealloc, errp);
|
2019-09-18 12:51:37 +03:00
|
|
|
} else if (bs->file && drv->is_filter) {
|
2019-09-18 12:51:40 +03:00
|
|
|
ret = bdrv_co_truncate(bs->file, offset, exact, prealloc, errp);
|
2019-09-18 12:51:37 +03:00
|
|
|
} else {
|
2018-06-26 14:55:20 +03:00
|
|
|
error_setg(errp, "Image format driver does not support resize");
|
|
|
|
ret = -ENOTSUP;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
if (ret < 0) {
|
|
|
|
goto out;
|
|
|
|
}
|
2019-09-18 12:51:37 +03:00
|
|
|
|
2018-06-26 14:55:20 +03:00
|
|
|
ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
|
|
|
|
if (ret < 0) {
|
|
|
|
error_setg_errno(errp, -ret, "Could not refresh total sector count");
|
|
|
|
} else {
|
|
|
|
offset = bs->total_sectors * BDRV_SECTOR_SIZE;
|
|
|
|
}
|
2018-07-10 09:31:24 +03:00
|
|
|
/* It's possible that truncation succeeded but refresh_total_sectors
|
|
|
|
* failed, but the latter doesn't affect how we should finish the request.
|
|
|
|
* Pass 0 as the last parameter so that dirty bitmaps etc. are handled. */
|
|
|
|
bdrv_co_write_req_finish(child, offset - new_bytes, new_bytes, &req, 0);
|
2018-06-26 14:55:20 +03:00
|
|
|
|
|
|
|
out:
|
2018-06-26 15:23:23 +03:00
|
|
|
tracked_request_end(&req);
|
2018-06-26 14:55:20 +03:00
|
|
|
bdrv_dec_in_flight(bs);
|
2018-06-26 15:23:23 +03:00
|
|
|
|
2018-06-26 14:55:20 +03:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
typedef struct TruncateCo {
|
|
|
|
BdrvChild *child;
|
|
|
|
int64_t offset;
|
2019-09-18 12:51:40 +03:00
|
|
|
bool exact;
|
2018-06-26 14:55:20 +03:00
|
|
|
PreallocMode prealloc;
|
|
|
|
Error **errp;
|
|
|
|
int ret;
|
|
|
|
} TruncateCo;
|
|
|
|
|
|
|
|
static void coroutine_fn bdrv_truncate_co_entry(void *opaque)
|
|
|
|
{
|
|
|
|
TruncateCo *tco = opaque;
|
2019-09-18 12:51:40 +03:00
|
|
|
tco->ret = bdrv_co_truncate(tco->child, tco->offset, tco->exact,
|
|
|
|
tco->prealloc, tco->errp);
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
aio_wait_kick();
|
2018-06-26 14:55:20 +03:00
|
|
|
}
|
|
|
|
|
2019-09-18 12:51:40 +03:00
|
|
|
int bdrv_truncate(BdrvChild *child, int64_t offset, bool exact,
|
|
|
|
PreallocMode prealloc, Error **errp)
|
2018-06-26 14:55:20 +03:00
|
|
|
{
|
|
|
|
Coroutine *co;
|
|
|
|
TruncateCo tco = {
|
|
|
|
.child = child,
|
|
|
|
.offset = offset,
|
2019-09-18 12:51:40 +03:00
|
|
|
.exact = exact,
|
2018-06-26 14:55:20 +03:00
|
|
|
.prealloc = prealloc,
|
|
|
|
.errp = errp,
|
|
|
|
.ret = NOT_DONE,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (qemu_in_coroutine()) {
|
|
|
|
/* Fast-path if already in coroutine context */
|
|
|
|
bdrv_truncate_co_entry(&tco);
|
|
|
|
} else {
|
|
|
|
co = qemu_coroutine_create(bdrv_truncate_co_entry, &tco);
|
block: Fix hangs in synchronous APIs with iothreads
In the block layer, synchronous APIs are often implemented by creating a
coroutine that calls the asynchronous coroutine-based implementation and
then waiting for completion with BDRV_POLL_WHILE().
For this to work with iothreads (more specifically, when the synchronous
API is called in a thread that is not the home thread of the block
device, so that the coroutine will run in a different thread), we must
make sure to call aio_wait_kick() at the end of the operation. Many
places are missing this, so that BDRV_POLL_WHILE() keeps hanging even if
the condition has long become false.
Note that bdrv_dec_in_flight() involves an aio_wait_kick() call. This
corresponds to the BDRV_POLL_WHILE() in the drain functions, but it is
generally not enough for most other operations because they haven't set
the return value in the coroutine entry stub yet. To avoid race
conditions there, we need to kick after setting the return value.
The race window is small enough that the problem doesn't usually surface
in the common path. However, it does surface and causes easily
reproducible hangs if the operation can return early before even calling
bdrv_inc/dec_in_flight, which many of them do (trivial error or no-op
success paths).
The bug in bdrv_truncate(), bdrv_check() and bdrv_invalidate_cache() is
slightly different: These functions even neglected to schedule the
coroutine in the home thread of the node. This avoids the hang, but is
obviously wrong, too. Fix those to schedule the coroutine in the right
AioContext in addition to adding aio_wait_kick() calls.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
2019-01-07 15:02:48 +03:00
|
|
|
bdrv_coroutine_enter(child->bs, co);
|
2018-06-26 14:55:20 +03:00
|
|
|
BDRV_POLL_WHILE(child->bs, tco.ret == NOT_DONE);
|
|
|
|
}
|
|
|
|
|
|
|
|
return tco.ret;
|
|
|
|
}
|