qemu/block
Hanna Reitz 0bc329fbb0 block: block-status cache for data regions
As we have attempted before
(https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06451.html,
"file-posix: Cache lseek result for data regions";
https://lists.nongnu.org/archive/html/qemu-block/2021-02/msg00934.html,
"file-posix: Cache next hole"), this patch seeks to reduce the number of
SEEK_DATA/HOLE operations the file-posix driver has to perform.  The
main difference is that this time it is implemented as part of the
general block layer code.

The problem we face is that on some filesystems or in some
circumstances, SEEK_DATA/HOLE is unreasonably slow.  Given the
implementation is outside of qemu, there is little we can do about its
performance.

We have already introduced the want_zero parameter to
bdrv_co_block_status() to reduce the number of SEEK_DATA/HOLE calls
unless we really want zero information; but sometimes we do want that
information, because for files that consist largely of zero areas,
special-casing those areas can give large performance boosts.  So the
real problem is with files that consist largely of data, so that
inquiring the block status does not gain us much performance, but where
such an inquiry itself takes a lot of time.

To address this, we want to cache data regions.  Most of the time, when
bad performance is reported, it is in places where the image is iterated
over from start to end (qemu-img convert or the mirror job), so a simple
yet effective solution is to cache only the current data region.

(Note that only caching data regions but not zero regions means that
returning false information from the cache is not catastrophic: Treating
zeroes as data is fine.  While we try to invalidate the cache on zero
writes and discards, such incongruences may still occur when there are
other processes writing to the image.)

We only use the cache for nodes without children (i.e. protocol nodes),
because that is where the problem is: Drivers that rely on block-status
implementations outside of qemu (e.g. SEEK_DATA/HOLE).

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/307
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20210812084148.14458-3-hreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[hreitz: Added `local_file == bs` assertion, as suggested by Vladimir]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-15 15:54:06 +02:00
..
export block/export/fuse.c: fix fuse-lseek on uclibc or musl 2021-09-01 14:38:08 +02:00
monitor block/monitor: Consolidate hmp_handle_error calls to reduce redundant code 2021-09-01 12:57:31 +02:00
accounting.c block/accounting: Use lock guard macros 2020-12-11 17:52:39 +01:00
aio_task.c block: introduce aio task pool 2019-10-10 10:56:17 +02:00
amend.c block/amend: Check whether the node exists 2020-07-27 12:37:25 +02:00
backup.c block/copy-before-write: initialize block-copy bitmap 2021-09-01 14:03:47 +02:00
blkdebug.c blkdebug: protect rules and suspended_reqs with a lock 2021-07-19 17:38:38 +02:00
blklogwrites.c block: check return value of bdrv_open_child and drop error propagation 2021-03-08 15:07:09 -06:00
blkreplay.c block: check return value of bdrv_open_child and drop error propagation 2021-03-08 15:07:09 -06:00
blkverify.c block: check return value of bdrv_open_child and drop error propagation 2021-03-08 15:07:09 -06:00
block-backend.c block: introduce blk_replace_bs 2021-09-01 12:57:31 +02:00
block-copy.c block/block-copy: block_copy_state_new(): drop extra arguments 2021-09-01 14:38:08 +02:00
block-gen.h scripts: add block-coroutine-wrapper.py 2020-10-05 10:59:06 +01:00
bochs.c block: Use bdrv_default_perms() 2020-05-18 19:05:25 +02:00
cloop.c block: Use bdrv_default_perms() 2020-05-18 19:05:25 +02:00
commit.c block/commit: use QEMU_AUTO_VFREE 2021-06-29 16:51:21 +02:00
copy-before-write.c block/block-copy: block_copy_state_new(): drop extra arguments 2021-09-01 14:38:08 +02:00
copy-before-write.h block/copy-before-write: bdrv_cbw_append(): drop unused compress arg 2021-09-01 14:03:47 +02:00
copy-on-read.c block/copy-on-read: use bdrv_drop_filter() and drop s->active 2021-05-14 16:14:10 +02:00
copy-on-read.h copy-on-read: add filter drop function 2021-01-26 11:26:54 +01:00
coroutines.h block/nbd: reuse nbd_co_do_establish_connection() in nbd_open() 2021-06-18 12:21:22 -05:00
create.c block/create: Do not abort if a block driver is not available 2019-09-13 12:18:37 +02:00
crypto.c block: add bdrv_co_delete_file_noerr 2021-02-15 15:10:14 +01:00
crypto.h nomaintainer: Fix Lesser GPL version number 2020-11-15 17:04:40 +01:00
curl.c curl: Disconnect sockets from CURLState 2021-03-19 10:15:06 +01:00
dirty-bitmap.c iotests: Improve and rename test 291 to qemu-img-bitmap 2021-07-21 14:14:41 -05:00
dmg-bz2.c Include qemu-common.h exactly where needed 2019-06-12 13:20:20 +02:00
dmg-lzfse.c block: Remove unused include 2020-11-09 15:44:21 +01:00
dmg.c block: Fix some code style problems, "foo* bar" should be "foo *bar" 2020-11-09 18:42:47 +01:00
dmg.h Include qemu-common.h exactly where needed 2019-06-12 13:20:20 +02:00
file-posix.c block/file-posix: Optimize for macOS 2021-07-06 14:28:55 +01:00
file-win32.c block/file-win32: add reopen handlers 2021-09-01 14:38:08 +02:00
filter-compress.c block: Inline bdrv_co_block_status_from_*() 2020-09-07 12:31:31 +02:00
gluster.c gluster: Align block-status tail 2021-09-15 15:54:06 +02:00
io_uring.c block/io_uring: resubmit when result is -EAGAIN 2021-07-29 17:14:55 +01:00
io.c block: block-status cache for data regions 2021-09-15 15:54:06 +02:00
iscsi-opts.c modules: add block module annotations 2021-07-09 18:20:27 +02:00
iscsi.c Remove superfluous timer_del() calls 2021-01-08 15:13:38 +00:00
linux-aio.c linux-aio: limit the batch size using aio-max-batch parameter 2021-07-21 13:47:50 +01:00
meson.build block: rename backup-top to copy-before-write 2021-09-01 12:57:31 +02:00
mirror.c block/mirror: fix active mirror dead-lock in mirror_wait_on_conflicts 2021-07-20 13:14:45 +02:00
nbd.c nbd: register yank function earlier 2021-07-12 11:24:00 -05:00
nfs.c util/uri: do not check argument of uri_free() 2021-07-09 12:26:05 +02:00
null.c block/null: Implement bdrv_get_allocated_file_size 2020-09-07 12:31:31 +02:00
nvme.c block/nvme: Only report VFIO error on failed retry 2021-09-07 09:08:24 +01:00
parallels-ext.c parallels: support bitmap extension for read-only mode 2021-03-08 14:56:55 +01:00
parallels.c parallels: support bitmap extension for read-only mode 2021-03-08 14:56:55 +01:00
parallels.h parallels: support bitmap extension for read-only mode 2021-03-08 14:56:55 +01:00
preallocate.c block: introduce preallocate filter 2020-12-18 12:35:55 +01:00
progress_meter.c progressmeter: protect with a mutex 2021-06-25 14:24:24 +03:00
qapi-sysemu.c block: Move system emulator QMP commands to block/qapi-sysemu.c 2020-03-06 17:15:38 +01:00
qapi.c block: use GDateTime for formatting timestamp when dumping snapshot info 2021-06-14 13:28:50 +01:00
qcow2-bitmap.c nbd patches for 2021-03-09 2021-03-11 13:57:08 +00:00
qcow2-cache.c core: replace getpagesize() with qemu_real_host_page_size 2019-10-26 15:38:06 +02:00
qcow2-cluster.c qcow2: Fix corruption on write_zeroes with MAY_UNMAP 2020-11-24 11:29:41 +01:00
qcow2-refcount.c qcow2: Make qcow2_free_any_clusters() free only one cluster 2020-09-15 11:05:13 +02:00
qcow2-snapshot.c block: consistently use bdrv_is_read_only() 2021-06-02 14:23:20 +02:00
qcow2-threads.c qcow2: add zstd cluster compression 2020-05-13 14:20:31 +02:00
qcow2.c qcow2: Fix dangling pointer after reopen for 'file' 2021-07-09 13:19:11 +02:00
qcow2.h block/qcow2-bitmap: return status from qcow2_store_persistent_dirty_bitmaps 2021-03-08 16:03:21 -06:00
qcow.c block/qcow: remove runtime opts 2020-09-15 11:05:13 +02:00
qed-check.c block/qed: add missed coroutine_fn markers 2019-04-30 15:29:00 +02:00
qed-cluster.c
qed-l2-cache.c
qed-table.c block/qed: add missed coroutine_fn markers 2019-04-30 15:29:00 +02:00
qed.c block/qed: bdrv_qed_do_open: deal with errp 2021-03-08 16:03:32 -06:00
qed.h qed: Simplify backing reads 2020-07-06 10:34:14 +02:00
quorum.c block/quorum: Provide .bdrv_co_flush instead of .bdrv_co_flush_to_disk 2021-06-02 14:23:20 +02:00
raw-format.c raw-format: drop WRITE and RESIZE child perms when possible 2021-09-01 12:57:31 +02:00
rbd.c block/rbd: fix type of task->complete 2021-07-09 12:26:05 +02:00
replication.c replication: Remove workaround 2021-07-20 16:11:53 +02:00
snapshot.c block/snapshot: Clarify goto fallback behavior 2021-06-24 09:49:04 +02:00
ssh.c util/uri: do not check argument of uri_free() 2021-07-09 12:26:05 +02:00
stream.c stream: Don't crash when node permission is denied 2021-03-19 10:15:06 +01:00
throttle-groups.c block/throttle-groups: throttle_group_co_io_limits_intercept(): 64bit bytes 2021-02-03 08:14:00 -06:00
throttle.c qemu/atomic.h: rename atomic_ to qatomic_ 2020-09-23 16:07:44 +01:00
trace-events block/nvme: Use safer trace format string 2021-09-07 09:08:24 +01:00
trace.h trace: switch position of headers to what Meson requires 2020-08-21 06:18:24 -04:00
vdi.c block/vdi: Don't assume that blocks are larger than VdiHeader 2021-03-31 10:44:21 +01:00
vhdx-endian.c Include qemu-common.h exactly where needed 2019-06-12 13:20:20 +02:00
vhdx-log.c block: consistently use bdrv_is_read_only() 2021-06-02 14:23:20 +02:00
vhdx.c block/vhdx: Support vhdx image only with 512 bytes logical sector size 2020-09-15 11:05:13 +02:00
vhdx.h block/vhdx: Use IEC binary prefixes for size constants 2019-04-30 15:29:00 +02:00
vmdk.c qapi: Use QAPI_LIST_APPEND in trivial cases 2021-01-28 08:08:45 +01:00
vpc.c block/vpc: Use sizeof() instead of HEADER_SIZE for footer size 2020-12-18 12:43:30 +01:00
vvfat.c block/vvfat: fix: drop backing 2021-07-20 16:30:20 +02:00
win32-aio.c Include qemu/module.h where needed, drop it from qemu-common.h 2019-06-12 13:18:33 +02:00
write-threshold.c write-threshold: deal with includes 2021-05-14 16:14:10 +02:00