2020-04-24 15:54:47 +03:00
|
|
|
== Commit tests ==
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 2097152/2097152 bytes at offset 0
|
|
|
|
|
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Check visible data ===
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Checking allocation status ===
|
|
|
|
|
1048576/1048576 bytes allocated at offset 0 bytes
|
|
|
|
|
1048576/1048576 bytes allocated at offset 1 MiB
|
|
|
|
|
|
|
|
|
|
0/1048576 bytes allocated at offset 0 bytes
|
|
|
|
|
0/0 bytes allocated at offset 1 MiB
|
|
|
|
|
|
|
|
|
|
0/1048576 bytes allocated at offset 0 bytes
|
|
|
|
|
0/1048576 bytes allocated at offset 1 MiB
|
|
|
|
|
|
|
|
|
|
=== Checking map ===
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
Offset Length Mapped to File
|
|
|
|
|
0 0x200000 0x50000 TEST_DIR/PID-base
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 1048576, "depth": 1, "present": true, "zero": false, "data": true, "offset": 327680}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
Offset Length Mapped to File
|
|
|
|
|
0 0x100000 0x50000 TEST_DIR/PID-base
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 1048576, "depth": 2, "present": true, "zero": false, "data": true, "offset": 327680},
|
|
|
|
|
{ "start": 1048576, "length": 1048576, "depth": 0, "present": false, "zero": true, "data": false}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
Offset Length Mapped to File
|
|
|
|
|
0 0x100000 0x50000 TEST_DIR/PID-base
|
|
|
|
|
|
|
|
|
|
=== Testing qemu-img commit (top -> mid) ===
|
|
|
|
|
Image committed.
|
|
|
|
|
|
|
|
|
|
image: TEST_IMG
|
|
|
|
|
file format: IMGFMT
|
|
|
|
|
virtual size: 2 MiB (2097152 bytes)
|
|
|
|
|
cluster_size: 65536
|
|
|
|
|
backing file: TEST_DIR/PID-base
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 23:39:52 +03:00
|
|
|
backing file format: IMGFMT
|
2020-04-24 15:54:47 +03:00
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 11:25:18 +03:00
|
|
|
compression type: zlib
|
2020-04-24 15:54:47 +03:00
|
|
|
lazy refcounts: false
|
|
|
|
|
refcount bits: 16
|
|
|
|
|
corrupt: false
|
2020-07-10 19:13:13 +03:00
|
|
|
extended l2: false
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Testing HMP commit (top -> mid) ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 2097152/2097152 bytes at offset 0
|
|
|
|
|
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
{"execute": "human-monitor-command", "arguments": {"command-line": "commit drive0"}}
|
|
|
|
|
{"return": ""}
|
|
|
|
|
image: TEST_IMG
|
|
|
|
|
file format: IMGFMT
|
|
|
|
|
virtual size: 2 MiB (2097152 bytes)
|
|
|
|
|
cluster_size: 65536
|
|
|
|
|
backing file: TEST_DIR/PID-base
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 23:39:52 +03:00
|
|
|
backing file format: IMGFMT
|
2020-04-24 15:54:47 +03:00
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 11:25:18 +03:00
|
|
|
compression type: zlib
|
2020-04-24 15:54:47 +03:00
|
|
|
lazy refcounts: false
|
|
|
|
|
refcount bits: 16
|
|
|
|
|
corrupt: false
|
2020-07-10 19:13:13 +03:00
|
|
|
extended l2: false
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Testing QMP active commit (top -> mid) ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 2097152/2097152 bytes at offset 0
|
|
|
|
|
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
{"execute": "block-commit", "arguments": {"auto-dismiss": false, "base-node": "mid", "device": "top", "job-id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
{"execute": "job-complete", "arguments": {"id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
{"data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
|
|
|
|
|
{"data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
|
|
|
|
|
{"execute": "job-dismiss", "arguments": {"id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
image: TEST_IMG
|
|
|
|
|
file format: IMGFMT
|
|
|
|
|
virtual size: 2 MiB (2097152 bytes)
|
|
|
|
|
cluster_size: 65536
|
|
|
|
|
backing file: TEST_DIR/PID-base
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 23:39:52 +03:00
|
|
|
backing file format: IMGFMT
|
2020-09-24 22:40:03 +03:00
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
|
|
|
|
compression type: zlib
|
|
|
|
|
lazy refcounts: false
|
|
|
|
|
refcount bits: 16
|
|
|
|
|
corrupt: false
|
|
|
|
|
extended l2: false
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Testing qemu-img commit (top -> base) ===
|
|
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
wrote 2097152/2097152 bytes at offset 0
|
|
|
|
|
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image committed.
|
|
|
|
|
|
|
|
|
|
image: TEST_IMG
|
|
|
|
|
file format: IMGFMT
|
|
|
|
|
virtual size: 2 MiB (2097152 bytes)
|
|
|
|
|
cluster_size: 65536
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
|
|
|
|
compression type: zlib
|
|
|
|
|
lazy refcounts: false
|
|
|
|
|
refcount bits: 16
|
|
|
|
|
corrupt: false
|
|
|
|
|
extended l2: false
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
=== Testing QMP active commit (top -> base) ===
|
|
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
|
|
|
|
|
|
|
|
|
wrote 2097152/2097152 bytes at offset 0
|
|
|
|
|
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
{"execute": "block-commit", "arguments": {"auto-dismiss": false, "base-node": "base", "device": "top", "job-id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
{"execute": "job-complete", "arguments": {"id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
{"data": {"device": "job0", "len": 1048576, "offset": 1048576, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
|
|
|
|
|
{"data": {"device": "job0", "len": 1048576, "offset": 1048576, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
|
|
|
|
|
{"execute": "job-dismiss", "arguments": {"id": "job0"}}
|
|
|
|
|
{"return": {}}
|
|
|
|
|
image: TEST_IMG
|
|
|
|
|
file format: IMGFMT
|
|
|
|
|
virtual size: 1 MiB (1048576 bytes)
|
|
|
|
|
cluster_size: 65536
|
|
|
|
|
backing file: TEST_DIR/PID-base
|
|
|
|
|
backing file format: IMGFMT
|
2020-04-24 15:54:47 +03:00
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 11:25:18 +03:00
|
|
|
compression type: zlib
|
2020-04-24 15:54:47 +03:00
|
|
|
lazy refcounts: false
|
|
|
|
|
refcount bits: 16
|
|
|
|
|
corrupt: false
|
2020-07-10 19:13:13 +03:00
|
|
|
extended l2: false
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 0
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
read 1048576/1048576 bytes at offset 1048576
|
|
|
|
|
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
== Resize tests ==
|
|
|
|
|
=== preallocation=off ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 5368709120
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 5368709120
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
7 GiB (0x1c0000000) bytes allocated at offset 1 GiB (0x40000000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 1073741824, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 1073741824, "length": 7516192768, "depth": 0, "present": true, "zero": true, "data": false}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=metadata ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=34359738368 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=32212254720 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 33285996544
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 33285996544
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
3 GiB (0xc0000000) bytes allocated at offset 30 GiB (0x780000000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 32212254720, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 32212254720, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 327680},
|
|
|
|
|
{ "start": 32749125632, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 537264128},
|
|
|
|
|
{ "start": 33285996544, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 1074200576},
|
|
|
|
|
{ "start": 33822867456, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 1611137024},
|
|
|
|
|
{ "start": 34359738368, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 2148139008},
|
|
|
|
|
{ "start": 34896609280, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 2685075456}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=falloc ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10485760 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=5242880 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 9437184
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 9437184
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
10 MiB (0xa00000) bytes allocated at offset 5 MiB (0x500000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 5242880, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 5242880, "length": 10485760, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=full ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=16777216 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=8388608 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 11534336
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 11534336
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
4 MiB (0x400000) bytes allocated at offset 8 MiB (0x800000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 8388608, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 8388608, "length": 4194304, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=off ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=393216 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=259072 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 259072
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 259072
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
320 KiB (0x50000) bytes allocated at offset 192 KiB (0x30000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 196608, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 196608, "length": 65536, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680},
|
|
|
|
|
{ "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=off ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=409600 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=262144 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 344064
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 344064
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
256 KiB (0x40000) bytes allocated at offset 256 KiB (0x40000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 262144, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
=== preallocation=off ===
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=524288 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
2020-07-10 19:13:13 +03:00
|
|
|
Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=262144 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2020-04-24 15:54:47 +03:00
|
|
|
|
|
|
|
|
wrote 65536/65536 bytes at offset 446464
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
Image resized.
|
|
|
|
|
|
|
|
|
|
read 65536/65536 bytes at offset 446464
|
|
|
|
|
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
|
|
|
|
|
|
|
|
|
|
256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
|
|
|
|
|
244 KiB (0x3d000) bytes allocated at offset 256 KiB (0x40000)
|
|
|
|
|
|
qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 22:06:55 +03:00
|
|
|
[{ "start": 0, "length": 262144, "depth": 1, "present": false, "zero": true, "data": false},
|
|
|
|
|
{ "start": 262144, "length": 249856, "depth": 0, "present": true, "zero": true, "data": false}]
|
2020-04-24 15:54:47 +03:00
|
|
|
|