Commit Graph

1925 Commits

Author SHA1 Message Date
Vladimir Sementsov-Ogievskiy
e2c5093c99 iotests: 30: drop from auto group (and effectively from make check)
I reproduced the following crash fast enough:

0  raise () at /lib64/libc.so.6
1  abort () at /lib64/libc.so.6
2  _nl_load_domain.cold () at /lib64/libc.so.6
3  annobin_assert.c_end () at /lib64/libc.so.6
4  bdrv_reopen_multiple (bs_queue=0x55de75fa9b70, errp=0x0)
   at ../block.c:3820
5  bdrv_reopen_set_read_only (bs=0x55de760fc020, read_only=true,
   errp=0x0) at ../block.c:3870
6  stream_clean (job=0x55de75fa9410) at ../block/stream.c:99
7  job_clean (job=0x55de75fa9410) at ../job.c:680
8  job_finalize_single (job=0x55de75fa9410) at ../job.c:696
9  job_txn_apply (job=0x55de75fa9410,
   fn=0x55de741eee27 <job_finalize_single>) at ../job.c:158
10 job_do_finalize (job=0x55de75fa9410) at ../job.c:805
11 job_completed_txn_success (job=0x55de75fa9410) at ../job.c:855
12 job_completed (job=0x55de75fa9410) at ../job.c:868
13 job_exit (opaque=0x55de75fa9410) at ../job.c:888
14 aio_bh_call (bh=0x55de76b9b4e0) at ../util/async.c:136
15 aio_bh_poll (ctx=0x55de75bc5300) at ../util/async.c:164
16 aio_dispatch (ctx=0x55de75bc5300) at ../util/aio-posix.c:381
17 aio_ctx_dispatch (source=0x55de75bc5300, callback=0x0,
   user_data=0x0) at ../util/async.c:306
18 g_main_context_dispatch () at /lib64/libglib-2.0.so.0
19 glib_pollfds_poll () at ../util/main-loop.c:232
20 os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:255
21 main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
22 qemu_main_loop () at ../softmmu/runstate.c:722
23 main (argc=20, argv=0x7ffe218f0268, envp=0x7ffe218f0310) at
   ../softmmu/main.c:50

(gdb) fr 4
4  bdrv_reopen_multiple (bs_queue=0x55de75fa9b70, errp=0x0) at
      ../block.c:3820
3820                assert(perm == state->perm);
(gdb) list
3815
3816            if (ret == 0) {
3817                uint64_t perm, shared;
3818
3819                bdrv_get_cumulative_perm(state->bs, &perm,
                    &shared);
3820                assert(perm == state->perm);
3821                assert(shared == state->shared_perm);
3822
3823                bdrv_set_perm(state->bs);
3824            } else {
(gdb) p perm
$1 = 1
(gdb) p state->perm
$2 = 0

Then I had 38 successful iterations and another crash:
0  bdrv_check_update_perm (bs=0x5631ac97bc50, q=0x0, new_used_perm=1,
   new_shared_perm=31, ignore_children=0x0, errp=0x7ffd9d477cf8) at
   ../block.c:2197
1  bdrv_root_attach_child
    (child_bs=0x5631ac97bc50, child_name=0x5631aaf6b1f9 "backing",
    child_class=0x5631ab280ca0 <child_of_bds>, child_role=8,
    ctx=0x5631ab757300, perm=1, shared_perm=31, opaque=0x5631abb8c020,
    errp=0x7ffd9d477cf8)
    at ../block.c:2642
2  bdrv_attach_child (parent_bs=0x5631abb8c020,
   child_bs=0x5631ac97bc50, child_name=0x5631aaf6b1f9 "backing",
   child_class=0x5631ab280ca0 <child_of_bds>, child_role=8,
   errp=0x7ffd9d477cf8)
    at ../block.c:2719
3  bdrv_set_backing_hd (bs=0x5631abb8c020, backing_hd=0x5631ac97bc50,
   errp=0x7ffd9d477cf8) at ../block.c:2854
4  stream_prepare (job=0x5631ac751eb0) at ../block/stream.c:74
5  job_prepare (job=0x5631ac751eb0) at ../job.c:784
6  job_txn_apply (job=0x5631ac751eb0, fn=0x5631aacb1156 <job_prepare>)
   at ../job.c:158
7  job_do_finalize (job=0x5631ac751eb0) at ../job.c:801
8  job_completed_txn_success (job=0x5631ac751eb0) at ../job.c:855
9  job_completed (job=0x5631ac751eb0) at ../job.c:868
10 job_exit (opaque=0x5631ac751eb0) at ../job.c:888
11 aio_bh_call (bh=0x7f3d9c007680) at ../util/async.c:136
12 aio_bh_poll (ctx=0x5631ab757300) at ../util/async.c:164
13 aio_dispatch (ctx=0x5631ab757300) at ../util/aio-posix.c:381
14 aio_ctx_dispatch (source=0x5631ab757300, callback=0x0,
   user_data=0x0) at ../util/async.c:306
15 g_main_context_dispatch () at /lib64/libglib-2.0.so.0
16 glib_pollfds_poll () at ../util/main-loop.c:232
17 os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:255
18 main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
19 qemu_main_loop () at ../softmmu/runstate.c:722
20 main (argc=20, argv=0x7ffd9d478198, envp=0x7ffd9d478240) at
   ../softmmu/main.c:50
(gdb) list
2192        QLIST_FOREACH(c, &bs->parents, next_parent) {
2193            if (g_slist_find(ignore_children, c)) {
2194                continue;
2195            }
2196
2197            if ((new_used_perm & c->shared_perm) != new_used_perm)
                {
2198                char *user = bdrv_child_user_desc(c);
2199                char *perm_names = bdrv_perm_names(new_used_perm &
                    ~c->shared_perm);
2200
2201                error_setg(errp, "Conflicts with use by %s as '%s',
                    which does not "
(gdb) p c
$1 = (BdrvChild *) 0x8585858585858585

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20210205111021.715240-1-vsementsov@virtuozzo.com
Reviewed-by: Eric Blake <eblake@redhat.com>
[PMM: trimmed the part of the commit message referring to
as-yet-unapplied patchseries]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-02-05 15:16:13 +00:00
Vladimir Sementsov-Ogievskiy
69b55e03f7 block: refactor bdrv_check_request: add errp
It's better to pass &error_abort than just assert that result is 0: on
crash, we'll immediately see the reason in the backtrace.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201211183934.169161-2-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: fix iotest 206 fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-02-03 08:00:33 -06:00
Eric Blake
c930831446 iotests: Fix expected whitespace for 185
Commit f93e19fb03 adjusted various iotest whitespace discrepancies.
But another one snuck in during 61623f8215, and we missed the
semantic merge conflict at the time because 185 is not run as part of
the default 'make check'.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210202185914.614705-1-eblake@redhat.com>
[eblake: adjust commit message]
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-02-03 08:00:33 -06:00
Kevin Wolf
0212fa2afd iotests: Fix -makecheck output
For -makecheck, the old 'check' implementation skipped the output when
starting a test. It only had the condensed output at the end of a test.

testrunner.py prints the normal output when starting a test even for
-makecheck. This output contains '\r' at the end so that it can be
overwritten with the result at the end of the test. However, for
-makecheck this is shorter output in a different format, so effectively
we end up with garbled output that mixes both output forms.

Revert to the old behaviour of only printing a message after the test
had completed in -makecheck mode.

Fixes: d74c754c92
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20210201161024.127921-1-kwolf@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-02-02 17:05:38 +01:00
Vladimir Sementsov-Ogievskiy
3ae50942f1 iotests: check: return 1 on failure
We should indicate failure by exit code, not only output.

Reported-by: Peter Maydell
Fixes: f203080bbd
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210201085041.3079-1-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-02-02 17:05:38 +01:00
Kevin Wolf
ca502ca60d iotests: Revert emulator selection to old behaviour
If the qemu-system-{arch} binary for the host architecture can't be
found, the old 'check' implementation selected the alphabetically first
system emulator binary that it could find. The new Python implementation
just uses the first result of glob.iglob(), which has an undefined
order.

This is a problem that breaks CI because the iotests aren't actually
prepared to run on any emulator. They should be, so this is really a bug
in the failing test cases that should be fixed there, but as a quick
fix, let's revert to the old behaviour to let CI runs succeed again.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20210202142802.119999-1-kwolf@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-02-02 17:05:04 +01:00
Vladimir Sementsov-Ogievskiy
d0bc412302 iotests/297: pylint: ignore too many statements
Ignore two complains, which now lead to 297 failure on testenv.py and
testrunner.py.

Fixes: 2e5a2f57db
Fixes: d74c754c92
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210129161323.615027-1-vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-02-02 13:23:47 +01:00
Peter Maydell
74208cd252 * Replace --enable/disable-git-update with --with-git-submodules
to allow improved control over use of git submodules
 * Deprecate the -enable-fips option
 * Ensure docs use prefer format for bool options
 * Clarify platform support rules
 * Misc fixes to keymap conversions
 * Fix misc problems on macOS
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE2vOm/bJrYpEtDo4/vobrtBUQT98FAmAUQXUACgkQvobrtBUQ
 T9/9Zw//VV3z7MI70HjWk17TUmkKLGFbmrc3xvtS6EMZbhfXpsR8qMfSjoRxP4Gn
 CgeftOa/lK5XjL7FcEUmcZeO8jZNE/+S5wgP2upP5Ae1zuDeyaSPTGR6UYs7WqSa
 P1kQsot2sAsZCP7Lko0jSv7rEDInK1bcdWr0a/xR5M2TvKiXoEdbElIZdwe3yLbC
 qWPWiv2pp6z2eGtJK/9VBxXjP0tlkNAR+jz8p/8RSZEEKf3aSq6HvVae+bHU86nX
 9t4baZk3ZuuwkmHJrHJP/72BjjitqEOiydtSsOCVYmY2GOclYeTnezG/vm75cGIK
 ej6hwFGElAEQk+KU9g7HOXi1eqXncSBjP0xWqc8PaZLW7wW/Uv8UV94fk+SCkIQ0
 GVBBLP8PzySL+w33NN7sv14cWeBpsjy9arMr4okZ80klFlS2hx/FqnUmhFhoN8ub
 ptIPUcSJoL+6pjxatV2yJ6bHkVga9PiwtLC8/rqETYPJz0b756xlPaEGd+0B1P0V
 ZMraE7lCgS0kJM6w0xF3R7JlGVtKouj4ZePCK5MGjpByBAK/vn/MbhbEmcyh+uJD
 r6sakKwqLjDLKdmQDd+biq3q6mvKE8ZKaUj8dXYdvxXSxxqXZ8DnrNpj/UkDTi3S
 chlTT1qUYrp2MVYqGmFjZ0XcEdEUR1I39rUs+wRxE3RgtE56M1A=
 =g6bC
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging

* Replace --enable/disable-git-update with --with-git-submodules
  to allow improved control over use of git submodules
* Deprecate the -enable-fips option
* Ensure docs use prefer format for bool options
* Clarify platform support rules
* Misc fixes to keymap conversions
* Fix misc problems on macOS

# gpg: Signature made Fri 29 Jan 2021 17:10:13 GMT
# gpg:                using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg:                 aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E  8E3F BE86 EBB4 1510 4FDF

* remotes/berrange-gitlab/tags/misc-fixes-pull-request:
  tests: Replace deprecated ASN1 code
  tests: Fix runtime error in test-authz-pam
  ui: update keycodemapdb submodule commit
  crypto: Add spaces around operator
  configure: replace --enable/disable-git-update with --with-git-submodules
  docs: fix missing backslash in certtool shell example
  docs: simplify and clarify the platform support rules
  Prefer 'on' | 'off' over 'yes' | 'no' for bool options
  os: deprecate the -enable-fips option and QEMU's FIPS enforcement
  crypto: Fix memory leaks in set_loaded for tls-*
  crypto: Forbid broken unloading of secrets
  crypto: Move USER_CREATABLE to secret_common base class
  crypto: Fix some code style problems, add spaces around operator

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-01-29 19:51:25 +00:00
Daniel P. Berrangé
4d7beeab38 Prefer 'on' | 'off' over 'yes' | 'no' for bool options
Update some docs and test cases to use 'on' | 'off' as the preferred
value for bool options.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-29 17:07:53 +00:00
Vladimir Sementsov-Ogievskiy
9da4a314a0 iotests/testrunner: fix recognition of python tests
We should drop final '\n' before comparing with python3 shebang.

Fixes: d74c754c92
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210129090616.84145-1-vsementsov@virtuozzo.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-29 12:32:36 +01:00
Vladimir Sementsov-Ogievskiy
a44be0334b iotests: rename and move 169 and 199 tests
Rename bitmaps migration tests and move them to tests subdirectory to
demonstrate new human-friendly test naming.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210125185056.129513-7-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:14 +01:00
Vladimir Sementsov-Ogievskiy
f203080bbd iotests: rewrite check into python
Just use classes introduced in previous three commits. Behavior
difference is described in these three commits.

Drop group file, as it becomes unused.

Drop common.env: now check is in python, and for tests we use same
python interpreter that runs the check itself. Use build environment
PYTHON in check-block instead, to keep "make check" use the same
python.

Checking for virtio-blk moved to iotests.py, as it actually iotests.py
dependency. Actually not all python iotests depend on it, so in future
it may be refactored to checked only when really needed.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210125185056.129513-6-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:14 +01:00
Vladimir Sementsov-Ogievskiy
d74c754c92 iotests: add testrunner.py
Add TestRunner class, which will run tests in a new python iotests
running framework.

There are some differences with current ./check behavior, most
significant are:
- Consider all tests self-executable, just run them, don't run python
  by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- Instead of using "diff -w" which ignores all whitespace differences,
  manually strip whitespace at line end then use python difflib, which
  no longer ignores spacing mid-line

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210125185056.129513-5-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:14 +01:00
Vladimir Sementsov-Ogievskiy
2e5a2f57db iotests: add testenv.py
Add TestEnv class, which will handle test environment in a new python
iotests running framework.

Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210125185056.129513-4-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:14 +01:00
Vladimir Sementsov-Ogievskiy
b25a948875 iotests: add findtests.py
Add python script with new logic of searching for tests:

Current ./check behavior:
 - tests are named [0-9][0-9][0-9]
 - tests must be registered in group file (even if test doesn't belong
   to any group, like 142)

Behavior of findtests.py:
 - group file is dropped
 - tests are all files in tests/ subdirectory (except for .out files),
   so it's not needed more to "register the test", just create it with
   appropriate name in tests/ subdirectory. Old names like
   [0-9][0-9][0-9] (in root iotests directory) are supported too, but
   not recommended for new tests
 - groups are parsed from '# group: ' line inside test files
 - optional file group.local may be used to define some additional
   groups for downstreams
 - 'disabled' group is used to temporary disable tests. So instead of
   commenting tests in old 'group' file you now can add them to
   disabled group with help of 'group.local' file
 - selecting test ranges like 5-15 are not supported more
   (to support restarting failed ./check command from the middle of the
    process, new argument is added: --start-from)

Benefits:
 - no rebase conflicts in group file on patch porting from branch to
   branch
 - no conflicts in upstream, when different series want to occupy same
   test number
 - meaningful names for test files
   For example, with digital number, when some person wants to add some
   test about block-stream, he most probably will just create a new
   test. But if there would be test-block-stream test already, he will
   at first look at it and may be just add a test-case into it.
   And anyway meaningful names are better.

This commit doesn't update check behavior (which will be done in
further commit), still, the documentation changed like new behavior is
already here.  Let's live with this small inconsistency for the
following few commits, until final change.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210125185056.129513-3-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:14 +01:00
Vladimir Sementsov-Ogievskiy
362ef77f9b iotests: 146: drop extra whitespaces from .out file
check script will be stricter soon about whitespaces, so fix 146.out
now.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210125185056.129513-2-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2021-01-27 20:53:02 +01:00
Max Reitz
bb24cdc5ef iotests/178: Pass value to invalid option
ccd3b3b811 has deprecated short-hand boolean options (i.e., options
with values).  All options without values are interpreted as boolean
options, so this includes the invalid option "snapshot.foo" used in
iotest 178.

So after ccd3b3b811, 178 fails with:

  +qemu-img: warning: short-form boolean option 'snapshot.foo' deprecated
  +Please use snapshot.foo=on instead

Suppress that deprecation warning by passing some value to it (it does
not matter which, because the option is invalid anyway).

Fixes: ccd3b3b811
       ("qemu-option: warn for short-form boolean options")
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210126123834.115915-1-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
014bb4b0e5 iotests/118: Drop 'change' test
Commit 0afec75734 removed the 'change' QMP command, so we can no
longer test it in 118.

Fixes: 0afec75734
       ('qmp: remove deprecated "change" command')
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210126104833.57026-1-mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-01-26 14:36:37 +01:00
Alberto Garcia
0f46147e10 iotests: Add test for the regression fixed in c8bf9a9169
Signed-off-by: Alberto Garcia <berto@igalia.com>
Suggested-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20210112170540.2912-1-berto@igalia.com>
[mreitz: Add "# group:" line]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
2d0f32e3fc iotests: 257: prepare for backup over block-copy
Iotest 257 dumps a lot of in-progress information of backup job, such
as offset and bitmap dirtiness. Further commit will move backup to be
one block-copy call, which will introduce async parallel requests
instead of plain cluster-by-cluster copying. To keep things
deterministic, allow only one worker (only one copy request at a time)
for this test.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-15-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
34a5de525a iotests: 219: prepare for backup over block-copy
The further change of moving backup to be a one block-copy call will
make copying chunk-size and cluster-size two separate things. So, even
with 64k cluster sized qcow2 image, default chunk would be 1M.
Test 219 depends on specified chunk-size. Update it for explicit
chunk-size for backup as for mirror.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-14-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
61623f8215 iotests: 185: prepare for backup over block-copy
The further change of moving backup to be a one block-copy call will
make copying chunk-size and cluster-size two separate things. So, even
with 64k cluster sized qcow2 image, default chunk would be 1M.
185 test however assumes, that with speed limited to 64K, one iteration
would result in offset=64K. It will change, as first iteration would
result in offset=1M independently of speed.

So, let's explicitly specify, what test wants: set max-chunk to 64K, so
that one iteration is 64K. Note, that we don't need to limit
max-workers, as block-copy rate limiter will handle the situation and
wouldn't start new workers when speed limit is obviously reached.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-13-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
67a066fbe4 iotests/129: Limit backup's max-chunk/max-workers
Right now, this does not change anything, because backup ignores
max-chunk and max-workers.  However, as soon as backup is switched over
to block-copy for the background copying process, we will need it to
keep 129 passing.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210120102043.28346-1-mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
55f3e5cb3b iotests: 56: prepare for backup over block-copy
After introducing parallel async copy requests instead of plain
cluster-by-cluster copying loop, we'll have to wait for paused status,
as we need to wait for several parallel request. So, let's gently wait
instead of just asserting that job already paused.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-12-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
3ee1483b95 job: call job_enter from job_pause
If main job coroutine called job_yield (while some background process
is in progress), we should give it a chance to call job_pause_point().
It will be used in backup, when moved on async block-copy.

Note, that job_user_pause is not enough: we want to handle
child_job_drained_begin() as well, which call job_pause().

Still, if job is already in job_do_yield() in job_pause_point() we
should not enter it.

iotest 109 output is modified: on stop we do bdrv_drain_all() which now
triggers job pause immediately (and pause after ready is standby).

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210116214705.822267-10-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
59c9466d58 iotests/300: Clean up pylint and mypy complaints
And consequentially drop it from 297's skip list.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-11-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
636aa64d48 iotests/129: Clean up pylint and mypy complaints
And consequentially drop it from 297's skip list.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210118105720.14824-10-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
20c15f7c52 iotests/129: Limit mirror job's buffer size
Issuing 'stop' on the VM drains all nodes.  If the mirror job has many
large requests in flight, this may lead to significant I/O that looks a
bit like 'stop' would make the job try to complete (which is what 129
should verify not to happen).

We can limit the I/O in flight by limiting the buffer size, so mirror
will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of
512 kB by default; or backup, which goes cluster by cluster.  Once we
have asynchronous requests for backup, that will change, but then we can
fine-tune the backup job to only perform a single request on a very
small chunk, too.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-9-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
55557b0245 iotests/129: Actually test a commit job
Before this patch, test_block_commit() performs an active commit, which
under the hood is a mirror job.  If we want to test various different
block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the
commit is done (and then specifying the source node as the top node for
the commit job).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-8-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
a1933dac8f iotests/129: Use throttle node
Throttling on the BB has not affected block jobs in a while, so it is
possible that one of the jobs in 129 finishes before the VM is stopped.
We can fix that by running the job from a throttle node.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-7-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
f9a6256b48 iotests/129: Do not check @busy
@busy is false when the job is paused, which happens all the time
because that is how jobs yield (e.g. for mirror at least since commit
565ac01f8d).

Back when 129 was added (2015), perhaps there was no better way of
checking whether the job was still actually running.  Now we have the
@status field (as of 58b295ba52, i.e. 2018), which can give us exactly
that information.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-6-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
20e2580eec iotests/129: Remove test images in tearDown()
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-5-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
c5ff5a3caa iotests: Move try_remove to iotests.py
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-4-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Max Reitz
59aec86982 iotests/297: Rewrite in Python and extend reach
Instead of checking iotests.py only, check all Python files in the
qemu-iotests/ directory.  Of course, most of them do not pass, so there
is an extensive skip list for now.  (The only files that do pass are
209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of
files that we do want to check, but I think it is better to check files
by default.)

Unless started in debug mode (./check -d), the output has no information
on which files are tested, so we will not have a problem e.g. with
backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors
  setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include
  paths set by the environment.  Maybe at some point we want to let the
  check script add '../../python/' to PYTHONPATH so that iotests.py does
  not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO
  comments.  TODO is fine, we do not need 297 to complain about such
  comments.

- The "Success" line from mypy's output is suppressed, because (A) it
  does not add useful information, and (B) it would leak information
  about the files having been tested to the reference output, which we
  decidedly do not want.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210118105720.14824-3-mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
2021-01-26 14:36:37 +01:00
Max Reitz
3e0105e059 iotests.py: Assume a couple of variables as given
There are a couple of environment variables that we fetch with
os.environ.get() without supplying a default.  Clearly they are required
and expected to be set by the ./check script (as evidenced by
execute_setup_common(), which checks for test_dir and
qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str]
type, which mypy will complain about when tests just assume these values
to be str.

Use [] instead, which raises a KeyError for environment variables that
are not set.  When this exception is raised, catch it and move the abort
code from execute_setup_common() there.

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because
that sort of thing is precisely what this patch wants to prevent.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210118105720.14824-2-mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Andrey Shinkevich
205736f488 block: apply COR-filter to block-stream jobs
This patch completes the series with the COR-filter applied to
block-stream operations.

Adding the filter makes it possible in future implement discarding
copied regions in backing files during the block-stream job, to reduce
the disk overuse (we need control on permissions).

Also, the filter now is smart enough to do copy-on-read with specified
base, so we have benefit on guest reads even when doing block-stream of
the part of the backing chain.

Several iotests are slightly modified due to filter insertion.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201216061703.70908-14-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
9126a2dc4b iotests: 30: prepare to COR filter insertion by stream job
test_stream_parallel run parallel stream jobs, intersecting so that top
of one is base of another. It's OK now, but it would be a problem if
insert the filter, as one job will want to use another job's filter as
above_base node.

Correct thing to do is move to new interface: "bottom" argument instead
of base. This guarantees that jobs don't intersect by their actions.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201216061703.70908-12-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Andrey Shinkevich
b6e0985a4c iotests: add #310 to test bottom node in COR driver
The test case #310 is similar to #216 by Max Reitz. The difference is
that the test #310 involves a bottom node to the COR filter driver.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  [vsementsov: detach backing to test reads from top, limit to qcow2]
Message-Id: <20201216061703.70908-7-vsementsov@virtuozzo.com>
[mreitz: Add "# group:" line]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:29 +01:00
Vladimir Sementsov-Ogievskiy
0c8de0a133 iotests: fix _check_o_direct
Unfortunately commit "iotests: handle tmpfs" breaks running iotests
with -nbd -nocache, as _check_o_direct tries to create
$TEST_IMG.test_o_direct, but in case of nbd TEST_IMG is something like
nbd+unix:///... , and test fails with message

  qemu-img: nbd+unix:///?socket[...]test_o_direct: Protocol driver
    'nbd' does not support image creation, and opening the image
    failed: Failed to connect to '/tmp/tmp.[...]/nbd/test_o_direct': No
    such file or directory

Use TEST_DIR instead.

Fixes: cfdca2b9f9
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201218182012.47607-1-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 11:26:54 +01:00
Vladimir Sementsov-Ogievskiy
f874e7fa3b iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
Just drop code duplication.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201130134024.19212-5-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 20:24:51 -06:00
Vladimir Sementsov-Ogievskiy
5bd04f613a iotests.py: fix qemu_tool_pipe_and_status()
qemu_img_args variable is unrelated here. We should print just args.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201130134024.19212-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 20:22:39 -06:00
Vladimir Sementsov-Ogievskiy
51223cbfcc iotests/264: fix style
Fix long line, extra import and one mypy complaint about incompatible
int and float.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201118180433.11931-7-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 20:22:31 -06:00
Vladimir Sementsov-Ogievskiy
9dd003a998 iotests: define group in each iotest
We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

    cd tests/qemu-iotests

    grep '^[0-9]\{3\} ' group | while read line; do
        file=$(awk '{print $1}' <<< "$line");
        groups=$(sed -e 's/^... //' <<< "$line");
        awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
        cat tmp > $file;
    done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Vladimir Sementsov-Ogievskiy
92a476e959 iotests/294: add shebang line
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-6-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Vladimir Sementsov-Ogievskiy
fd6ce1d085 iotests: make tests executable
All other test files are executable. Fix these.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Vladimir Sementsov-Ogievskiy
f93e19fb03 iotests: fix some whitespaces in test output files
We are going to be stricter about comparing test result with .out
files. So, fix some whitespaces now.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-4-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Vladimir Sementsov-Ogievskiy
fb07b5c7f2 iotests/303: use dot slash for qcow2.py running
If you run './check 303', check includes common.config which adjusts
$PATH to include '.' first, and therefore finds qcow2.py on PATH.  But
if you run './303' directly, there is nothing to adjust PATH, and if
'.' is not already on your PATH by other means, the test fails because
the executable is not found.  Adjust how we invoke the helper
executable to avoid needing a PATH search in the first place.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-3-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Vladimir Sementsov-Ogievskiy
ec77662e60 iotests/277: use dot slash for nbd-fault-injector.py running
If you run './check 277', check includes common.config which adjusts
$PATH to include '.' first, and therefore finds nbd-fault-injector.py
on PATH.  But if you run './277' directly, there is nothing to adjust
PATH, and if '.' is not already on your PATH by other means, the test
fails because the executable is not found.  Adjust how we invoke the
helper executable to avoid needing a PATH search in the first place.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-2-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2021-01-20 14:53:22 -06:00
Alex Bennée
f4c6557bfe tests/iotests: drop test 312 from auto group
The "auto" documentation states:

  That means they should run with every QEMU binary (also non-x86)

which is not the case as the check-system-fedora build which only
includes a rag tag group of rare and deprecated targets doesn't
support the virtio device required.

Fixes: ef9bba1484 ("quorum: Implement bdrv_co_block_status()")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210105100402.12350-1-alex.bennee@linaro.org>
2021-01-06 17:30:02 +00:00
Paolo Bonzini
acb1e6db84 configure: move tests/qemu-iotests/common.env generation to meson
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-01-06 10:21:20 +01:00