Commit Graph

338 Commits

Author SHA1 Message Date
John Snow
45b14be9b6 python: enable testing for 3.13
Python 3.13 is in beta and Fedora 41 is preparing to make it the default
system interpreter; enable testing for it.

(In the event problems develop prior to release, it should only impact
the check-python-tox job, which is not run by default and is allowed to
fail.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2024-07-12 16:36:20 -04:00
John Snow
c5be244534 python: Do not use pylint 3.2.4 with python 3.8
There is a bug in this version,
see: https://github.com/pylint-dev/pylint/issues/9751

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2024-07-12 16:36:20 -04:00
John Snow
84e327e844 python: linter changes for pylint 3.x
New bleeding edge versions, new nits to iron out. This addresses the
'check-python-tox' optional GitLab test, while 'check-python-minreqs'
saw no regressions, since it's frozen on an older version of pylint.

Fixes:
qemu/machine/machine.py:345:52: E0606: Possibly using variable 'sock' before assignment (possibly-used-before-assignment)
qemu/utils/qemu_ga_client.py:168:4: R1711: Useless return at end of function or method (useless-return)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2024-07-12 16:36:20 -04:00
Paolo Bonzini
fc00123f3a python: mkvenv: remove ensure command
This was used to bootstrap the venv with a TOML parser, after which
ensuregroup is used.  Now that we expect it to be present as a system
package (either tomli or, for Python 3.11, tomllib), it is not needed
anymore.

Note that this means that, when implemented, the hypothetical "isolated"
mode that does not use any system packages will only work with Python
3.11+.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2024-06-08 10:33:39 +02:00
Paolo Bonzini
1f97715c83 Revert "python: use vendored tomli"
Now that Ubuntu 20.04 is not included anymore, there is no need to ship
it as part of QEMU; Ubuntu 22.04 includes it and Leap users anyway
need to install all the required dependencies from PyPI.

This mostly reverts commit ec77ee7634de123b7c899739711000fd21dab68b,
with just some changes to the wording.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2024-06-08 10:33:39 +02:00
Maksim Davydov
33956e4768 python/qemu/machine: add method to retrieve QEMUMachine::binary field
Add a supportive property to access the path to the QEMU binary

Signed-off-by: Maksim Davydov <davydov-max@yandex-team.ru>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240318213550.155573-4-davydov-max@yandex-team.ru>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
2024-04-25 10:12:43 +02:00
Philippe Mathieu-Daudé
1a1e889f35 buildsys: Bump known good meson version to v1.2.3
We need meson v1.2.3 to build QEMU on macOS Sonoma.  It
also builds fine all our CI jobs (as tested by also bumping
"accepted" in pythondeps.toml), so let's use it as our
"good enough" packaged wheel.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1939
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20231109160504.93677-2-philmd@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-11-24 16:21:55 +01:00
Vladimir Sementsov-Ogievskiy
4e620ff48f python/machine.py: upgrade vm.cmd() method
The method is not popular in iotests, we prefer use vm.qmp() and then
check success by hand. But that's not optimal. To simplify movement to
vm.cmd() let's support same interface improvements like in vm.qmp().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20231006154125.1068348-7-vsementsov@yandex-team.ru
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-12 14:21:43 -04:00
Vladimir Sementsov-Ogievskiy
684750ab4f python/qemu: rename command() to cmd()
Use a shorter name. We are going to move in iotests from qmp() to
command() where possible. But command() is longer than qmp() and don't
look better. Let's rename.

You can simply grep for '\.command(' and for 'def command(' to check
that everything is updated (command() in tests/docker/docker.py is
unrelated).

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-id: 20231006154125.1068348-6-vsementsov@yandex-team.ru
[vsementsov: also update three occurrences in
   tests/avocado/machine_aspeed.py and keep r-b]
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-12 14:21:43 -04:00
Vladimir Sementsov-Ogievskiy
37274707f6 python: rename QEMUMonitorProtocol.cmd() to cmd_raw()
Having cmd() and command() methods in one class doesn't look good.
Rename cmd() to cmd_raw(), to show its meaning better.

We also want to rename command() to cmd() in future, so this commit is
a necessary step.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20231006154125.1068348-5-vsementsov@yandex-team.ru
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-12 14:21:43 -04:00
Vladimir Sementsov-Ogievskiy
2cee9ca97d qmp_shell.py: _fill_completion() use .command() instead of .cmd()
We just want to ignore failure, so we don't need low level .cmd(). This
helps further renaming .command() to .cmd().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20231006154125.1068348-3-vsementsov@yandex-team.ru
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-12 14:21:43 -04:00
Vladimir Sementsov-Ogievskiy
f187cfefd2 python/qemu/qmp/legacy: cmd(): drop cmd_id unused argument
The argument is unused, let's drop it for now, as we are going to
refactor the interface and don't want to refactor unused things.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20231006154125.1068348-2-vsementsov@yandex-team.ru
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-12 14:21:43 -04:00
John Snow
19a39e270b Python: Enable python3.12 support
Python 3.12 has released, so update the test infrastructure to test
against this version. Update the configure script to look for it when an
explicit Python interpreter isn't chosen.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-id: 20231006195243.3131140-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:34 -04:00
John Snow
acf873873a python/qmp: remove Server.wait_closed() call for Python 3.12
This patch is a backport from
e03a3334b6

According to Guido in https://github.com/python/cpython/issues/104344 ,
this call was never meant to wait for the server to shut down - that is
handled synchronously - but instead, this waits for all connections to
close. Or, it would have, if it wasn't broken since it was introduced.

3.12 fixes the bug, which now causes a hang in our code. The fix is just
to remove the wait.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-id: 20231006195243.3131140-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:34 -04:00
John Snow
46d4747ab5 python/machine: remove unused sock_dir argument
By using a socketpair for all of the sockets managed by the VM class and
its extensions, we don't need the sock_dir argument anymore, so remove
it.

We only added this argument so that we could specify a second, shorter
temporary directory for cases where the temp/log dirs were "too long" as
a socket name on macOS. We don't need it for this class now. In one
case, avocado testing takes over responsibility for creating an
appropriate sockdir.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
John Snow
d396737813 python/machine: use socketpair() for qtest connection
Like the QMP and console sockets, begin using socketpairs for the qtest
connection, too. After this patch, we'll be able to remove the vestigial
sock_dir argument, but that cleanup is best done in its own patch.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
John Snow
1d4796cd00 python/machine: use socketpair() for console connections
Create a socketpair for the console output. This should help eliminate
race conditions around console text early in the boot process that might
otherwise have been dropped on the floor before being able to connect to
QEMU under "server,nowait".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Ani Sinha <anisinha@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
John Snow
5f263cb1cc python/console_socket: accept existing FD in initializer
Useful if we want to use ConsoleSocket() for a socket created by
socketpair().

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Ani Sinha <anisinha@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
John Snow
612b3ba218 python/machine: close sock_pair in cleanup path
If everything has gone smoothly, we'll already have closed the socket we
gave to the child during post_launch. The other half of the pair that we
gave to the QMP connection should, likewise, be definitively closed by
now.

However, in the cleanup path, it's possible we've created the socketpair
but flubbed the launch and need to clean up resources. These resources
*would* be handled by the garbage collector, but that can happen at
unpredictable times. Nicer to just clean them up synchronously on the
exit path, here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Ani Sinha <anisinha@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
John Snow
91e11db7bc python/machine: move socket setup out of _base_args property
This property isn't meant to do much else besides return a list of
strings, so move this setup back out into _pre_launch().

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Ani Sinha <anisinha@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20230928044943.849073-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-11 16:02:02 -04:00
Nicholas Piggin
f0ec14c78c tests/avocado: Fix console data loss
Occasionally some avocado tests will fail waiting for console line
despite the machine running correctly. Console data goes missing, as can
be seen in the console log. This is due to _console_interaction calling
makefile() on the console socket each time it is invoked, which must be
losing old buffer contents when going out of scope.

It is not enough to makefile() with buffered=0. That helps significantly
but data loss is still possible. My guess is that readline() has a line
buffer even when the file is in unbuffered mode, that can eat data.

Fix this by providing a console file that persists for the life of the
console.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Message-Id: <20230912131340.405619-1-npiggin@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: John Snow <jsnow@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230914155422.426639-9-alex.bennee@linaro.org>
2023-09-20 15:06:33 +01:00
Paolo Bonzini
0a88ac9662 Revert "mkvenv: work around broken pip installations on Debian 10"
Debian 10 has Python 3.7, so it is not possible to use it anymore
now that Python 3.8 is required.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-09-07 13:32:37 +02:00
Paolo Bonzini
3e4b6b0ad9 mkvenv: assume presence of importlib.metadata
importlib.metadata is included in Python 3.8, so there is no
need to fallback to either importlib-metadata or pkgresources
when generating console script shims.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-09-07 13:32:37 +02:00
Paolo Bonzini
ca056f4499 Python: Drop support for Python 3.7
Debian 10 is not anymore a supported distro, since Debian 12 was
released on June 10, 2023.  Our supported build platforms as of today
all support at least 3.8 (and all of them except for Ubuntu 20.04
support 3.9):

openSUSE Leap 15.5: 3.6.15 (3.11.2)
CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16, 3.11.4)
CentOS Stream 9:    3.9.17 (3.11.4)
Fedora 37:          3.11.4
Fedora 38:          3.11.4
Debian 11:          3.9.2
Debian 12:          3.11.2
Alpine 3.14, 3.15:  3.9.16
Alpine 3.16, 3.17:  3.10.10
Ubuntu 20.04 LTS:   3.8.10
Ubuntu 22.04 LTS:   3.10.12
NetBSD 9.3:         3.9.13*
FreeBSD 12.4:       3.9.16
FreeBSD 13.1:       3.9.18
OpenBSD 7.2:        3.9.17

Note: NetBSD does not appear to have a default meta-package, but offers
several options, the lowest of which is 3.7.15. However, "python39"
appears to be a pre-requisite to one of the other packages we request
in tests/vm/netbsd.

Since it is safe under our supported platform policy, bump our
minimum supported version of Python to 3.8.  The two most interesting
features to have by default include:

- the importlib.metadata module, whose lack is responsible for over 100
  lines of code in mkvenv.py

- improvements to asyncio, for example asyncio.CancelledError
  inherits from BaseException rather than Exception

In addition, code can now use the assignment operator ':='

Because mypy now learns about importlib.metadata, a small change to
mkvenv.py is needed to pass type checking.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-09-07 13:32:37 +02:00
Paolo Bonzini
c03f57fd5b Revert "tests: Use separate virtual environment for avocado"
This reverts commit e8e4298fea.

ensuregroup allows to specify both the acceptable versions of avocado,
and a locked version to be used when avocado is not installed as a system
pacakge.  This lets us install avocado in pyvenv/ using "mkvenv.py" and
reuse the distro package on Fedora and CentOS Stream (the only distros
where it's available).

ensuregroup's usage of "(>=..., <=...)" constraints when evaluating
the distro package, and "==" constraints when installing it from PyPI,
makes it possible to avoid conflicts between the known-good version and
a package plugins included in the distro.

This is because package plugins have "==" constraints on the version
that is included in the distro, and, using "pip install avocado==88.1"
on a venv that includes system packages will result in an error:

   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.

But at the same time, if the venv does not include a system distribution
of avocado then we can install a known-good version and stick to LTS
releases.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28 09:55:48 +02:00
Paolo Bonzini
edc2107895 python: use vendored tomli
Debian only introduced tomli in the bookworm release.  Use a
vendored wheel to avoid requiring a package that is only in
bullseye-backports and is also absent in Ubuntu 20.04.

While at it, fix an issue in the vendor.py scripts which does
not add a newline after each package and hash.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28 09:55:48 +02:00
Paolo Bonzini
71ed611cd4 python: mkvenv: add ensuregroup command
Introduce a new subcommand that retrieves the packages to be installed
from a TOML file. This allows being more flexible in using the system
version of a package, while at the same time using a known-good version
when installing the package.  This is important for packages that
sometimes have backwards-incompatible changes or that depend on
specific versions of their dependencies.

Compared to JSON, TOML is more human readable and easier to edit.  A
parser is available in 3.11 but also available as a small (12k) package
for older versions, tomli.  While tomli is bundled with pip, this is only
true of recent versions of pip.  Of all the supported OSes pretty much
only FreeBSD has a recent enough version of pip while staying on Python
<3.11.  So we cannot use the same trick that is in place for distlib.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28 09:55:36 +02:00
Paolo Bonzini
0f1ec0705b python: mkvenv: introduce TOML-like representation of dependencies
We would like to place all Python dependencies in the same file, so that
we can add more information without having long and complex command lines.
The plan is to have a TOML file with one entry per package, for example

  [avocado]
  avocado-framework = {
    accepted = "(>=88.1, <93.0)",
    installed = "88.1",
    canary = "avocado"
  }

Each TOML section will thus be a dictionary of dictionaries.  Modify
mkvenv.py's workhorse function, _do_ensure, to already operate on such
a data structure.  The "ensure" subcommand is modified to separate the
depspec into a name and a version part, and use the result (plus the
--diagnose argument) to build a dictionary for each command line argument.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28 09:55:05 +02:00
Paolo Bonzini
67b9a83daf python: mkvenv: tweak the matching of --diagnose to depspecs
Move the matching between the "absent" array and dep_specs[0] inside
the loop, preparing for the possibility of having multiple canaries
among the installed packages.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28 09:55:05 +02:00
Paolo Bonzini
3d7b89748a python: bump minimum requirements so they are compatible with 3.12
There are many Python 3.12 issues right now, but a particularly
problematic one when debugging them is that one cannot even use
minreqs.txt in a Python 3.12 virtual environment to test with
locked package versions.

Bump the mypy and wrapt versions to fix this, while remaining
within the realm of versions compatible with Python 3.7.

This requires a workaround for a mypy false positive

    qemu/qmp/qmp_tui.py:350: error: Non-overlapping equality check (left operand type: "Literal[Runstate.DISCONNECTING]", right operand type: "Literal[Runstate.IDLE]")  [comparison-overlap]

where mypy does not realize that self.disconnect() could change
the value of self.runstate.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-07-07 12:49:22 +02:00
Paolo Bonzini
47a90a51a9 mkvenv: always pass locally-installed packages to pip
Let pip decide whether a new version should be installed or the current
one is okay.  This ensures that the virtual environment is updated
(either upgraded or downgraded) whenever a new version of a package is
requested.

The hardest part here is figuring out if a package is installed in
the venv (which also has to be done twice to account for the presence
of either setuptools in Python <3.8, or importlib in Python >=3.8).

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Cc: John Snow <jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06 16:30:01 +02:00
John Snow
c76e7652c7 Revert "python/qmp/protocol: add open_with_socket()"
This reverts commit a3cfea92e2.

(It's being rolled back in favor of a different API, which brings the
in-tree and out-of-tree versions of qemu.qmp back in sync.)

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230517163406.2593480-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31 16:25:35 -04:00
John Snow
5bbc5936bb python/qmp/legacy: remove open_with_socket() calls
Favor using connect() when passing a socket instead of
open_with_socket(). Simultaneously, update constructor calls to use the
combined address argument for QEMUMonitorProtocol().

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230517163406.2593480-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31 16:25:35 -04:00
John Snow
7f5f3ae7d5 python/machine: use connect-based interface for existing sockets
Instead of using accept() with sockets (which uses open_with_socket()),
use calls to connect() to utilize existing sockets instead. A benefit of
this is more robust error handling already present within the connect()
call that isn't present in open_with_socket().

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230517163406.2593480-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31 16:25:35 -04:00
John Snow
b8d4ca1823 python/qmp/legacy: allow using sockets for connect()
Instead of asserting that we have an address, allow the use of sockets
instead of addresses during a call to connect().

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230517163406.2593480-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31 16:25:35 -04:00
John Snow
9341b2a6b9 python/qmp: allow sockets to be passed to connect()
Allow existing sockets to be passed to connect(). The changes are pretty
minimal, and this allows for far greater flexibility in setting up
communications with an endpoint.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230517163406.2593480-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31 16:25:35 -04:00
Richard Henderson
886c0453cb QAPI patches patches for 2023-05-17
-----BEGIN PGP SIGNATURE-----
 
 iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMFAmRrTcgSHGFybWJydUBy
 ZWRoYXQuY29tAAoJEDhwtADrkYZTMycP/3sP6/U4kwOKMGGcB+n2pHJeioQS4xgF
 94NCW+KpewxApP0XzIC2nDGjUe/rPcUfQmBNUumvYbqHO91tq91wFwkllBv2UR0q
 6qfRji+e8+9H9hMDeVzzSNjlZZg/tSdIJlhkJDw1u4/3fpjfAmzVx6DO3wepSQ9Q
 m5Af/+uhVZWyUXMZqcKr2Zq8qur6ZFEBNpXpPvT60Tvy2heuQ+vcoE3tl2ZRQbmj
 b/jhtCu+NPjgOHtg9Gr2BPXqQiZBR4vFA7WBsB8wCf2xxULfTwHJvFz/e0vx5fUC
 q0Fsyybf4USo2PRMsRFv2v4dEuVGHb3E1RIJY4NTAxQMqqm4zfOyK0BzOGNDkxCn
 owNP4vKly0e/CfYDY74FHaPId295xyeo6S4Cj5ib9W23AAWUNt6f6vbjlDOLCLON
 c7yXP/aJwhTb2w1t0mLTmsKum3DpLlrudPudTylVlmYfwchkvUGsWYbaxu6H6XWk
 49Ox/QPVwqG6elXNn3kTY4QqTAppXhE7QcPbioX9WOThVPf6aJCLdZSHEHu4HXkZ
 4FRu73Z2wcPNB789xOrQoXs24GdKmWXQ6K01KC4v7WNJQBXccec52yGxvktQRZBm
 GL3zYdOOJEL+Y/8JrXTIo26M8HP/4kxV2VqB6KOuaGygMsW9w9jbG+ygLyjqUDQg
 3APV3hdmVOht
 =6anf
 -----END PGP SIGNATURE-----

Merge tag 'pull-qapi-2023-05-17-v2' of https://repo.or.cz/qemu/armbru into staging

QAPI patches patches for 2023-05-17

# -----BEGIN PGP SIGNATURE-----
#
# iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMFAmRrTcgSHGFybWJydUBy
# ZWRoYXQuY29tAAoJEDhwtADrkYZTMycP/3sP6/U4kwOKMGGcB+n2pHJeioQS4xgF
# 94NCW+KpewxApP0XzIC2nDGjUe/rPcUfQmBNUumvYbqHO91tq91wFwkllBv2UR0q
# 6qfRji+e8+9H9hMDeVzzSNjlZZg/tSdIJlhkJDw1u4/3fpjfAmzVx6DO3wepSQ9Q
# m5Af/+uhVZWyUXMZqcKr2Zq8qur6ZFEBNpXpPvT60Tvy2heuQ+vcoE3tl2ZRQbmj
# b/jhtCu+NPjgOHtg9Gr2BPXqQiZBR4vFA7WBsB8wCf2xxULfTwHJvFz/e0vx5fUC
# q0Fsyybf4USo2PRMsRFv2v4dEuVGHb3E1RIJY4NTAxQMqqm4zfOyK0BzOGNDkxCn
# owNP4vKly0e/CfYDY74FHaPId295xyeo6S4Cj5ib9W23AAWUNt6f6vbjlDOLCLON
# c7yXP/aJwhTb2w1t0mLTmsKum3DpLlrudPudTylVlmYfwchkvUGsWYbaxu6H6XWk
# 49Ox/QPVwqG6elXNn3kTY4QqTAppXhE7QcPbioX9WOThVPf6aJCLdZSHEHu4HXkZ
# 4FRu73Z2wcPNB789xOrQoXs24GdKmWXQ6K01KC4v7WNJQBXccec52yGxvktQRZBm
# GL3zYdOOJEL+Y/8JrXTIo26M8HP/4kxV2VqB6KOuaGygMsW9w9jbG+ygLyjqUDQg
# 3APV3hdmVOht
# =6anf
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 22 May 2023 04:11:04 AM PDT
# gpg:                using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg:                issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [undefined]
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* tag 'pull-qapi-2023-05-17-v2' of https://repo.or.cz/qemu/armbru:
  docs/interop: Delete qmp-intro.txt
  docs/interop/qmp-spec: Update error description for parsing errors
  docs/interop: Convert qmp-spec.txt to rST
  qapi: Improve error message for description following section

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2023-05-22 15:54:21 -07:00
Peter Maydell
d56572584d docs/interop: Convert qmp-spec.txt to rST
Convert the qmp-spec.txt document to restructuredText.
Notable points about the conversion:
 * numbers at the start of section headings are removed, to match
   the style of the rest of the manual
 * cross-references to other sections or documents are hyperlinked
 * various formatting tweaks (notably the examples, which need the
   -> and <- prefixed so the QMP code-block lexer will accept them)
 * English prose fixed in a few places

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20230515162245.3964307-2-peter.maydell@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
[.. code-block:: dumbed down to :: to work around CI failure]
2023-05-22 10:21:01 +02:00
Paolo Bonzini
d37c21b5fb mkvenv: pass first missing package to diagnose()
If sphinx is present but the theme is not, mkvenv will print an
inaccurate diagnostic:

ERROR: Could not find a version that satisfies the requirement sphinx-rtd-theme>=0.5.0 (from versions: none)
ERROR: No matching distribution found for sphinx-rtd-theme>=0.5.0

'sphinx>=1.6.0' not found:
 • Python package 'sphinx' version '5.3.0' was found, but isn't suitable.
 • mkvenv was configured to operate offline and did not check PyPI.

Instead, ignore the packages that were found to be present, and report
an error based on the first absent package.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-19 20:40:29 +02:00
Paolo Bonzini
c673f3d0fe mkvenv: replace distlib.database with importlib.metadata/pkg_resources
importlib.metadata is just as good as distlib.database and a bit more
battle-proven for "egg" based distributions, and in fact that is exactly
why mkvenv.py is not using distlib.database to find entry points: it
simply does not work for eggs.

The only disadvantage of importlib.metadata is that it is not available
by default before Python 3.8, so we need a fallback to pkg_resources
(again, just like for the case of finding entry points).  Do so to
fix issues where incorrect egg metadata results in a JSONDecodeError.

While at it, reuse the new _get_version function to diagnose an incorrect
version of the package even if importlib.metadata is not available.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-19 20:09:21 +02:00
Paolo Bonzini
3b087f79a4 meson: require 0.63.0
This version allows cleanups in modinfo collection, but they only
work with Ninja 1.9.x and 1.8.x is still supported.  It also supports the
equivalent of QEMU's --static option to configure.

The wheel file is bumped to 0.63.3, the last release in the 0.63 branch.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
7b4b98c46c python: bump some of the dependencies
The version of pyflakes that is listed in python/tests/minreqs.txt
breaks on Python 3.8 with the following message:

  AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'

Now that we do not support EOL'd Python versions anymore, we can
update to newer, fixed versions.  It is a good time to do so, before
Python packages start dropping support for Python 3.7 as well!

The new mypy is also a bit smarter about which packages are actually
being used, so remove the now-unnecessary sections from setup.cfg.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-27-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
02312f1af1 mkvenv: mark command as required
This is only available in Python 3.7+.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-26-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
5591b74511 Python: Drop support for Python 3.6
Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
begun dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6, but won't until QEMU does.

Versions of Python available in our supported build platforms as of today,
with optional versions available in parentheses:

openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16)
CentOS Stream 9:    3.9.13
Fedora 36:          3.10
Fedora 37:          3.11
Debian 11:          3.9.2
Alpine 3.14, 3.15:  3.9.16
Alpine 3.16, 3.17:  3.10.10
Ubuntu 20.04 LTS:   3.8.10
Ubuntu 22.04 LTS:   3.10.4
NetBSD 9.3:         3.9.13*
FreeBSD 12.4:       3.9.16
FreeBSD 13.1:       3.9.16
OpenBSD 7.2:        3.9.16

Note: Our VM tests install 3.9 explicitly for FreeBSD and 3.10 for
NetBSD; the default for "python" or "python3" in FreeBSD is
3.9.16. NetBSD does not appear to have a default meta-package, but
offers several options, the lowest of which is 3.7.15. "python39"
appears to be a pre-requisite to one of the other packages we request in
tests/vm/netbsd. pip, ensurepip and other Python essentials are
currently only available for Python 3.10 for NetBSD.

CentOS and OpenSUSE support parallel installation of multiple Python
interpreters, and binaries in /usr/bin will always use Python 3.6.  However,
the newly introduced support for virtual environments ensures that all build
steps that execute QEMU Python code use a single interpreter.

Since it is safe to under our supported platform policy, bump our
minimum supported version of Python to 3.7.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20230511035435.734312-24-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
e80bdbf283 python/wheels: add vendored meson package
In preference to vendoring meson source, vendor a built distributable
("bdist" in python parlance). This has some benefits:

(1) We can get rid of a git submodule,
(2) Installing built meson into a venv doesn't require any extra
    dependencies (the python "wheel" package, chiefly.)
(3) We don't treat meson any differently than we would any other python
    package (we install it, end of story, done.)
(4) All future tarball *and* developer checkouts will function offline;
    No git or PyPI connection needed to fetch meson.

Note that because mkvenv prefers vendored packages to PyPI, as mkvenv is
currently written we will never consult PyPI for meson. (Do keep in mind
that your distribution's meson will be preferred above the vendored
version, though.)

```
jsnow@scv ~/s/q/python (python-configure-venv)> python3 scripts/vendor.py
pip download --dest /home/jsnow/src/qemu/python/wheels --require-hashes -r /tmp/tmpvo5qav7i
Collecting meson==0.61.5
  Using cached meson-0.61.5-py3-none-any.whl (862 kB)
Saved ./wheels/meson-0.61.5-py3-none-any.whl
Successfully downloaded meson
```

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-17-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
2274817f6c python: add vendor.py utility
This is a teeny-tiny script that just downloads any packages we want to
vendor from PyPI and stores them in qemu.git/python/wheels/. If I'm hit
by a meteor, it'll be easy to replicate what I have done in order to
udpate the vendored source.

We don't really care which python runs it; it exists as a meta-utility
with no external dependencies and we won't package or install it. It
will be monitored by the linters/type checkers, though; so it's
guaranteed safe on python 3.6+.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-15-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
c804962617 mkvenv: work around broken pip installations on Debian 10
This is a workaround intended for Debian 10, where the debian-patched
pip does not function correctly if accessed from within a virtual
environment.

We don't support Debian 10 as a build platform any longer, though we do
still utilize it for our build-tricore-softmmu CI test. It's also
possible that this bug might appear on other derivative platforms and
this workaround may prove useful.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-11-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
f1ad527ff5 mkvenv: avoid ensurepip if pip is installed
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
68ea6d17fe mkvenv: use pip's vendored distlib as a fallback
distlib is usually not installed on Linux distribution, but it is vendored
into pip.  Because the virtual environment has pip via ensurepip, we
can piggy-back on pip's vendored version.  This could break if they move
our cheese in the future, but the fix would be simply to require distlib.

If it is debundled, as it is on msys, it is simply available directly.

Signed-off-by: John Snow <jsnow@redhat.com>
[Move to toplevel. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
928348949d mkvenv: add console script entry point generation
When creating a virtual environment that inherits system packages,
script entry points (like "meson", "sphinx-build", etc) are not
re-generated with the correct shebang. When you are *inside* of the
venv, this is not a problem, but if you are *outside* of it, you will
not have a script that engages the virtual environment appropriately.

Add a mechanism that generates new entry points for pre-existing
packages so that we can use these scripts to run "meson",
"sphinx-build", "pip", unambiguously inside the venv.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-9-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00