The Message class is here primarily to serve as a solid type to use for
mypy static typing for unambiguous annotation and documentation.
We can also stuff JSON serialization and deserialization into this class
itself so it can be re-used even outside this infrastructure.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This is added as a courtesy: many protocols are line-based, including
QMP. Putting it in AsyncProtocol lets us keep the QMP class
implementation just a pinch more abstract.
(And, if we decide to add a QTEST implementation later, it will need
this, too. (Yes, I have a QTEST implementation.))
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Add hooks designed to log/filter incoming/outgoing messages. The primary
intent for these is to be able to support iotests which may want to log
messages with specific filters for reproducible output.
Another use is for plugging into Urwid frameworks; all messages in/out
can be automatically added to a rendering list for the purposes of a
qmp-shell like tool.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
QMP can transmit some pretty big messages, and the default limit of 64KB
isn't sufficient. Make sure that we can configure it.
Reported-by: G S Niteesh Babu <niteesh.gs@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
It's a little messier than connect, because it wasn't designed to accept
*precisely one* connection. Such is life.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Give the connection and the reader/writer tasks nicknames, and add
logging statements throughout.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This serves a few purposes:
1. Protect interfaces when it's not safe to call them (via @require)
2. Add an interface by which an async client can determine if the state
has changed, for the purposes of connection management.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This is the bare minimum that you need to establish a full-duplex async
message-based protocol with Python's asyncio.
The features to be added in forthcoming commits are:
- Runstate tracking
- Logging
- Support for incoming connections via accept()
- _cb_outbound, _cb_inbound message hooks
- _readline() method
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Python 3.6 does not have all of the goodies that Python 3.7 does, and we
need to support both. Add some compatibility wrappers needed for this
purpose.
(Note: Python 3.6 is EOL December 2021.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
'T' is a common TypeVar name, allow its use.
See also https://github.com/PyCQA/pylint/issues/3401 -- In the future,
we might be able to have a separate list of acceptable names for
TypeVars exclusively.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
For now, it's empty! Soon, it won't be.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
We're not ready to enforce f-strings everywhere, so just silence this
new warning.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210916182248.721529-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
A few new annoyances. Of note is the new warning for an unspecified
encoding when opening a text file, which actually does indicate a
potentially real problem; see
https://www.python.org/dev/peps/pep-0597/#motivation
Use LC_CTYPE to determine an encoding to use for interpreting QEMU's
terminal output. Note that Python states: "language code and encoding
may be None if their values cannot be determined" -- use a platform
default as a backup.
Notes: Passing encoding=None will generate a suppressed warning on
Python 3.10+ that 'None' should not be passed as the encoding
argument. This behavior may be deprecated in the future and the default
switched to be a ubiquitous UTF-8. Opting in to the locale default will
be done by passing the encoding 'locale', but that isn't available in
3.6 through 3.9. Presumably this warning will be unsuppressed some time
prior to the actual switch and we can re-investigate these issues at
that time if necessary.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210916182248.721529-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
mypy thinks that return value of these methods in subclusses is
QEMUMachine, which is wrong. So, make typing smarter.
Suggested-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210824083856.17408-26-vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
We often call qmp() with unpacking dict, like qmp('foo', **{...}).
mypy don't really like it, it thinks that passed unpacked dict is a
positional argument and complains that it type should be bool (because
second argument of qmp() is conv_keys: bool).
Allow passing dict directly, simplifying interface, and giving a way to
satisfy mypy.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20210824083856.17408-25-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
- use shorter construction
- don't create new dict if not needed
- drop extra unpacking key-val arguments
- drop extra default values
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20210824083856.17408-24-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Using the flag -p, allow the qemu binary to print to stdout.
Also create the common function _close_qemu_log_file() to
avoid accessing machine.py private fields directly and have
duplicate code.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210809090114.64834-16-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Message-Id: <20210809090114.64834-4-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Pylint prior to 2.8.3 (We pin at >= 2.8.0) includes function and method
signatures as part of its duplicate checking algorithm. This check does
not listen to pragmas, so the only way to disable it is to turn it off
completely or increase the minimum duplicate lines so that it doesn't
trigger for functions with long, multi-line signatures.
When we decide to upgrade to pylint 2.8.3 or greater, we will be able to
use 'ignore-signatures = true' to the config instead.
I'd prefer not to keep us on the very bleeding edge of pylint if I can
help it -- 2.8.3 came out only three days ago at time of writing.
See: https://github.com/PyCQA/pylint/pull/4474
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Acked-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210809090114.64834-3-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Also add a new _qmp_timer field to the QEMUMachine class.
Let's change the default socket timeout to None, so that if
a subclass needs to add a timer, it can be done by modifying
this private field.
At the same time, restore the timer to be 15 seconds in iotests.py, to
give an upper bound to the QMP monitor test command execution.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210809090114.64834-2-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Currently tox tests against the installed interpreters, however if any
supported interpreter is absent then it will return fail. It seems not
reasonable to expect developers to have all supported interpreters
installed on their systems. Luckily tox can be configured to skip
missing interpreters.
This changed the tox setup so that missing interpreters are skipped by
default. On the CI, however, we still want to enforce it tests
against all supported. This way on CI the
--skip-missing-interpreters=false option is passed to tox.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630184546.456582-1-wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This added the args property to QEMUMachine so that users of the class
can access and handle the list of arguments to be given to the QEMU
binary.
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210430133414.39905-6-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Logs can be very important to debug issues, and currently QEMUMachine
instances will remove logs that are created under the temporary
directories.
With this change, the stdout and stderr generated by the QEMU process
started by QEMUMachine will always be kept along the test results
directory.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-6-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This patch *doesn't* update all of the docstring standards across the
QEMU package directory to make our docstring usage consistent. It
*doesn't* fix the formatting to make it look pretty or reasonable in
generated output. It *does* fix a few small instances where Sphinx would
emit a build warning because of malformed ReST -- If we built our Python
docs with Sphinx.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
For reasons that at-present escape me, pipenv insists on creating a stub
pyproject.toml file. This file is a nuisance, because its mere presence
changes the behavior of various tools.
For instance, this stub file will cause "pip install --user -e ." to
fail in spectacular fashion with misleading errors. "pip install -e ."
works okay, but for some reason pip does not support editable installs
to the user directory when using PEP517.
References:
https://github.com/pypa/pip/pull/9990https://github.com/pypa/pip/issues/7953
As outlined in ea1213b7cc, it is still too early for us to consider
moving to a PEP-517 exclusive package. We must support older
distributions, so squash the annoyance for now. (Python 3.6 shipped Dec
2016, PEP517 support showed up in pip sometime in 2019 or so.)
Add 'pyproject.toml' to the 'make clean' target, and also delete it
after every pipenv invocation issued by the Makefile.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Update for visual parity with all the remaining targets.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Update for visual parity with the other targets.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This is a *third* way to run the Python tests. Unlike the first two
(check-pipenv, check-tox), this version does not require any specific
interpreter version -- making it a lot easier to tell people to run it
as a quick smoketest prior to submission to GitLab CI.
Summary:
Checked via GitLab CI:
- check-pipenv: tests our oldest python & dependencies
- check-tox: tests newest dependencies on all non-EOL python versions
Executed only incidentally:
- check-dev: tests newest dependencies on whichever python version
('make check' does not set up any environment at all, it just runs the
tests in your current environment. All four invocations perform the
exact same tests, just in different execution environments.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-12-jsnow@redhat.com
[Maintainer edit: added .dev-venv/ to .gitignore. --js]
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Acked-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
flake8 is a little eager to check everything it can. Limit it to
checking inside the qemu namespace directory only. Update setup.cfg now
that the exclude patterns are no longer necessary.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
I missed the 'check-tox' target. Add that, but split the large .PHONY
specifier at the top into its component pieces and move them near the
targets they describe so that they're much harder to forget to update.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210629214323.1329806-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Move it up near the check-pipenv help text, and update it to suggest parity.
(At the time I first added it, I wasn't sure if I would be keeping it,
but I've come to appreciate it as it has actually helped uncover bugs I
would not have noticed without it. It should stay.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Well, Cleber was right, this is a better name.
In preparation for adding a different kind of virtual environment check
(One that simply uses whichever version of Python you happen to have),
rename this test 'check-pipenv' so that it matches the CI job
'check-python-pipenv'.
Remove the "If you don't know which test to run" hint, because it's not
actually likely you have Python 3.6 installed to be able to run the
test. It's still the test I'd most prefer you to run, but it's not the
test you are most likely to be able to run.
Rename the 'venv' target to 'pipenv' as well, and move the more
pertinent help text under the 'check-pipenv' target.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
It's not encouraged, but it's legitimate to want to know how to do.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Clarifying a few points; removing the reference to 'setuptools' because
it isn't referenced anywhere else in this document and doesn't really
provide any useful information to a Python newcomer.
Adjusting the language elsewhere to be less ambiguous and have fewer
run-on sentences.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
tox is already testing the most recent versions. Let's use pipenv to
test the oldest versions we claim to support. This matches the stylistic
choice to have pipenv always test our oldest supported Python version, 3.6.
The effect of this is that the python-check-pipenv CI job on gitlab will
now test against much older versions of these linters, which will help
highlight incompatible changes that might otherwise go unnoticed.
Update instructions for adding and bumping versions in setup.cfg. The
reason for deleting the line that gets added to Pipfile is largely just
to avoid having the version minimums specified in multiple places in
config checked into the tree.
(This patch was written by deleting Pipfile and Pipfile.lock, then
explicitly installing each dependency manually at a specific
version. Then, I restored the prior Pipfile and re-ran `pipenv lock
--dev --keep-outdated` to re-add the qemu dependency back to the pipenv
environment while keeping the "old" packages. It's annoying, yes, but I
think the improvement to test coverage is worthwhile.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
These suppressions only apply to a small handful of places. Instead of
disabling them globally, disable them just in the cases where we
need. The design of the machine class grew quite organically with tons
of constructor and class instance variables -- there's little chance of
meaningfully refactoring it in the near term, so just suppress the
warnings for that class.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
https://www.python.org/dev/peps/pep-0561/#specification
Create 'py.typed' files in each subpackage that indicate to mypy that
this is a typed module, so that users of any of these packages can use
mypy to check their code as well.
Note: Theoretically it's possible to ditch MANIFEST.in in favor of using
package_data in setup.cfg, but I genuinely could not figure out how to
get it to include things from the *source root* into the *package root*;
only how to include things from each subpackage. I tried!
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Pylint updated to 2.9.0 upstream, adding new warnings for things that
re-use the 'err' variable. Luckily, this only breaks the
python-check-tox job, which is allowed to fail as a warning.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210629214323.1329806-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
now 'qmp-shell' should be available from the command line when
installing the python package.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-42-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The script will be unavailable for a commit or two, which will help
preserve development history attached to the new file. A forwarder will
be added shortly afterwards.
With qmp_shell in the python qemu.qmp package, now it is fully type
checked, linted, etc. via the Python CI. It will be quite a bit harder
to accidentally break it again in the future.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-41-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
__enter__ can be invoked from a subclass, so it needs a more flexible
type.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-31-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This is meant to represent any generic object seen in a QMPMessage, not
just the root object itself.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-27-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Remove the shebang, and add a package-defined entry point instead. Now,
it can be accessed using 'qemu-ga-client' from the command line after
installing the package.
The next commit adds a forwarder shim that allows the running of this
script without needing to install the package again.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The script itself will be unavailable for a few commits before being
restored, with no way to run it right after this commit. This helps move
git history into the new file. To prevent linter regressions, though, we
do need to immediately touch up the filename to remove dashes (to make
the module importable), and remove the executable bit.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
It's only a Dict[str, Any] most of the time. It's not actually
guaranteed to be anything in particular. Fix this type to be
more accurate to the reality we live in.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The 'fuse' command will be unavailable if 'fusepy' is not installed. It
will simply not load and subsequently be unavailable as a subcommand.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Move qom-fuse over to the python package now that it passes the
linter. Update the import paradigms so that it continues to pass in the
context of the Python package.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
In preparation for moving qom-fuse over to the python package, we need
some new dependencies to support it.
Add an optional 'fusepy' dependency that users of the package can opt
into with e.g. "pip install qemu[fuse]" which installs the requirements
necessary to obtain the additional functionality.
Add the same fusepy dependency to the 'devel' extras group --
unfortunately I do not see a way for optional groups to imply other
optional groups at present, so the dependency is repeated. The
development group needs to include the full set of dependencies for the
purpose of static analysis of all features offered by this library.
Lastly, add the [fuse] extras group to tox's configuration as a
workaround so that if a stale tox environment is found when running
`make check-tox`, tox will know to rebuild its environments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Because fusepy does not have type hints, add some targeted warning
suppressions.
Namely, we need to allow subclassing something of an unknown type (in
qom_fuse.py), and we need to allow missing imports (recorded against
fuse itself) because mypy will be unable to import fusepy (even when
installed) as it has no types nor type stubs available.
Note: Until now, it was possible to run invocations like 'mypy qemu/'
from ./python and have that work. However, these targeted suppressions
require that you run 'mypy -p qemu/' instead. The correct, canonical
invocation is recorded in ./python/tests/mypy.sh and all of the various
CI invocations always use this correct form.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
fd and fh are fine: we often use these for "file descriptor" or "file
handle" accordingly. It is rarely the case that you need to enforce a
more semantically meaningful name beyond "This is the file we are using
right now."
While we're here: add comments for all of the non-standard pylint
names. (And the underscore.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Add the 'qom', 'qom-set', 'qom-get', 'qom-list', and 'qom-tree' scripts
to the qemu.qmp package. When you install this package, these scripts
will become available on your command line.
(e.g. when inside of a venv, `cd python && pip install .` will add
'qom', 'qom-set', etc to your $PATH.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Inspired by qom-set, qom-get, qom-tree and qom-list; combine all four of
those scripts into a single script.
A later addition of qom-fuse as an 'extension' necessitates that some
common features are split out and shared between them.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This takes the place of qmp-shell's __get_address function. It also
allows other utilities to share the same parser and syntax for
specifying QMP locations.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
In porting the qom tools, qmp-shell, etc; it becomes evident that this
type is wrong.
This is an integer, not a string. We didn't catch this before because
none of QEMUMonitorProtocol's *users* happen to be checked, and the
internal logic of this class is otherwise self-consistent. Additionally,
mypy was not introspecting into the socket() interface to realize we
were passing a bad type for AF_INET. Fixed now.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
In a previous commit, I added tox to the development requirements of the
Python library. I never bothered to add them to the Pipfile, because
they aren't needed there. Here, I sync it anyway in its own commit so
that when we add new packages later that the diffstats will not
confusingly appear to pull in lots of extra packages.
Ideally I could tell Pipenv simply not to install these, but it doesn't
seem to support that, exactly. The alternative is removing Tox from the
development requires, which I'd rather not do.
The other alternative is re-specifying all of the dependencies of
setup.cfg in the Pipfile, which I'd also rather not do.
Picking what feels least-worst here.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This is intended to be a manually run, non-CI script.
Use tox to test the linters against all python versions from 3.6 to
3.10. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:
> sudo dnf install python3.6 python3.7 python3.8 python3.9 python3.10
Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.
With confidence that the tests pass on 3.6 through 3.10 inclusive, add
the appropriate classifiers to setup.cfg to indicate which versions we
claim to support.
Tox 3.18.0 or above is required to use the 'allowlist_externals' option.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-31-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build. They are created by
running the commands 'python3 setup.py <sdist|bdist>' when preparing
tarballs to upload to e.g. PyPI.
Ignore miscellaneous cached python confetti (mypy, pylint, et al)
Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-30-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Add "make venv" to create the pipenv-managed virtual environment that
contains our explicitly pinned dependencies.
Add "make check" to run the python linters [in the host execution
environment].
Add "make venv-check" which combines the above two: create/update the
venv, then run the linters in that explicitly managed environment.
Add "make develop" which canonizes the runes needed to get both the
linting pre-requisites (the "[devel]" part), and the editable
live-install (the "-e" part) of these python libraries.
make clean: delete miscellaneous python packaging output possibly
created by pipenv, pip, or other python packaging utilities
make distclean: delete the above, the .venv, and the editable "qemu"
package forwarder (qemu.egg-info) if there is one.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-29-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Try using avocado to manage our various tests; even though right now
they're only invoking shell scripts and not really running any
python-native code.
Create tests/, and add shell scripts which call out to mypy, flake8,
pylint and isort to enforce the standards in this directory.
Add avocado-framework to the setup.cfg development dependencies, and add
avocado.cfg to store some preferences for how we'd like the test output
to look.
Finally, add avocado-framework to the Pipfile environment and lock the
new dependencies. We are using avocado >= 87.0 here to take advantage of
some features that Cleber has helpfully added to make the test output
here *very* friendly and easy to read for developers that might chance
upon the output in Gitlab CI.
[Note: ALL of the dependencies get updated to the most modern versions
that exist at the time of this writing. No way around it that I have
seen. Not ideal, but so it goes.]
Provided you have the right development dependencies (mypy, flake8,
isort, pylint, and now avocado-framework) You should be able to run
"avocado --config avocado.cfg run tests/" from the python folder to run
all of these linters with the correct arguments.
(A forthcoming commit adds the much easier 'make check'.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-28-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
setuptools doesn't have a formal understanding of development requires,
but it has an optional feataures section. Fine; add a "devel" feature
and add the requirements to it.
To avoid duplication, we can modify pipenv to install qemu[devel]
instead. This enables us to run invocations like "pip install -e
.[devel]" and test the package on bleeding-edge packages beyond those
specified in Pipfile.lock.
Importantly, this also allows us to install the qemu development
packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
will now fail if the proper development dependencies are not already
met. This can be useful for automated build scripts where fetching
network packages may be undesirable.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-27-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)
The qemu packages are installed in 'editable' mode; all changes made to
the python package inside the git tree will be reflected in the
installed package without reinstallation. This includes changes made
via git pull and so on.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-26-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.
isort < 5.1.1 has a bug where it does not handle comments near import
statements correctly.
Require 5.1.2 or greater.
isort can be run (in "check" mode) with 'isort -c qemu' from the python
root. isort can also be used to fix/rewrite import order automatically
by using 'isort qemu'.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-25-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-24-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.
0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.
Now that we are checking a namespace package, we need to tell mypy to
allow PEP420 namespaces, so modify the mypy config as part of the move.
mypy can now be run from the python root by typing 'mypy -p qemu'.
A note on mypy invocation: Running it as "mypy qemu/" changes the import
path detection mechanisms in mypy slightly, and it will fail. See
https://github.com/python/mypy/issues/8584 for a decent entry point with
more breadcrumbs on the various behaviors that contribute to this subtle
difference.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-23-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
mypy supports reading its configuration values from a central project
configuration file; do so.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-22-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.
flake8 can be run from the python/ directory with no arguments.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-21-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Instruct flake8 to avoid certain well-known directories created by
python tooling that it ought not check.
Note that at-present, nothing actually creates a ".venv" directory; but
it is in such widespread usage as a de-facto location for a developer's
virtual environment that it should be excluded anyway. A forthcoming
commit canonizes this with a "make venv" command.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:
A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-19-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
We are specifying >= pylint 2.8.x for several reasons:
1. For setup.cfg support, added in pylint 2.5.x
2. To specify a version that has incompatibly dropped
bad-whitespace checks (2.6.x)
3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9
4. 2.8.x adds a new, incompatible 'consider-using-with'
warning that must be disabled in some cases.
These pragmas cause warnings themselves in 2.7.x.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Delete the empty settings now that it's sharing a home with settings for
other tools.
pylint can now be run from this folder as "pylint qemu".
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.
pipenv uses two files to do this:
(1) Pipfile, which is similar in purpose and scope to what setup.cfg
lists. It specifies the requisite minimum to get a functional
environment for using this package.
(2) Pipfile.lock, which is similar in purpose to `pip freeze >
requirements.txt`. It specifies a canonical virtual environment used for
deployment or testing. This ensures that all users have repeatable
results.
The primary benefit of using this tool is to ensure *rock solid*
repeatable CI results with a known set of packages. Although I endeavor
to support as many versions as I can, the fluid nature of the Python
toolchain often means tailoring code for fairly specific versions.
Note that pipenv is *not* required to install or use this module; this is
purely for the sake of repeatable testing by CI or developers.
Here, a "blank" pipfile is added with no dependencies, but specifies
Python 3.6 for the virtual environment.
Pipfile will specify our version minimums, while Pipfile.lock specifies
an exact loadout of packages that were known to operate correctly. This
latter file provides the real value for easy setup of container images
and CI environments.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
When creating a source or binary distribution via 'python3 setup.py
<sdist|bdist>', the VERSION and PACKAGE.rst files aren't bundled by
default. Create a MANIFEST.in file that instructs the build tools to
include these so that installation from these files won't fail.
This is required by 'tox', as well as by the tooling needed to upload
packages to PyPI.
Exclude the 'README.rst' file -- that's intended as a guidebook to our
source tree, not a file that needs to be distributed.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source tree.
They are not designed for inclusion in a published manual.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible.
The problem is that when using pip to build and install a python
package, it copies files over to a temporary directory and performs its
build there. This loses access to any information in the parent
directory, including git itself.
Further, Python versions have a standard (PEP 440) that may or may not
follow QEMU's versioning. In general, it does; but naturally QEMU does
not follow PEP 440. To avoid any automatically-generated conflict, a
manual version file is preferred.
I am proposing:
- Python tooling follows the QEMU version, indirectly, but with a major
version of 0 to indicate that the API is not expected to be
stable. This would mean version 0.5.2.0, 0.5.1.1, 0.5.3.0, etc.
- In the event that a Python package needs to be updated independently
of the QEMU version, a pre-release alpha version should be preferred,
but *only* after inclusion to the qemu development or stable branches.
e.g. 0.5.2.0a1, 0.5.2.0a2, and so on should be preferred prior to
5.2.0's release.
- The Python core tooling makes absolutely no version compatibility
checks or constraints. It *may* work with releases of QEMU from the
past or future, but it is not required to.
i.e., "qemu.machine" will, for now, remain in lock-step with QEMU.
- We reserve the right to split the qemu package into independently
versioned subpackages at a later date. This might allow for us to
begin versioning QMP independently from QEMU at a later date, if
we so choose.
Implement this versioning scheme by adding a VERSION file and setting it
to 0.6.0.0a1.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
this package is for and who to contact for more information. This
document will be used as the landing page for the package on PyPI.
List the subpackages we intend to package by name instead of using
find_namespace because find_namespace will naively also packages tests,
things it finds in the dist/ folder, etc. I could not figure out how to
modify this behavior; adding allow/deny lists to setuptools kept
changing the packaged hierarchy. This works, roll with it.
I am not yet using a pyproject.toml style package manifest, because
"editable" installs are not defined (yet?) by PEP-517/518.
I consider editable installs crucial for development, though they have
(apparently) always been somewhat poorly defined.
Pip now (19.2 and later) now supports editable installs for projects
using pyproject.toml manifests, but might require the use of the
--no-use-pep517 flag, which somewhat defeats the point. Full support for
setup.py-less editable installs was not introduced until pip 21.1.1:
7a95720e79
For now, while the dust settles, stick with the de-facto
setup.py/setup.cfg combination supported by setuptools. It will be worth
re-evaluating this point again in the future when our supported build
platforms all ship a fairly modern pip.
Additional reading on this matter:
https://github.com/pypa/packaging-problems/issues/256https://github.com/pypa/pip/issues/6334https://github.com/pypa/pip/issues/6375https://github.com/pypa/pip/issues/6434https://github.com/pypa/pip/issues/6438
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
move python/qemu/*.py to python/qemu/[machine, qmp, utils]/*.py and
update import directives across the tree.
This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into new 'machine',
'qmp' and 'utils' subpackages instead.
Implement machine/__init__.py making the top-level classes and functions
from its various modules available directly inside the package. Change
qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
library classes are available directly from "qemu.qmp" instead of
"qemu.qmp.qmp".
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
One more little delinting fix that snuck in.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
We handle this resource rather meticulously in
shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in
_do_shutdown().
Quiet this pylint warning here.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-7-jsnow@redhat.com
Message-id: 20210517184808.3562549-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Shift the open() call later so that the pylint pragma applies *only* to
that one open() call. Add a note that suggests why this is safe: the
resource is unconditionally cleaned up in _post_shutdown().
_post_shutdown is called after failed launches (see launch()), and
unconditionally after every call to shutdown(), and therefore also on
__exit__.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-6-jsnow@redhat.com
Message-id: 20210517184808.3562549-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
use run() instead of Popen() -- to assert to pylint that we are not
forgetting to close a long-running program.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-4-jsnow@redhat.com
Message-id: 20210517184808.3562549-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
One less file resource to manage, and it helps quiet some pylint >=
2.8.0 warnings about not using a with-context manager for the open call.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-3-jsnow@redhat.com
Message-id: 20210517184808.3562549-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Slightly different versions for the same utility code are currently
present on different locations. This unifies them all, giving
preference to the version from virtiofs_submounts.py, because of the
last tweaks added to it.
While at it, this adds a "qemu.utils" module to host the utility
function and a test.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-4-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
[Squashed in below fix. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210601154546.130870-2-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Each instance of qemu.machine.QEMUMachine currently has a "test
directory", which may not have any relation to a "test", and it's
really a temporary directory.
Users instantiating the QEMUMachine class will be able to set the
location of the directory that will *contain* the QEMUMachine unique
temporary directory, so that parameter name has been changed from
test_dir to base_temp_dir.
A property has been added to allow users to access it without using
private attributes, and with that, the directory is created on first
use of the property.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-3-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Closing a file that is open for writing, and then reading from it
sounds like a better idea than the opposite, given that the content
will be flushed.
Reference: https://docs.python.org/3/library/io.html#io.IOBase.close
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-2-crosa@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
While attempting to debug some console weirdness I thought it would be
worth making it easier to see what it had inside.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20201210190417.31673-6-alex.bennee@linaro.org>
The first step to debug a thing is to know what created the thing in
the first place. Add some prefixes so random tmpdir's have something
grep in the code.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201117173635.29101-3-alex.bennee@linaro.org>
We enabled callers to interface directly with settimeout, but this
reacts poorly with blocking/nonblocking operation; as they are using the
same internal mechanism.
1. Whenever we change the blocking mechanism temporarily, always set it
back to what it was afterwards.
2. Disallow callers from setting a timeout of "0", which means
Non-blocking mode. This is going to create more weird problems than
anybody wants, so just forbid it.
I opt not to coerce '0' to 'None' to maintain the principal of least
surprise in mirroring the semantics of Python's interface.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201009175123.249009-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Nested if conditions don't change when the exception block fires; we
need to explicitly re-raise the error if we didn't intend to capture and
suppress it.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201009175123.249009-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Formalize the options used for checking the python library. You can run
mypy from the directory that mypy.ini is in by typing `mypy qemu/`.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20201009175123.249009-2-jsnow@redhat.com
[Edit: Added newline; thanks Bin Meng --js]
Signed-off-by: John Snow <jsnow@redhat.com>
Use the "from ..." phrasing when re-raising errors to preserve their
initial context, to help aid debugging when things go wrong.
This also silences a pylint 2.6.0+ error.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
We can work directly in bytes instead of translating back and forth to
string, which removes the question of which encodings to use.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Finish the typing of console_socket.py with annotations and no code
changes.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Mypy needs just a little help to guess the type here.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The types and names of the parameters must match the socket.socket interface.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The type and parameter names of recv() should match socket.socket().
OK, easy enough, but in the cases we don't pass straight through to the
real socket implementation, we probably can't accept such flags. OK, for
now, assert that we don't receive flags in such cases.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
These should all be purely annotations with no changes in behavior at
all. You need to be in the python folder, but you should be able to
confirm that these annotations are correct (or at least self-consistent)
by running `mypy --strict qemu`.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
These arguments don't need to be mutable and aren't really used as
such. Clarify their types as immutable and adjust code to match where
necessary.
In general, It's probably best not to accept a user-defined mutable
object and store it as internal object state unless there's a strong
justification for doing so. Instead, try to use generic types as input
with empty tuples as the default, and coerce to list where necessary.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201006235817.3280413-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
As always, Optional[T] causes problems with unchecked access. Add a
helper that asserts the pipe is present before we attempt to talk with
it.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Like many other Optional[] types, it's not always a given that this
object will be set. Wrap it in a type-shim that raises a meaningful
error and will always return a concrete type.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
machine.py and qmp.py both do the same thing here; refactor machine.py
to use qmp.py's functionality more directly.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201006235817.3280413-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
If the timeout is 0, we can get None back. Handle this explicitly.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Don't append to the _remove_files list during _base_args; instead do so
during _launch. Rework _base_args as a @property to help facilitate
this impression.
This has the additional benefit of making the type of _console_address
easier to analyze statically.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Put the init arg handling all at the top, and mostly in order (deviating
when one is dependent on another), and put what is effectively runtime
state declaration at the bottom.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Prior to this, it's difficult for mypy to intuit what the concrete type
of the monitor address is; it has difficulty inferring the type across
two variables.
Create _monitor_address as a property that always returns a valid
address to simplify static type analysis.
To preserve our ability to clean up, use a simple boolean to indicate
whether or not we should try to clean up the sock file after execution.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
Borrowed from the QAPI cleanup series, use the same configuration to
standardize the way we write and sort imports.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
The primary purpose of this change is to clean up
machine.py's console_socket property to return a single type,
a ConsoleSocket.
ConsoleSocket now derives from a socket, which means that
in the default case (of not draining), machine.py
will see the same behavior as it did prior to ConsoleSocket.
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200717203041.9867-3-robert.foley@linaro.org>
Message-Id: <20200724064509.331-16-alex.bennee@linaro.org>
The changes to console_socket.py and machine.py are to
cleanup for pylint and flake8.
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200717203041.9867-2-robert.foley@linaro.org>
Message-Id: <20200724064509.331-15-alex.bennee@linaro.org>
3 seconds is too short for some tests running inside busy VMs. Build it out to
a rather generous 30 seconds to find out conclusively if there are more severe
problems in the merge/CI tests.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-id: 20200720160252.104139-2-jsnow@redhat.com
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
In the case that we receive a reply but are unable to understand it,
use this exception name to indicate that case.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-7-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
mypy and python type hints are not powerful enough to properly describe
JSON messages in Python 3.6. The best we can do, generally, is describe
them as Dict[str, Any].
Add casts to coerce this type for static analysis; but do NOT enforce
this type at runtime in any way.
Note: Python 3.8 adds a TypedDict construct which allows for the
description of more arbitrary Dictionary shapes. There is a third-party
module, "Pydantic", which is compatible with 3.6 that can be used
instead of the JSON library that parses JSON messages to fully-typed
Python objects, and may be preferable in some cases.
(That is well beyond the scope of this commit or series.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-6-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This makes typing the qmp library difficult, as it necessitates wrapping
Optional[] around the type for every return type up the stack. At some
point, it becomes difficult to discern or remember why it's None instead
of the expected object.
Use the python exception system to tell us exactly why we didn't get an
object. Remove this special-cased return.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-5-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
When I initially split this out, I considered this more of a machine
error than a QMP protocol error, but I think that's misguided.
Move this back to qmp.py and name it QMPResponseError. Convert
qmp.command() to use this exception type.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-4-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Define some common types that we'll need to annotate a lot of other
functions going forward.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-2-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Machine.wait() does not appear to be used except in the acceptance tests,
and an infinite timeout by default in a test suite is not the most helpful.
Change it to 3 seconds, like the default shutdown timeout.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-13-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
If the user kills QEMU on purpose, we don't need to warn
them about that having happened: they know already.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-12-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This is done primarily to avoid the 'bare except' pattern, which
suppresses all exceptions during shutdown and can obscure errors.
Replace this with a pattern that isolates the different kind of shutdown
paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown
handler (_do_shutdown) that gracefully attempts one before the other.
This split now also ensures that no matter what happens,
_post_shutdown() is always invoked.
shutdown() changes in behavior such that if it attempts to do a graceful
shutdown and is unable to, it will now always raise an exception to
indicate this. This can be avoided by the test writer in three ways:
1. If the VM is expected to have already exited or is in the process of
exiting, wait() can be used instead of shutdown() to clean up resources
instead. This helps avoid race conditions in shutdown.
2. If a test writer is expecting graceful shutdown to fail, shutdown
should be called in a try...except block.
3. If the test writer has no interest in performing a graceful shutdown
at all, kill() can be used instead.
Handling shutdown in this way makes it much more explicit which type of
shutdown we want and allows the library to report problems with this
process.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-11-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
At this point, shutdown(has_quit=True) and wait() do essentially the
same thing; they perform cleanup without actually instructing QEMU to
quit.
Define one in terms of the other.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-8-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Three seconds is hardcoded. Use it as a default parameter instead, and use that
value for both waits that may occur in the function.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-7-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
If the VM is not launched, don't try to shut it down. As a change,
_post_shutdown now unconditionally also calls _early_cleanup in order to
offer comprehensive object cleanup in failure cases.
As a courtesy, treat it as a NOP instead of rejecting it as an
error. This is slightly nicer for acceptance tests where vm.shutdown()
is issued unconditionally in tearDown callbacks.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200710050649.32434-6-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This is primarily for consistency, and is a step towards wait() and
shutdown() sharing the same implementation so that the two cleanup paths
cannot diverge.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-5-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Some parts of cleanup need to occur prior to shutdown, otherwise
shutdown might break. Move this into a suitably named method/callback.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-4-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
It's not important to do this before waiting for the process to exit, so
it can be done during generic post-shutdown cleanup.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-3-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Move more cleanup actions into _post_shutdown. As a change, if QEMU
should so happen to be terminated during a call to wait(), that event
will now be logged.
This is not likely to occur during normative use.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200710050649.32434-2-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
We add the ConsoleSocket object, which has a socket interface
and which will consume all arriving characters on the
socket, placing them into an in memory buffer.
This will also provide those chars via recv() as
would a regular socket.
ConsoleSocket also has the option of dumping
the console bytes to a log file.
We also give QEMUMachine the option of using ConsoleSocket
to drain and to use for logging console to a file.
By default QEMUMachine does not use ConsoleSocket.
This is added in preparation for use by basevm.py in a later commit.
This is a workaround we found was needed for basevm.py since
there is a known issue where QEMU will hang waiting
for console characters to be consumed.
Cc: Eduardo Habkost <ehabkost@redhat.com>
Cc: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200601211421.1277-9-robert.foley@linaro.org>
Message-Id: <20200701135652.1366-13-alex.bennee@linaro.org>
It can be None; so add assertions or exceptions where appropriate to
guard the access accordingly.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-30-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
In truth, if you don't do this, you'll just get a TypeError
exception. Now, you'll get an AssertionError.
Is this tangibly better? No.
Does mypy complain less? Yes.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-21-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
The type system doesn't want integers.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-15-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
mypy considers it incorrect to use `bool` to statically return false,
because it will assume that it could conceivably return True, and gives
different analysis in that case. Use a None return to achieve the same
effect, but make mypy happy.
Note: Pylint considers function signatures as code that might trip the
duplicate-code checker. I'd rather not disable this as it does not
trigger often in practice, so I'm disabling it as a one-off and filed a
change request; see https://github.com/PyCQA/pylint/issues/3619
Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-14-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Note:
A bug in typeshed (https://github.com/python/typeshed/issues/3977)
misinterprets the type of makefile(). Work around this by explicitly
stating that we are opening a text-mode file.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-13-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Use the Python3 style instead.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-12-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Mostly, ignore the "no bare except" rule, because flake8 is not
contextual and cannot determine if we re-raise. Pylint can, though, so
always prefer pylint for that.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200528222129.23826-5-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Bring our these files up to speed with pylint 2.5.0.
Add a pylintrc file to formalize which pylint subset
we are targeting.
The similarity ignore is there to suppress similarity
reports across imports, which for typing constants,
are going to trigger this report erroneously.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200528222129.23826-4-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Python 3.5 and above do not print a warning when logging is not
configured. As a library, it's best practice to leave logging
configuration to the client executable.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200514055403.18902-22-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Add method to hard-kill vm, without any quit commands.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200217150246.29180-19-vsementsov@virtuozzo.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
With a QEMU bug, it can happen that the QEMU process doesn't react to a
'quit' QMP command. If we got an exception during previous QMP
communication (e.g. iotests Timeout expiring), we could also be in an
inconsistent state where after sending 'quit' we immediately read an old
response and close the socket even though the 'quit' command wasn't
processed yet. Both cases would lead to a hanging test.
Fix this by waiting for the QEMU process to exit after sending 'quit'
with a timeout, and if it doesn't happen within three seconds, send
SIGKILL.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20200313083617.8326-3-kwolf@redhat.com
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
QEMUMachine writes some messages to the default logger.
But it sometimes hard to read the output if we have requests to
more than one VM.
This patch adds a label to the logger in the debug mode.
Signed-off-by: Oksana Vohchana <ovoshcha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200316103203.10046-1-ovoshcha@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Currently the QEMU Python module limits the QEMUMachine class to
use the first serial console.
Some machines/guest might use another console than the first one as
the 'boot console'. For example the Raspberry Pi uses the second
(AUX) console.
To be able to use the Nth console as default, we simply need to
connect all the N - 1 consoles to the null chardev.
Add an index argument, so we can use a specific serial console as
default.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Liam Merwick <liam.merwick@oracle.com>
Tested-by: Liam Merwick <liam.merwick@oracle.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20200120235159.18510-5-f4bug@amsat.org>
[PMD: zero-initialize _console_index in __init__()]
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
In case qemu process dies the "monitor.cmd" returns None which gets
passed to the "__negotiate_capabilities" and leads to unhandled
exception. Let's only check the resp in case it has a value.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20200120071202.30646-1-ldoktor@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
On ppc64le, the accel.kvm_available() check may wrongly
return False because the host arch (as returned by os.uname[4])
and the target arch (ppc64) mismatch. In order to solve this
it is added an ppc64le -> ppc64 mapping which is used as an
fallback verification.
Fixes: 53a049d7d7
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200205203250.30526-5-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
The `error` and `timeout` attributes in QEMUMonitorProtocol are
not used, so this delete them.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191227134101.244496-6-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This implement the __enter__ and __exit__ functions on
QEMUMonitorProtocol class so that it can be used on 'with'
statement and the resources will be free up on block end:
with QEMUMonitorProtocol(socket_path) as qmp:
qmp.connect()
qmp.command('query-status')
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20200204141111.3207-5-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Currently the timeout of QEMUMonitorProtocol.accept() is
hard-coded to 15.0 seconds. This added the parameter `timeout`
so the value can be configured by the user.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20200204141111.3207-4-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
This clean up the pylint-3 report on qmp:
************* Module qemu.qmp
python/qemu/qmp.py:1:0: C0111: Missing module docstring (missing-docstring)
python/qemu/qmp.py:17:0: C0111: Missing class docstring (missing-docstring)
python/qemu/qmp.py:21:0: C0111: Missing class docstring (missing-docstring)
python/qemu/qmp.py:25:0: C0111: Missing class docstring (missing-docstring)
python/qemu/qmp.py:29:0: C0111: Missing class docstring (missing-docstring)
python/qemu/qmp.py:33:0: C0111: Missing class docstring (missing-docstring)
python/qemu/qmp.py:33:0: R0205: Class 'QEMUMonitorProtocol' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
python/qemu/qmp.py:80:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
python/qemu/qmp.py:131:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
python/qemu/qmp.py:159:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
python/qemu/qmp.py:245:4: C0111: Missing method docstring (missing-docstring)
python/qemu/qmp.py:249:4: C0111: Missing method docstring (missing-docstring)
python/qemu/qmp.py:252:4: C0111: Missing method docstring (missing-docstring)
python/qemu/qmp.py:255:4: C0111: Missing method docstring (missing-docstring)
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191227134101.244496-3-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
The socket.error is deprecated from Python 3.3, instead it is
made a link to OSError. This change replaces the occurences
of socket.error with OSError.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20191227134101.244496-2-wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
__init_.py import some sub-modules unnecessarily. So let's
clean it up.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Suggested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-6-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This adds a method to check if the tcg accelerator is enabled
in the QEMU binary.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-5-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Currently kvm_available() checks for the presence of kvm module
and, if target and host arches don't mismatch. This patch adds
an 3rd checking: if QEMU binary was compiled with kvm
support.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-4-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Since commit cbe6d6365a the command `qemu -accel help` returns
the list of accelerators enabled in the QEMU binary. This adds
the list_accel() method which return that same list.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20191216191438.93418-3-wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This creates the 'accel' Python module to be the home for
utilities that deal with accelerators. Also moved kvm_available()
from __init__.py to this new module.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191216191438.93418-2-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
The QEMUMachine VM has a monitor setup on which an QMP
connection is always attempted on _post_launch() (executed
by launch()). In case the QEMU process immediatly exits
then the qmp.accept() (used to establish the connection) stalls
until it reaches timeout and consequently an exception raises.
That behavior is undesirable when, for instance, it needs to
gather information from the QEMU binary ($ qemu -cpu list) or a
test which launches the VM expecting its failure.
This patch adds the set_qmp_monitor() method to QEMUMachine that
allows turn off the creation of the monitor machinery on VM launch.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191211185536.16962-2-wainersm@redhat.com>
[Cleber: trivial indentation fix]
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Currently, the console socket on QEMUMachine is closed after the QMP
command to gracefully exit QEMU is executed. Because of a possible
deadlock (QEMU waiting for the socket to become writable) let's close
the console socket earlier.
Reference: <20190607034214.GB22416@habkost.net>
Reference: https://bugs.launchpad.net/qemu/+bug/1829779
From: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20190911023558.4880-2-crosa@redhat.com>
iotests.py itself does not store socket files, but machine.py and
qtest.py do. iotests.py needs to pass the respective path to them, and
they need to adhere to it.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 20191017133155.5327-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
We've got a separate option to configure the accelerator nowadays, which
is shorter to type and the preferred way of specifying an accelerator.
Use it in the source and examples to show that it is the favored option.
(However, do not touch the places yet which also specify other machine
options or multiple accelerators - these are currently still better
handled with one single "-machine" statement instead)
Signed-off-by: Thomas Huth <thuth@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20190904052739.22123-1-thuth@redhat.com>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>
If a test has issued a quit command already (which may be useful to do
explicitly because the test wants to show its effects),
QEMUMachine.shutdown() should not do so again. Otherwise, the VM may
well return an ECONNRESET which will lead QEMUMachine.shutdown() to
killing it, which then turns into a "qemu received signal 9" line.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Since we're out in a new module, do a quick cursory pass of some of the
more obvious style issues.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20190627212816.27298-3-jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
It's not obvious that something named __init__.py actually houses
important code that isn't relevant to python packaging glue. Move the
QEMUMachine and related error classes out into their own module.
Adjust users to the new import location.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20190627212816.27298-2-jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Before, event_match didn't always recurse if the event value was not a
dictionary, and would instead check for equality immediately.
By delaying equality checking to post-recursion, we can allow leaf
values like "5" to match "None" and take advantage of the generic
None-returns-True clause.
This makes the matching a little more obviously consistent at the
expense of being able to check for explicit None values, which is
probably not that important given what this function is used for.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190528183857.26167-1-jsnow@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-4-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
The set_console() utility function either adds a device based on the
explicitly given device type, or adds a known good type of device
based on the machine type.
But, for a number of machine types, it may be impossible or
inconvenient to add the devices by means of "-device" command line
options, and then it may better to just use the "-serial" option and
let QEMU itself, based on the machine type, set the device
accordingly.
To achieve that, the behavior of set_console() now flags the intention
to add a console device on launch(), and if no explicit device type is
given the "-serial" option is going to be added to the QEMU command
line, instead of raising exceptions.
Based on testing with different machine types, the CONSOLE_DEV_TYPES
is not necessary anymore, so it's being removed, as is the logic to
use it.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-13-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Python:
* introduce "python" directory with module namespace
* log QEMU launch command line on qemu.QEMUMachine
Acceptance Tests:
* initrd 4GiB+ test
* migration test
* multi vm support in test class
* bump Avocado version and drop "🥑 enable"
-----BEGIN PGP SIGNATURE-----
iQIcBAABCAAGBQJccE9jAAoJEGV+jTOl8gnzb1cP/j99kGbgfQJA4CftO9eRXdIm
FKms4Z42n7KPus+/DphgfOXGYaHzPcqJQNguQYHuPlWaM3DWNU0rcFfAi/QdcZC1
3iYMyQwiRubjnCMN0Ab4k+GhpCPW6fea6GTzyvqha4jNRhCIhx7v54GTDfxWESQp
nqW40gAONGSG98DdFgubxg1YYqt7zlI9EVogGixe1gO9SVDkMEe7uH8tPCl9mt2m
VjN7AeP/NTDmidiwu+2LwSpDC0UmpDAsFnxGI6rDcNx8NOnjSHkSHmtxNJ8j2uZz
9P0ncGui+LfivdQh/yiBgrjTWXEXAx/oHKQCz7r8uJ8f60eYLFtjTHm//2G7lG48
luLSnNKq/niM4k/vNhBQr0ByqoHHlpmqAjbmYqw7wdvImBbkXN2Gh9kjNs55S8VZ
Z7wTceC0G7pyM3LCdFnikyCXKoRxLZ3AXQ3YXFN0PgX/IsyHVuBWBGPFkPkLwcRa
JW3DEmwx/oeTg2MKp7iA3dGTUIarbsjp+R04erMznlLvE+NgmB8ENY8T+qZ6c+NM
ZNyp1MH2nuTJsYxY3CkVKwPUqNSoaTLkMxvoZW5rKQdtvNinCYZpaeHuBchaHJed
E63r0+1n9vAMH3PHDrypW5qjcjSDBOHS+8ajhr0jr2r+6grLQKYEP8q+PwubUaMq
BsS5jOb8gLGC8ESfZxx/
=dwff
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/cleber/tags/python-next-pull-request' into staging
Python queue, 2019-02-22
Python:
* introduce "python" directory with module namespace
* log QEMU launch command line on qemu.QEMUMachine
Acceptance Tests:
* initrd 4GiB+ test
* migration test
* multi vm support in test class
* bump Avocado version and drop "🥑 enable"
# gpg: Signature made Fri 22 Feb 2019 19:37:07 GMT
# gpg: using RSA key 657E8D33A5F209F3
# gpg: Good signature from "Cleber Rosa <crosa@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3
* remotes/cleber/tags/python-next-pull-request:
Acceptance tests: expect boot to extract 2GiB+ initrd with linux-v4.16
Acceptance tests: use linux-3.6 and set vm memory to 4GiB
tests.acceptance: adds simple migration test
tests.acceptance: adds multi vm capability for acceptance tests
scripts/qemu.py: log QEMU launch command line
Introduce a Python module structure
Acceptance tests: drop usage of "🥑 enable"
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Even when the launch of QEMU succeeds, it's useful to have the command
line recorded.
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20190202005610.24048-2-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This is a simple move of Python code that wraps common QEMU
functionality, and are used by a number of different tests
and scripts.
By treating that code as a real Python module, we can more easily:
* reuse code
* have a proper place for the module's own unittests
* apply a more consistent style
* generate documentation
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20190206162901.19082-2-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
