Some very minor housekeeping to make the linters happy once more.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20231004230532.3002201-4-jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The error message is bad when the section is untagged. For instance,
test case doc-interleaved-section produces "'@foobar:' can't follow
'Note' section", which is okay, but if we drop the "Note:" tag, we get
"'@foobar:' can't follow 'None' section, which is bad.
Change the error message to "description of '@foobar:' follows a
section".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230510141637.3685080-1-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Conflict with commit 3e32dca3f0 resolved]
Two type hints fail centos-stream-8-x86_64 CI. They are actually
broken. Changing them to Optional[re.Match[str]] fixes them locally
for me, but then CI fails differently. Drop them for now.
Fixes: 3e32dca3f0 (qapi: Rewrite parsing of doc comment section symbols and tags)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230517061600.1782455-1-armbru@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-15-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
The QAPI schema doc comment language provides special syntax for
command and event arguments, struct and union members, alternate
branches, enumeration values, and features: descriptions starting with
"@name:".
By convention, we format them like this:
# @name: Lorem ipsum dolor sit amet, consectetur adipiscing elit,
# sed do eiusmod tempor incididunt ut labore et dolore
# magna aliqua.
Okay for names as short as "name", but we have much longer ones. Their
description gets squeezed against the right margin, like this:
# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could
# not avoid copying dirty pages. This is between
# 0 and @dirty-sync-count * @multifd-channels.
# (since 7.1)
The description text is effectively just 50 characters wide. Easy
enough to read, but can be cumbersome to write.
The awkward squeeze against the right margin makes people go beyond it,
which produces two undesirables: arguments about style, and descriptions
that are unnecessarily hard to read, like this one:
# @postcopy-vcpu-blocktime: list of the postcopy blocktime per vCPU. This is
# only present when the postcopy-blocktime migration capability
# is enabled. (Since 3.0)
We could instead format it like
# @postcopy-vcpu-blocktime:
# list of the postcopy blocktime per vCPU. This is only present
# when the postcopy-blocktime migration capability is
# enabled. (Since 3.0)
or, since the commit before previous, like
# @postcopy-vcpu-blocktime:
# list of the postcopy blocktime per vCPU. This is only present
# when the postcopy-blocktime migration capability is
# enabled. (Since 3.0)
However, I'd rather have
# @postcopy-vcpu-blocktime: list of the postcopy blocktime per vCPU.
# This is only present when the postcopy-blocktime migration
# capability is enabled. (Since 3.0)
because this is how rST field and option lists work.
To get this, we need to let the first non-blank line after the
"@name:" line determine expected indentation.
This fills up the indentation pitfall mentioned in
docs/devel/qapi-code-gen.rst. A related pitfall still exists. Update
the text to show it.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-14-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Work around lack of walrus operator in Python 3.7 and older]
To recognize a line starting with a section symbol and or tag, we
first split it at the first space, then examine the part left of the
space. We can just as well examine the unsplit line, so do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-13-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Work around lack of walrus operator in Python 3.7 and older]
When an argument's description starts on the line after the "#arg: "
line, indentation is stripped only from the description's first line,
as demonstrated by the previous commit. Moreover, subsequent lines
with less indentation are not rejected.
Make the first line's indentation the expected indentation for the
remainder of the description. This fixes indentation stripping, and
also requires at least that much indentation.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-12-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
When the lexer chokes on a stray character, its shows the characters
until the next structural character in the error message. It uses a
regular expression to match a non-empty string of non-structural
characters. Bug: the regular expression treats '"' as structural.
When the lexer chokes on '"', the match fails, and trips
must_match()'s assertion. Fix the regular expression.
Fixes: 14c3279502 (qapi: Improve reporting of lexical errors)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-4-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
With the two major JSON-ish type hierarchies clarified for distinct
purposes; QAPIExpression for parsed expressions and JSONValue for
introspection data, remove this FIXME as no longer an action item.
A third JSON-y data type, _ExprValue, is not meant to represent JSON in
the abstract but rather only the possible legal return values from a
single function, get_expr(). It isn't appropriate to attempt to merge it
with either of the above two types.
In theory, it may be possible to define a completely agnostic
one-size-fits-all JSON type hierarchy that any other user could borrow -
in practice, it's tough to wrangle the differences between invariant,
covariant and contravariant types: input and output parameters demand
different properties of such a structure.
However, QAPIExpression serves to authoritatively type user input to the
QAPI parser, while JSONValue serves to authoritatively type qapi
generator *output* to be served back to client users at runtime via
QMP. The AST for these two types are different and cannot be wholly
merged into a unified syntax.
They could, in theory, share some JSON primitive definitions. In
practice, this is currently more trouble than it's worth with mypy's
current expressive power. As such, declare this "done enough for now".
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
We can remove this alias as it only has two usages now, and no longer
pays for the confusion of "yet another type".
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
This patch creates a new type, QAPIExpression, which represents a parsed
expression complete with QAPIDoc and QAPISourceInfo.
This patch turns parser.exprs into a list of QAPIExpression instead,
and adjusts expr.py to match.
This allows the types we specify in parser.py to be "remembered" all the
way through expr.py and into schema.py. Several assertions around
packing and unpacking this data can be removed as a result.
It also corrects a harmless typing error. Before the patch,
check_exprs() allegedly takes a List[_JSONObject]. It actually takes
a list of dicts of the form
{'expr': E, 'info': I, 'doc': D}
where E is of type _ExprValue, I is of type QAPISourceInfo, and D is
of type QAPIDoc. Key 'doc' is optional. This is not a _JSONObject!
Passes type checking anyway, because _JSONObject is Dict[str, object].
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message amended to point out the typing fix]
Eh. Not worth the fuss today. There are bigger fish to fry.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-13-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The fix for this comment is forthcoming in a future commit, but this
will keep me honest. The linting configuration in ./python/setup.cfg
prohibits 'FIXME' comments. A goal of this long-running series is to
move ./scripts/qapi to ./python/qemu/qapi so that the QAPI generator is
regularly type-checked by GitLab CI.
This comment is a time-bomb to force me to address this issue prior to
that step.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Annotations do not change runtime behavior.
This commit consists of only annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Adding static types causes a cycle in the QAPI generator:
[schema -> expr -> parser -> schema]. It exists because the QAPIDoc
class needs the names of types defined by the schema module, but the
schema module needs to import both expr.py/parser.py to do its actual
parsing.
Ultimately, the layering violation is that parser.py should not have any
knowledge of specifics of the Schema. QAPIDoc performs double-duty here
both as a parser *and* as a finalized object that is part of the schema.
In this patch, add the offending type hints alongside the workaround to
avoid the cycle becoming a problem at runtime. See
https://mypy.readthedocs.io/en/latest/runtime_troubles.html#import-cycles
for more information on this workaround technique.
I see three ultimate resolutions here:
(1) Just keep this patch and use the TYPE_CHECKING trick to eliminate
the cycle which is only present during static analysis.
(2) Don't bother to annotate connect_member() et al, give them 'object'
or 'Any'. I don't particularly like this, because it diminishes the
usefulness of type hints for documentation purposes. Still, it's an
extremely quick fix.
(3) Reimplement doc <--> definition correlation directly in schema.py,
integrating doc fields directly into QAPISchemaMember and relieving
the QAPIDoc class of the responsibility. Users of the information
would instead visit the members first and retrieve their
documentation instead of the inverse operation -- visiting the
documentation and retrieving their members.
My preference is (3), but in the short-term (1) is the easiest way to
have my cake (strong type hints) and eat it too (Not have import
cycles). Do (1) for now, but plan for (3).
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-9-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Here's the weird bit. QAPIDoc generally expects -- virtually everywhere
-- that it will always have a current section. The sole exception to
this is in the case that end_comment() is called, which leaves us with
*no* section. However, in this case, we also don't expect to actually
ever mutate the comment contents ever again.
NullSection is just a Null-object that allows us to maintain the
invariant that we *always* have a current section, enforced by static
typing -- allowing us to type that field as QAPIDoc.Section instead of
the more ambiguous Optional[QAPIDoc.Section].
end_section is renamed to switch_section and now accepts as an argument
the new section to activate, clarifying that no callers ever just
unilaterally end a section; they only do so when starting a new section.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-8-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The "if self._section" clause in end_section is mysterious: In which
circumstances might we end a section when we don't have one?
QAPIDoc always expects there to be a "current section", only except
after a call to end_comment(). This actually *shouldn't* ever be 'None',
so let's remove that logic so I don't wonder why it's like this again in
three months.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
True, we do not check the validity of this symbol -- but we don't check
the validity of definition names during parse, either -- that happens
later, during the expr check. I don't want to introduce a dependency on
expr.py:check_name_str here and introduce a cycle.
Instead, rest assured that a documentation block is required for each
definition. This requirement uses the names of each section to ensure
that we fulfilled this requirement.
e.g., let's say that block-core.json has a comment block for
"Snapshot!Info" by accident. We'll see this error message:
In file included from ../../qapi/block.json:8:
../../qapi/block-core.json: In struct 'SnapshotInfo':
../../qapi/block-core.json:38: documentation comment is for 'Snapshot!Info'
That's a pretty decent error message.
Now, let's say that we actually mangle it twice, identically:
../../qapi/block-core.json: In struct 'Snapshot!Info':
../../qapi/block-core.json:38: struct has an invalid name
That's also pretty decent. If we forget to fix it in both places, we'll
just be back to the first error.
Therefore, let's just drop this FIXME and adjust the error message to
not imply a more thorough check than is actually performed.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Pylint informs us we're not using these arguments. Oops, it's
right. Correct the error message and remove the remaining unused
parameter.
Fix test output now that the error message is improved.
Fixes: e151941d1b
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-4-jsnow@redhat.com>
[Commit message formatting tweaked]
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
A generator suffices (and quiets a pylint warning).
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-14-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Annotations do not change runtime behavior.
This commit *only* adds annotations.
(Annotations for QAPIDoc are in a forthcoming commit.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-13-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
TypeGuards wont exist in Python proper until 3.10. Ah well. We can hack
up our own by declaring this function to return the type we claim it
checks for and using this to safely downcast object -> List[str].
In so doing, I bring this function under _pragma so it can use the
'info' object in its closure. Having done this, _pragma also now no
longer needs to take a 'self' parameter, so drop it.
To help with line-length, and with the context evident from its new
scope, rename the function to the shorter check_list_str().
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-12-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
When the token can be None (EOF), we can't use 'x in "abc"' style
membership tests to group types of tokens together, because 'None in
"abc"' is a TypeError.
Easy enough to fix. (Use a tuple: It's neither a static typing error nor
a runtime error to check for None in Tuple[str, ...])
Add tests to prevent a regression. (Note: they cannot be added prior to
this fix, as the unhandled stack trace will not match test output in the
CI system.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Mypy cannot generally understand that these regex functions cannot
possibly fail. Add a "must_match" helper that makes this clear for
mypy.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
No self, no thank you!
(Quiets pylint warnings.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-9-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The single quote token implies the value is a string. Assert this to be
the case, to allow us to write an accurate return type for get_members.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-8-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Instead of using get_expr nested=False, allow get_expr to always return
any expression. In exchange, add a new error message to the top-level
parser that explains the semantic error: Top-level expressions must
always be JSON objects.
This helps mypy understand the rest of this function which assumes that
get_expr did indeed return a dict.
The exception type changes from QAPIParseError to QAPISemError as a
result, and the error message in two tests now changes.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The type checker can't narrow the type of the token value to string,
because it's only loosely correlated with the return token.
We know that a token of '#' should always have a "str" value.
Add an assertion.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
For the sake of keeping __init__ smaller (and treating it more like a
gallery of what state variables we can expect to see), put the actual
parsing action into a parse method. It remains invoked from the init
method to reduce churn.
To accomplish this, @previously_included becomes the private data
member ._included, and the filename is stashed as ._fname.
Add any missing declarations to the init method, and group them by
function so they can be understood quickly at a glance.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
With the QAPISourceInfo(None, None, None) construct gone, there's no
longer any reason to have to specify that a file starts on the first
line. Remove it from the initializer and default it to 1.
Remove the last vestiges where we check for 'line' being unset, that
can't happen, now.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-4-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Fixes: f5d4361cda
Fixes: 52a474180a
Fixes: 46f49468c6
Remove the try/except block that handles file-opening errors in
QAPISchemaParser.__init__() and add one each to
QAPISchemaParser._include() and QAPISchema.__init__() respectively.
This simultaneously fixes the typing of info.fname (f5d4361cda), A
static typing violation in test-qapi (46f49468c6), and a regression of
an error message (52a474180a).
The short-ish version of what motivates this patch is:
- It's hard to write a good error message in the init method,
because we need to determine the context of our caller to do so.
It's easier to just let the caller write the message.
- We don't want to allow QAPISourceInfo(None, None, None) to exist. The
typing introduced by commit f5d4361cda types the 'fname' field as
(non-optional) str, which was premature until the removal of this
construct.
- Errors made using such an object are currently incorrect (since
52a474180a)
- It's not technically a semantic error if we cannot open the schema.
- There are various typing constraints that make mixing these two cases
undesirable for a single special case.
- test-qapi's code handling an fname of 'None' is now dead, drop it.
Additionally, Not all QAPIError objects have an 'info' field (since
46f49468), so deleting this stanza corrects a typing oversight in
test-qapi introduced by that commit.
Other considerations:
- open() is moved to a 'with' block to ensure file pointers are
cleaned up deterministically.
- Python 3.3 deprecated IOError and made it a synonym for OSError.
Avoid the misleading perception these exception handlers are
narrower than they really are.
The long version:
The error message here is incorrect (since commit 52a474180a):
> python3 qapi-gen.py 'fake.json'
qapi-gen.py: qapi-gen.py: can't read schema file 'fake.json': No such file or directory
In pursuing it, we find that QAPISourceInfo has a special accommodation
for when there's no filename. Meanwhile, the intent when QAPISourceInfo
was typed (f5d4361cda) was non-optional 'str'. This usage was
overlooked.
To remove this, I'd want to avoid having a "fake" QAPISourceInfo
object. I also don't want to explicitly begin accommodating
QAPISourceInfo itself being None, because we actually want to eventually
prove that this can never happen -- We don't want to confuse "The file
isn't open yet" with "This error stems from a definition that wasn't
defined in any file".
(An earlier series tried to create a dummy info object, but it was tough
to prove in review that it worked correctly without creating new
regressions. This patch avoids that distraction. We would like to first
prove that we never raise QAPISemError for any built-in object before we
add "special" info objects. We aren't ready to do that yet.)
So, which way out of the labyrinth?
Here's one way: Don't try to handle errors at a level with "mixed"
semantic contexts; i.e. don't mix inclusion errors (should report a
source line where the include was triggered) and command line errors
(where we specified a file we couldn't read).
Remove the error handling from the initializer of the parser. Pythonic!
Now it's the caller's job to figure out what to do about it. Handle the
error in QAPISchemaParser._include() instead, where we can write a
targeted error message where we are guaranteed to have an 'info' context
to report with.
The root level error can similarly move to QAPISchema.__init__(), where
we know we'll never have an info context to report with, so we use a
more abstract error type.
Now the error looks sensible again:
> python3 qapi-gen.py 'fake.json'
qapi-gen.py: can't read schema file 'fake.json': No such file or directory
With these error cases separated, QAPISourceInfo can be solidified as
never having placeholder arguments that violate our desired types. Clean
up test-qapi along similar lines.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210519183951.3946870-2-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Keeping it in error.py will create some cyclic import problems when we
add types to the QAPISchemaParser. Callers don't need to know the
details of QAPIParseError unless they are parsing or dealing directly
with the parser, so this won't create any harsh new requirements for
callers in the general case.
Update error.py with a little docstring that gives a nod to where the
error may now be found.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210421192233.3542904-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Command names should be lower-case. Enforce this. Fix the fixable
offenders (all in tests/), and add the remainder to pragma
command-name-exceptions.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20210323094025.3569441-25-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Rename pragma returns-whitelist to command-returns-exceptions, and
name-case-whitelist to member-name-case-exceptions.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20210323094025.3569441-20-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20210323094025.3569441-17-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Commit 9d55380b5a "qapi: Remove null from schema language" (v4.2.0)
neglected to update two error messages. Do that now.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20210224101442.1837475-1-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.
force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long
imports.
include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!
Suggested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-8-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
All of the QAPI include statements are changed to be package-aware, as
explicit relative imports.
A quirk of Python packages is that the name of the package exists only
*outside* of the package. This means that to a module inside of the qapi
folder, there is inherently no such thing as the "qapi" package. The
reason these imports work is because the "qapi" package exists in the
context of the caller -- the execution shim, where sys.path includes a
directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder
'qapi', but rather "any package named qapi in sys.path". If you should
so happen to have a 'qapi' package in your path, it will use *that*
package.
When we write "from .sibling import foo", we always reference explicitly
our sibling module; guaranteeing consistency in *where* we are importing
these modules from.
This can be useful when working with virtual environments and packages
in development mode. In development mode, a package is installed as a
series of symlinks that forwards to your same source files. The problem
arises because code quality checkers will follow "import qapi.x" to the
"installed" version instead of the sibling file and -- even though they
are the same file -- they have different module paths, and this causes
cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if
we allow qemu.core.qmp, qemu.qapi.parser, etc.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Make the handling of indentation in doc comments more sophisticated,
so that when we see a section like:
Notes: some text
some more text
indented line 3
we save it for the doc-comment processing code as:
some text
some more text
indented line 3
and when we see a section with the heading on its own line:
Notes:
some text
some more text
indented text
we also accept that and save it in the same form.
If we detect that the comment document text is not indented as much
as we expect it to be, we throw a parse error. (We don't complain
about over-indented sections, because for rST this can be legitimate
markup.)
The golden reference for the doc comment text is updated to remove
the two 'wrong' indents; these now form a test case that we correctly
stripped leading whitespace from an indented multi-line argument
definition.
We update the documentation in docs/devel/qapi-code-gen.txt to
describe the new indentation rules.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200925162316.21205-6-peter.maydell@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Whitespace between sentences tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
As we accumulate lines from doc comments when parsing the JSON, the
QAPIDoc class generally strips leading and trailing whitespace using
line.strip() when it calls _append_freeform(). This is fine for
Texinfo, but for rST leading whitespace is significant. We'd like to
move to having the text in doc comments be rST format rather than a
custom syntax, so move the removal of leading whitespace from the
QAPIDoc class to the texinfo-specific processing code in
texi_format() in qapi/doc.py.
(Trailing whitespace will always be stripped by the rstrip() in
Section::append regardless.)
In a followup commit we will make the whitespace in the lines of doc
comment sections more consistently follow the input source.
There is no change to the generated .texi files before and after this
commit.
Because the qapi-schema test checks the exact values of the
documentation comments against a reference, we need to update that
reference to match the new whitespace. In the first four places this
is now correctly checking that we did put in the amount of whitespace
to pass a rST-formatted list to the backend; in the last two places
the extra whitespace is 'wrong' and will go away again in the
following commit.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200925162316.21205-5-peter.maydell@linaro.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Our current QAPI doc-comment markup allows section headers (introduced
with a leading '=' or '==') anywhere in a free-form documentation
comment. This works for Texinfo because the generator simply prints a
Texinfo section command at that point in the output stream. For rST
generation, since we're assembling a tree of docutils nodes, this is
awkward because a new section implies starting a new section node at
the top level of the tree and generating text into there.
Make section headers start a new free-form documentation block, so the
future rST document generator doesn't have to look at every line in
free-form blocks and handle headings in odd places.
This change makes no difference to the generated Texinfo.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200320091805.5585-3-armbru@redhat.com>
Section markup in definition documentation makes no sense and can
produce invalid Texinfo. Reject.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200320091805.5585-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200304155932.20452-3-armbru@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200304155932.20452-2-armbru@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Commit f3ed93d545 "qapi: Allow documentation for features" neglected
to check documentation against the schema. Fix that: check them the
same way we check arguments.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20191024110237.30963-20-armbru@redhat.com>
Improve error messages from
the following documented members are not in the declaration: a
the following documented members are not in the declaration: aa, bb
to the more concise
documented member 'a' does not exist
documented members 'aa', 'bb' do not exist
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20191024110237.30963-19-armbru@redhat.com>
The QAPI code generator clocks in at some 3100 SLOC in 8 source files.
Almost 60% of the code is in qapi/common.py. Split it into more
focused modules:
* Move QAPISchemaPragma and QAPISourceInfo to qapi/source.py.
* Move QAPIError and its sub-classes to qapi/error.py.
* Move QAPISchemaParser and QAPIDoc to parser.py. Use the opportunity
to put QAPISchemaParser first.
* Move check_expr() & friends to qapi/expr.py. Use the opportunity to
put the code into a more sensible order.
* Move QAPISchema & friends to qapi/schema.py
* Move QAPIGen and its sub-classes, ifcontext,
QAPISchemaModularCVisitor, and QAPISchemaModularCVisitor to qapi/gen.py
* Delete camel_case(), it's unused since commit e98859a9b9 "qapi:
Clean up after recent conversions to QAPISchemaVisitor"
A number of helper functions remain in qapi/common.py. I considered
moving the code generator helpers to qapi/gen.py, but decided not to.
Perhaps we should rewrite them as methods of QAPIGen some day.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-7-armbru@redhat.com>
[Add "# -*- coding: utf-8 -*-" lines]
