Updates to the "lemon.html" document received from Andy Goth.
FossilOrigin-Name: 5b2002f3df1902aaa571a0efd01ab8bae7f4d37ac4819cc51595277f4de93433
This commit is contained in:
parent
cf51255c44
commit
9a243e69c2
379
doc/lemon.html
379
doc/lemon.html
@ -2,12 +2,12 @@
|
||||
<head>
|
||||
<title>The Lemon Parser Generator</title>
|
||||
</head>
|
||||
<body bgcolor=white>
|
||||
<h1 align=center>The Lemon Parser Generator</h1>
|
||||
<body bgcolor='white'>
|
||||
<h1 align='center'>The Lemon Parser Generator</h1>
|
||||
|
||||
<p>Lemon is an LALR(1) parser generator for C.
|
||||
It does the same job as "bison" and "yacc".
|
||||
But lemon is not a bison or yacc clone. Lemon
|
||||
But Lemon is not a bison or yacc clone. Lemon
|
||||
uses a different grammar syntax which is designed to
|
||||
reduce the number of coding errors. Lemon also uses a
|
||||
parsing engine that is faster than yacc and
|
||||
@ -16,7 +16,7 @@ bison and which is both reentrant and threadsafe.
|
||||
has also been updated so that it too can generate a
|
||||
reentrant and threadsafe parser.)
|
||||
Lemon also implements features that can be used
|
||||
to eliminate resource leaks, making is suitable for use
|
||||
to eliminate resource leaks, making it suitable for use
|
||||
in long-running programs such as graphical user interfaces
|
||||
or embedded controllers.</p>
|
||||
|
||||
@ -58,8 +58,8 @@ Lemon comes with a default parser template which works fine for most
|
||||
applications. But the user is free to substitute a different parser
|
||||
template if desired.</p>
|
||||
|
||||
<p>Depending on command-line options, Lemon will generate between
|
||||
one and three files of outputs.
|
||||
<p>Depending on command-line options, Lemon will generate up to
|
||||
three output files.
|
||||
<ul>
|
||||
<li>C code to implement the parser.
|
||||
<li>A header file defining an integer ID for each terminal symbol.
|
||||
@ -90,17 +90,20 @@ the states used by the parser automaton.</p>
|
||||
You can obtain a list of the available command-line options together
|
||||
with a brief explanation of what each does by typing
|
||||
<pre>
|
||||
lemon -?
|
||||
lemon "-?"
|
||||
</pre>
|
||||
As of this writing, the following command-line options are supported:
|
||||
<ul>
|
||||
<li><b>-b</b>
|
||||
Show only the basis for each parser state in the report file.
|
||||
<li><b>-c</b>
|
||||
Do not compress the generated action tables.
|
||||
Do not compress the generated action tables. The parser will be a
|
||||
little larger and slower, but it will detect syntax errors sooner.
|
||||
<li><b>-D<i>name</i></b>
|
||||
Define C preprocessor macro <i>name</i>. This macro is useable by
|
||||
"%ifdef" lines in the grammar file.
|
||||
Define C preprocessor macro <i>name</i>. This macro is usable by
|
||||
"<tt><a href='#pifdef'>%ifdef</a></tt>" and
|
||||
"<tt><a href='#pifdef'>%ifndef</a></tt>" lines
|
||||
in the grammar file.
|
||||
<li><b>-g</b>
|
||||
Do not generate a parser. Instead write the input grammar to standard
|
||||
output with all comments, actions, and other extraneous text removed.
|
||||
@ -108,9 +111,9 @@ output with all comments, actions, and other extraneous text removed.
|
||||
Omit "#line" directives in the generated parser C code.
|
||||
<li><b>-m</b>
|
||||
Cause the output C source code to be compatible with the "makeheaders"
|
||||
program.
|
||||
program.
|
||||
<li><b>-p</b>
|
||||
Display all conflicts that are resolved by
|
||||
Display all conflicts that are resolved by
|
||||
<a href='#precrules'>precedence rules</a>.
|
||||
<li><b>-q</b>
|
||||
Suppress generation of the report file.
|
||||
@ -165,7 +168,7 @@ once for each token:
|
||||
</pre>
|
||||
The first argument to the Parse() routine is the pointer returned by
|
||||
ParseAlloc().
|
||||
The second argument is a small positive integer that tells the parse the
|
||||
The second argument is a small positive integer that tells the parser the
|
||||
type of the next token in the data stream.
|
||||
There is one token type for each terminal symbol in the grammar.
|
||||
The gram.h file generated by Lemon contains #define statements that
|
||||
@ -173,7 +176,7 @@ map symbolic terminal symbol names into appropriate integer values.
|
||||
A value of 0 for the second argument is a special flag to the
|
||||
parser to indicate that the end of input has been reached.
|
||||
The third argument is the value of the given token. By default,
|
||||
the type of the third argument is integer, but the grammar will
|
||||
the type of the third argument is "void*", but the grammar will
|
||||
usually redefine this type to be some kind of structure.
|
||||
Typically the second argument will be a broad category of tokens
|
||||
such as "identifier" or "number" and the third argument will
|
||||
@ -181,7 +184,7 @@ be the name of the identifier or the value of the number.</p>
|
||||
|
||||
<p>The Parse() function may have either three or four arguments,
|
||||
depending on the grammar. If the grammar specification file requests
|
||||
it (via the <a href='#extraarg'><tt>extra_argument</tt> directive</a>),
|
||||
it (via the <tt><a href='#extraarg'>%extra_argument</a></tt> directive),
|
||||
the Parse() function will have a fourth parameter that can be
|
||||
of any type chosen by the programmer. The parser doesn't do anything
|
||||
with this argument except to pass it through to action routines.
|
||||
@ -191,20 +194,20 @@ to the action routines without having to use global variables.</p>
|
||||
<p>A typical use of a Lemon parser might look something like the
|
||||
following:
|
||||
<pre>
|
||||
01 ParseTree *ParseFile(const char *zFilename){
|
||||
02 Tokenizer *pTokenizer;
|
||||
03 void *pParser;
|
||||
04 Token sToken;
|
||||
05 int hTokenId;
|
||||
06 ParserState sState;
|
||||
07
|
||||
08 pTokenizer = TokenizerCreate(zFilename);
|
||||
09 pParser = ParseAlloc( malloc );
|
||||
10 InitParserState(&sState);
|
||||
11 while( GetNextToken(pTokenizer, &hTokenId, &sToken) ){
|
||||
12 Parse(pParser, hTokenId, sToken, &sState);
|
||||
1 ParseTree *ParseFile(const char *zFilename){
|
||||
2 Tokenizer *pTokenizer;
|
||||
3 void *pParser;
|
||||
4 Token sToken;
|
||||
5 int hTokenId;
|
||||
6 ParserState sState;
|
||||
7
|
||||
8 pTokenizer = TokenizerCreate(zFilename);
|
||||
9 pParser = ParseAlloc( malloc );
|
||||
10 InitParserState(&sState);
|
||||
11 while( GetNextToken(pTokenizer, &hTokenId, &sToken) ){
|
||||
12 Parse(pParser, hTokenId, sToken, &sState);
|
||||
13 }
|
||||
14 Parse(pParser, 0, sToken, &sState);
|
||||
14 Parse(pParser, 0, sToken, &sState);
|
||||
15 ParseFree(pParser, free );
|
||||
16 TokenizerFree(pTokenizer);
|
||||
17 return sState.treeRoot;
|
||||
@ -217,10 +220,10 @@ simple.)
|
||||
We assume the existence of some kind of tokenizer which is created
|
||||
using TokenizerCreate() on line 8 and deleted by TokenizerFree()
|
||||
on line 16. The GetNextToken() function on line 11 retrieves the
|
||||
next token from the input file and puts its type in the
|
||||
next token from the input file and puts its type in the
|
||||
integer variable hTokenId. The sToken variable is assumed to be
|
||||
some kind of structure that contains details about each token,
|
||||
such as its complete text, what line it occurs on, etc. </p>
|
||||
such as its complete text, what line it occurs on, etc.</p>
|
||||
|
||||
<p>This example also assumes the existence of structure of type
|
||||
ParserState that holds state information about a particular parse.
|
||||
@ -237,7 +240,7 @@ tree.</p>
|
||||
<pre>
|
||||
ParseFile(){
|
||||
pParser = ParseAlloc( malloc );
|
||||
while( GetNextToken(pTokenizer,&hTokenId, &sToken) ){
|
||||
while( GetNextToken(pTokenizer,&hTokenId, &sToken) ){
|
||||
Parse(pParser, hTokenId, sToken);
|
||||
}
|
||||
Parse(pParser, 0, sToken);
|
||||
@ -297,25 +300,25 @@ specifies additional information Lemon requires to do its job.
|
||||
Most of the work in using Lemon is in writing an appropriate
|
||||
grammar file.</p>
|
||||
|
||||
<p>The grammar file for lemon is, for the most part, free format.
|
||||
<p>The grammar file for Lemon is, for the most part, free format.
|
||||
It does not have sections or divisions like yacc or bison. Any
|
||||
declaration can occur at any point in the file.
|
||||
Lemon ignores whitespace (except where it is needed to separate
|
||||
tokens) and it honors the same commenting conventions as C and C++.</p>
|
||||
tokens), and it honors the same commenting conventions as C and C++.</p>
|
||||
|
||||
<h3>Terminals and Nonterminals</h3>
|
||||
|
||||
<p>A terminal symbol (token) is any string of alphanumeric
|
||||
and/or underscore characters
|
||||
that begins with an upper case letter.
|
||||
that begins with an uppercase letter.
|
||||
A terminal can contain lowercase letters after the first character,
|
||||
but the usual convention is to make terminals all upper case.
|
||||
but the usual convention is to make terminals all uppercase.
|
||||
A nonterminal, on the other hand, is any string of alphanumeric
|
||||
and underscore characters than begins with a lower case letter.
|
||||
Again, the usual convention is to make nonterminals use all lower
|
||||
case letters.</p>
|
||||
and underscore characters than begins with a lowercase letter.
|
||||
Again, the usual convention is to make nonterminals use all lowercase
|
||||
letters.</p>
|
||||
|
||||
<p>In Lemon, terminal and nonterminal symbols do not need to
|
||||
<p>In Lemon, terminal and nonterminal symbols do not need to
|
||||
be declared or identified in a separate section of the grammar file.
|
||||
Lemon is able to generate a list of all terminals and nonterminals
|
||||
by examining the grammar rules, and it can always distinguish a
|
||||
@ -339,7 +342,8 @@ The list of terminals and nonterminals on the right-hand side of the
|
||||
rule can be empty.
|
||||
Rules can occur in any order, except that the left-hand side of the
|
||||
first rule is assumed to be the start symbol for the grammar (unless
|
||||
specified otherwise using the <tt>%start</tt> directive described below.)
|
||||
specified otherwise using the <tt><a href='#start_symbol'>%start_symbol</a></tt>
|
||||
directive described below.)
|
||||
A typical sequence of grammar rules might look something like this:
|
||||
<pre>
|
||||
expr ::= expr PLUS expr.
|
||||
@ -382,7 +386,7 @@ names to each symbol in a grammar rule and then using those symbolic
|
||||
names in the action.
|
||||
In yacc or bison, one would write this:
|
||||
<pre>
|
||||
expr -> expr PLUS expr { $$ = $1 + $3; };
|
||||
expr -> expr PLUS expr { $$ = $1 + $3; };
|
||||
</pre>
|
||||
But in Lemon, the same rule becomes the following:
|
||||
<pre>
|
||||
@ -422,14 +426,14 @@ of the shift, and a reduce-reduce conflict is resolved by reducing
|
||||
whichever rule comes first in the grammar file.</p>
|
||||
|
||||
<p>Just like in
|
||||
yacc and bison, Lemon allows a measure of control
|
||||
over the resolution of paring conflicts using precedence rules.
|
||||
yacc and bison, Lemon allows a measure of control
|
||||
over the resolution of parsing conflicts using precedence rules.
|
||||
A precedence value can be assigned to any terminal symbol
|
||||
using the
|
||||
<a href='#pleft'>%left</a>,
|
||||
<a href='#pright'>%right</a> or
|
||||
<a href='#pnonassoc'>%nonassoc</a> directives. Terminal symbols
|
||||
mentioned in earlier directives have a lower precedence that
|
||||
using the
|
||||
<tt><a href='#pleft'>%left</a></tt>,
|
||||
<tt><a href='#pright'>%right</a></tt> or
|
||||
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives. Terminal symbols
|
||||
mentioned in earlier directives have a lower precedence than
|
||||
terminal symbols mentioned in later directives. For example:</p>
|
||||
|
||||
<p><pre>
|
||||
@ -505,29 +509,29 @@ as follows:
|
||||
<li> If the precedence of the token to be shifted is greater than
|
||||
the precedence of the rule to reduce, then resolve in favor
|
||||
of the shift. No parsing conflict is reported.
|
||||
<li> If the precedence of the token it be shifted is less than the
|
||||
<li> If the precedence of the token to be shifted is less than the
|
||||
precedence of the rule to reduce, then resolve in favor of the
|
||||
reduce action. No parsing conflict is reported.
|
||||
<li> If the precedences are the same and the shift token is
|
||||
right-associative, then resolve in favor of the shift.
|
||||
No parsing conflict is reported.
|
||||
<li> If the precedences are the same the shift token is
|
||||
<li> If the precedences are the same and the shift token is
|
||||
left-associative, then resolve in favor of the reduce.
|
||||
No parsing conflict is reported.
|
||||
<li> Otherwise, resolve the conflict by doing the shift and
|
||||
report the parsing conflict.
|
||||
<li> Otherwise, resolve the conflict by doing the shift, and
|
||||
report a parsing conflict.
|
||||
</ul>
|
||||
Reduce-reduce conflicts are resolved this way:
|
||||
<ul>
|
||||
<li> If either reduce rule
|
||||
<li> If either reduce rule
|
||||
lacks precedence information, then resolve in favor of the
|
||||
rule that appears first in the grammar and report a parsing
|
||||
rule that appears first in the grammar, and report a parsing
|
||||
conflict.
|
||||
<li> If both rules have precedence and the precedence is different
|
||||
<li> If both rules have precedence and the precedence is different,
|
||||
then resolve the dispute in favor of the rule with the highest
|
||||
precedence and do not report a conflict.
|
||||
precedence, and do not report a conflict.
|
||||
<li> Otherwise, resolve the conflict by reducing by the rule that
|
||||
appears first in the grammar and report a parsing conflict.
|
||||
appears first in the grammar, and report a parsing conflict.
|
||||
</ul>
|
||||
|
||||
<h3>Special Directives</h3>
|
||||
@ -536,40 +540,40 @@ Reduce-reduce conflicts are resolved this way:
|
||||
directives. We've described all the grammar rules, so now we'll
|
||||
talk about the special directives.</p>
|
||||
|
||||
<p>Directives in lemon can occur in any order. You can put them before
|
||||
the grammar rules, or after the grammar rules, or in the mist of the
|
||||
<p>Directives in Lemon can occur in any order. You can put them before
|
||||
the grammar rules, or after the grammar rules, or in the midst of the
|
||||
grammar rules. It doesn't matter. The relative order of
|
||||
directives used to assign precedence to terminals is important, but
|
||||
other than that, the order of directives in Lemon is arbitrary.</p>
|
||||
|
||||
<p>Lemon supports the following special directives:
|
||||
<ul>
|
||||
<li><tt>%code</tt>
|
||||
<li><tt>%default_destructor</tt>
|
||||
<li><tt>%default_type</tt>
|
||||
<li><tt>%destructor</tt>
|
||||
<li><tt>%endif</tt>
|
||||
<li><tt>%extra_argument</tt>
|
||||
<li><tt>%fallback</tt>
|
||||
<li><tt>%ifdef</tt>
|
||||
<li><tt>%ifndef</tt>
|
||||
<li><tt>%include</tt>
|
||||
<li><tt>%left</tt>
|
||||
<li><tt>%name</tt>
|
||||
<li><tt>%nonassoc</tt>
|
||||
<li><tt>%parse_accept</tt>
|
||||
<li><tt>%parse_failure </tt>
|
||||
<li><tt>%right</tt>
|
||||
<li><tt>%stack_overflow</tt>
|
||||
<li><tt>%stack_size</tt>
|
||||
<li><tt>%start_symbol</tt>
|
||||
<li><tt>%syntax_error</tt>
|
||||
<li><tt>%token_class</tt>
|
||||
<li><tt>%token_destructor</tt>
|
||||
<li><tt>%token_prefix</tt>
|
||||
<li><tt>%token_type</tt>
|
||||
<li><tt>%type</tt>
|
||||
<li><tt>%wildcard</tt>
|
||||
<li><tt><a href='#pcode'>%code</a></tt>
|
||||
<li><tt><a href='#default_destructor'>%default_destructor</a></tt>
|
||||
<li><tt><a href='#default_type'>%default_type</a></tt>
|
||||
<li><tt><a href='#destructor'>%destructor</a></tt>
|
||||
<li><tt><a href='#pifdef'>%endif</a></tt>
|
||||
<li><tt><a href='#extraarg'>%extra_argument</a></tt>
|
||||
<li><tt><a href='#pfallback'>%fallback</a></tt>
|
||||
<li><tt><a href='#pifdef'>%ifdef</a></tt>
|
||||
<li><tt><a href='#pifdef'>%ifndef</a></tt>
|
||||
<li><tt><a href='#pinclude'>%include</a></tt>
|
||||
<li><tt><a href='#pleft'>%left</a></tt>
|
||||
<li><tt><a href='#pname'>%name</a></tt>
|
||||
<li><tt><a href='#pnonassoc'>%nonassoc</a></tt>
|
||||
<li><tt><a href='#parse_accept'>%parse_accept</a></tt>
|
||||
<li><tt><a href='#parse_failure'>%parse_failure</a></tt>
|
||||
<li><tt><a href='#pright'>%right</a></tt>
|
||||
<li><tt><a href='#stack_overflow'>%stack_overflow</a></tt>
|
||||
<li><tt><a href='#stack_size'>%stack_size</a></tt>
|
||||
<li><tt><a href='#start_symbol'>%start_symbol</a></tt>
|
||||
<li><tt><a href='#syntax_error'>%syntax_error</a></tt>
|
||||
<li><tt><a href='#token_class'>%token_class</a></tt>
|
||||
<li><tt><a href='#token_destructor'>%token_destructor</a></tt>
|
||||
<li><tt><a href='#token_prefix'>%token_prefix</a></tt>
|
||||
<li><tt><a href='#token_type'>%token_type</a></tt>
|
||||
<li><tt><a href='#ptype'>%type</a></tt>
|
||||
<li><tt><a href='#pwildcard'>%wildcard</a></tt>
|
||||
</ul>
|
||||
Each of these directives will be described separately in the
|
||||
following sections:</p>
|
||||
@ -577,43 +581,42 @@ following sections:</p>
|
||||
<a name='pcode'></a>
|
||||
<h4>The <tt>%code</tt> directive</h4>
|
||||
|
||||
<p>The %code directive is used to specify addition C code that
|
||||
<p>The <tt>%code</tt> directive is used to specify additional C code that
|
||||
is added to the end of the main output file. This is similar to
|
||||
the <a href='#pinclude'>%include</a> directive except that %include
|
||||
is inserted at the beginning of the main output file.</p>
|
||||
the <tt><a href='#pinclude'>%include</a></tt> directive except that
|
||||
<tt>%include</tt> is inserted at the beginning of the main output file.</p>
|
||||
|
||||
<p>%code is typically used to include some action routines or perhaps
|
||||
a tokenizer or even the "main()" function
|
||||
<p><tt>%code</tt> is typically used to include some action routines or perhaps
|
||||
a tokenizer or even the "main()" function
|
||||
as part of the output file.</p>
|
||||
|
||||
<a name='default_destructor'></a>
|
||||
<h4>The <tt>%default_destructor</tt> directive</h4>
|
||||
|
||||
<p>The %default_destructor directive specifies a destructor to
|
||||
<p>The <tt>%default_destructor</tt> directive specifies a destructor to
|
||||
use for non-terminals that do not have their own destructor
|
||||
specified by a separate %destructor directive. See the documentation
|
||||
on the <a name='#destructor'>%destructor</a> directive below for
|
||||
specified by a separate <tt>%destructor</tt> directive. See the documentation
|
||||
on the <tt><a name='#destructor'>%destructor</a></tt> directive below for
|
||||
additional information.</p>
|
||||
|
||||
<p>In some grammers, many different non-terminal symbols have the
|
||||
same datatype and hence the same destructor. This directive is
|
||||
a convenience way to specify the same destructor for all those
|
||||
<p>In some grammars, many different non-terminal symbols have the
|
||||
same data type and hence the same destructor. This directive is
|
||||
a convenient way to specify the same destructor for all those
|
||||
non-terminals using a single statement.</p>
|
||||
|
||||
<a name='default_type'></a>
|
||||
<h4>The <tt>%default_type</tt> directive</h4>
|
||||
|
||||
<p>The %default_type directive specifies the datatype of non-terminal
|
||||
symbols that do no have their own datatype defined using a separate
|
||||
<a href='#ptype'>%type</a> directive.
|
||||
</p>
|
||||
<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal
|
||||
symbols that do not have their own data type defined using a separate
|
||||
<tt><a href='#ptype'>%type</a></tt> directive.</p>
|
||||
|
||||
<a name='destructor'></a>
|
||||
<h4>The <tt>%destructor</tt> directive</h4>
|
||||
|
||||
<p>The %destructor directive is used to specify a destructor for
|
||||
<p>The <tt>%destructor</tt> directive is used to specify a destructor for
|
||||
a non-terminal symbol.
|
||||
(See also the <a href='#token_destructor'>%token_destructor</a>
|
||||
(See also the <tt><a href='#token_destructor'>%token_destructor</a></tt>
|
||||
directive which is used to specify a destructor for terminal symbols.)</p>
|
||||
|
||||
<p>A non-terminal's destructor is called to dispose of the
|
||||
@ -635,7 +638,7 @@ or other resources held by that non-terminal.</p>
|
||||
%destructor nt { free($$); }
|
||||
nt(A) ::= ID NUM. { A = malloc( 100 ); }
|
||||
</pre>
|
||||
This example is a bit contrived but it serves to illustrate how
|
||||
This example is a bit contrived, but it serves to illustrate how
|
||||
destructors work. The example shows a non-terminal named
|
||||
"nt" that holds values of type "void*". When the rule for
|
||||
an "nt" reduces, it sets the value of the non-terminal to
|
||||
@ -651,17 +654,17 @@ stack, unless the non-terminal is used in a C-code action. If
|
||||
the non-terminal is used by C-code, then it is assumed that the
|
||||
C-code will take care of destroying it.
|
||||
More commonly, the value is used to build some
|
||||
larger structure and we don't want to destroy it, which is why
|
||||
larger structure, and we don't want to destroy it, which is why
|
||||
the destructor is not called in this circumstance.</p>
|
||||
|
||||
<p>Destructors help avoid memory leaks by automatically freeing
|
||||
allocated objects when they go out of scope.
|
||||
To do the same using yacc or bison is much more difficult.</p>
|
||||
|
||||
<a name="extraarg"></a>
|
||||
<a name='extraarg'></a>
|
||||
<h4>The <tt>%extra_argument</tt> directive</h4>
|
||||
|
||||
The %extra_argument directive instructs Lemon to add a 4th parameter
|
||||
The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
|
||||
to the parameter list of the Parse() function it generates. Lemon
|
||||
doesn't do anything itself with this extra argument, but it does
|
||||
make the argument available to C-code action routines, destructors,
|
||||
@ -679,61 +682,64 @@ in the most recent call to Parse().</p>
|
||||
<a name='pfallback'></a>
|
||||
<h4>The <tt>%fallback</tt> directive</h4>
|
||||
|
||||
<p>The %fallback directive specifies an alternative meaning for one
|
||||
<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one
|
||||
or more tokens. The alternative meaning is tried if the original token
|
||||
would have generated a syntax error.
|
||||
would have generated a syntax error.</p>
|
||||
|
||||
<p>The %fallback directive was added to support robust parsing of SQL
|
||||
syntax in <a href="https://www.sqlite.org/">SQLite</a>.
|
||||
<p>The <tt>%fallback</tt> directive was added to support robust parsing of SQL
|
||||
syntax in <a href='https://www.sqlite.org/'>SQLite</a>.
|
||||
The SQL language contains a large assortment of keywords, each of which
|
||||
appears as a different token to the language parser. SQL contains so
|
||||
many keywords, that it can be difficult for programmers to keep up with
|
||||
many keywords that it can be difficult for programmers to keep up with
|
||||
them all. Programmers will, therefore, sometimes mistakenly use an
|
||||
obscure language keyword for an identifier. The %fallback directive
|
||||
obscure language keyword for an identifier. The <tt>%fallback</tt> directive
|
||||
provides a mechanism to tell the parser: "If you are unable to parse
|
||||
this keyword, try treating it as an identifier instead."
|
||||
this keyword, try treating it as an identifier instead."</p>
|
||||
|
||||
<p>The syntax of %fallback is as follows:
|
||||
<p>The syntax of <tt>%fallback</tt> is as follows:
|
||||
|
||||
<blockquote>
|
||||
<tt>%fallback</tt> <i>ID</i> <i>TOKEN...</i> <b>.</b>
|
||||
</blockquote>
|
||||
<tt>%fallback</tt> <i>ID</i> <i>TOKEN...</i> <b>.</b>
|
||||
</blockquote></p>
|
||||
|
||||
<p>In words, the %fallback directive is followed by a list of token names
|
||||
terminated by a period. The first token name is the fallback token - the
|
||||
<p>In words, the <tt>%fallback</tt> directive is followed by a list of token
|
||||
names terminated by a period.
|
||||
The first token name is the fallback token — the
|
||||
token to which all the other tokens fall back to. The second and subsequent
|
||||
arguments are tokens which fall back to the token identified by the first
|
||||
argument.
|
||||
argument.</p>
|
||||
|
||||
<a name='pifdef'></a>
|
||||
<h4>The <tt>%ifdef</tt>, <tt>%ifndef</tt>, and <tt>%endif</tt> directives.</h4>
|
||||
<h4>The <tt>%ifdef</tt>, <tt>%ifndef</tt>, and <tt>%endif</tt> directives</h4>
|
||||
|
||||
<p>The %ifdef, %ifndef, and %endif directives are similar to
|
||||
#ifdef, #ifndef, and #endif in the C-preprocessor, just not as general.
|
||||
<p>The <tt>%ifdef</tt>, <tt>%ifndef</tt>, and <tt>%endif</tt> directives
|
||||
are similar to #ifdef, #ifndef, and #endif in the C-preprocessor,
|
||||
just not as general.
|
||||
Each of these directives must begin at the left margin. No whitespace
|
||||
is allowed between the "%" and the directive name.
|
||||
is allowed between the "%" and the directive name.</p>
|
||||
|
||||
<p>Grammar text in between "%ifdef MACRO" and the next nested "%endif" is
|
||||
<p>Grammar text in between "<tt>%ifdef MACRO</tt>" and the next nested
|
||||
"<tt>%endif</tt>" is
|
||||
ignored unless the "-DMACRO" command-line option is used. Grammar text
|
||||
betwen "%ifndef MACRO" and the next nested "%endif" is included except when
|
||||
the "-DMACRO" command-line option is used.
|
||||
betwen "<tt>%ifndef MACRO</tt>" and the next nested "<tt>%endif</tt>" is
|
||||
included except when the "-DMACRO" command-line option is used.</p>
|
||||
|
||||
<p>Note that the argument to %ifdef and %ifndef must be a single
|
||||
preprocessor symbol name, not a general expression. There is no "%else"
|
||||
directive.
|
||||
<p>Note that the argument to <tt>%ifdef</tt> and <tt>%ifndef</tt> must
|
||||
be a single preprocessor symbol name, not a general expression.
|
||||
There is no "<tt>%else</tt>" directive.</p>
|
||||
|
||||
|
||||
<a name='pinclude'></a>
|
||||
<h4>The <tt>%include</tt> directive</h4>
|
||||
|
||||
<p>The %include directive specifies C code that is included at the
|
||||
top of the generated parser. You can include any text you want --
|
||||
<p>The <tt>%include</tt> directive specifies C code that is included at the
|
||||
top of the generated parser. You can include any text you want —
|
||||
the Lemon parser generator copies it blindly. If you have multiple
|
||||
%include directives in your grammar file, their values are concatenated
|
||||
so that all %include code ultimately appears near the top of the
|
||||
generated parser, in the same order as it appeared in the grammer.</p>
|
||||
<tt>%include</tt> directives in your grammar file, their values are concatenated
|
||||
so that all <tt>%include</tt> code ultimately appears near the top of the
|
||||
generated parser, in the same order as it appeared in the grammar.</p>
|
||||
|
||||
<p>The %include directive is very handy for getting some extra #include
|
||||
<p>The <tt>%include</tt> directive is very handy for getting some extra #include
|
||||
preprocessor statements at the beginning of the generated parser.
|
||||
For example:</p>
|
||||
|
||||
@ -742,17 +748,19 @@ For example:</p>
|
||||
</pre></p>
|
||||
|
||||
<p>This might be needed, for example, if some of the C actions in the
|
||||
grammar call functions that are prototyed in unistd.h.</p>
|
||||
grammar call functions that are prototyped in unistd.h.</p>
|
||||
|
||||
<a name='pleft'></a>
|
||||
<h4>The <tt>%left</tt> directive</h4>
|
||||
|
||||
The %left directive is used (along with the <a href='#pright'>%right</a> and
|
||||
<a href='#pnonassoc'>%nonassoc</a> directives) to declare precedences of
|
||||
terminal symbols. Every terminal symbol whose name appears after
|
||||
a %left directive but before the next period (".") is
|
||||
The <tt>%left</tt> directive is used (along with the
|
||||
<tt><a href='#pright'>%right</a></tt> and
|
||||
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives) to declare
|
||||
precedences of terminal symbols.
|
||||
Every terminal symbol whose name appears after
|
||||
a <tt>%left</tt> directive but before the next period (".") is
|
||||
given the same left-associative precedence value. Subsequent
|
||||
%left directives have higher precedence. For example:</p>
|
||||
<tt>%left</tt> directives have higher precedence. For example:</p>
|
||||
|
||||
<p><pre>
|
||||
%left AND.
|
||||
@ -763,20 +771,21 @@ given the same left-associative precedence value. Subsequent
|
||||
%right EXP NOT.
|
||||
</pre></p>
|
||||
|
||||
<p>Note the period that terminates each %left, %right or %nonassoc
|
||||
<p>Note the period that terminates each <tt>%left</tt>,
|
||||
<tt>%right</tt> or <tt>%nonassoc</tt>
|
||||
directive.</p>
|
||||
|
||||
<p>LALR(1) grammars can get into a situation where they require
|
||||
a large amount of stack space if you make heavy use or right-associative
|
||||
operators. For this reason, it is recommended that you use %left
|
||||
rather than %right whenever possible.</p>
|
||||
operators. For this reason, it is recommended that you use <tt>%left</tt>
|
||||
rather than <tt>%right</tt> whenever possible.</p>
|
||||
|
||||
<a name='pname'></a>
|
||||
<h4>The <tt>%name</tt> directive</h4>
|
||||
|
||||
<p>By default, the functions generated by Lemon all begin with the
|
||||
five-character string "Parse". You can change this string to something
|
||||
different using the %name directive. For instance:</p>
|
||||
different using the <tt>%name</tt> directive. For instance:</p>
|
||||
|
||||
<p><pre>
|
||||
%name Abcde
|
||||
@ -790,22 +799,22 @@ functions named
|
||||
<li> AbcdeTrace(), and
|
||||
<li> Abcde().
|
||||
</ul>
|
||||
The %name directive allows you to generator two or more different
|
||||
parsers and link them all into the same executable.
|
||||
</p>
|
||||
The <tt>%name</tt> directive allows you to generate two or more different
|
||||
parsers and link them all into the same executable.</p>
|
||||
|
||||
<a name='pnonassoc'></a>
|
||||
<h4>The <tt>%nonassoc</tt> directive</h4>
|
||||
|
||||
<p>This directive is used to assign non-associative precedence to
|
||||
one or more terminal symbols. See the section on
|
||||
one or more terminal symbols. See the section on
|
||||
<a href='#precrules'>precedence rules</a>
|
||||
or on the <a href='#pleft'>%left</a> directive for additional information.</p>
|
||||
or on the <tt><a href='#pleft'>%left</a></tt> directive
|
||||
for additional information.</p>
|
||||
|
||||
<a name='parse_accept'></a>
|
||||
<h4>The <tt>%parse_accept</tt> directive</h4>
|
||||
|
||||
<p>The %parse_accept directive specifies a block of C code that is
|
||||
<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is
|
||||
executed whenever the parser accepts its input string. To "accept"
|
||||
an input string means that the parser was able to process all tokens
|
||||
without error.</p>
|
||||
@ -821,7 +830,7 @@ without error.</p>
|
||||
<a name='parse_failure'></a>
|
||||
<h4>The <tt>%parse_failure</tt> directive</h4>
|
||||
|
||||
<p>The %parse_failure directive specifies a block of C code that
|
||||
<p>The <tt>%parse_failure</tt> directive specifies a block of C code that
|
||||
is executed whenever the parser fails complete. This code is not
|
||||
executed until the parser has tried and failed to resolve an input
|
||||
error using is usual error recovery strategy. The routine is
|
||||
@ -837,14 +846,14 @@ only invoked when parsing is unable to continue.</p>
|
||||
<h4>The <tt>%right</tt> directive</h4>
|
||||
|
||||
<p>This directive is used to assign right-associative precedence to
|
||||
one or more terminal symbols. See the section on
|
||||
one or more terminal symbols. See the section on
|
||||
<a href='#precrules'>precedence rules</a>
|
||||
or on the <a href='#pleft'>%left</a> directive for additional information.</p>
|
||||
|
||||
<a name='stack_overflow'></a>
|
||||
<h4>The <tt>%stack_overflow</tt> directive</h4>
|
||||
|
||||
<p>The %stack_overflow directive specifies a block of C code that
|
||||
<p>The <tt>%stack_overflow</tt> directive specifies a block of C code that
|
||||
is executed if the parser's internal stack ever overflows. Typically
|
||||
this just prints an error message. After a stack overflow, the parser
|
||||
will be unable to continue and must be reset.</p>
|
||||
@ -857,7 +866,7 @@ will be unable to continue and must be reset.</p>
|
||||
|
||||
<p>You can help prevent parser stack overflows by avoiding the use
|
||||
of right recursion and right-precedence operators in your grammar.
|
||||
Use left recursion and and left-precedence operators instead, to
|
||||
Use left recursion and and left-precedence operators instead to
|
||||
encourage rules to reduce sooner and keep the stack size down.
|
||||
For example, do rules like this:
|
||||
<pre>
|
||||
@ -868,7 +877,7 @@ Not like this:
|
||||
<pre>
|
||||
list ::= element list. // right-recursion. Bad!
|
||||
list ::= .
|
||||
</pre>
|
||||
</pre></p>
|
||||
|
||||
<a name='stack_size'></a>
|
||||
<h4>The <tt>%stack_size</tt> directive</h4>
|
||||
@ -876,7 +885,7 @@ Not like this:
|
||||
<p>If stack overflow is a problem and you can't resolve the trouble
|
||||
by using left-recursion, then you might want to increase the size
|
||||
of the parser's stack using this directive. Put an positive integer
|
||||
after the %stack_size directive and Lemon will generate a parse
|
||||
after the <tt>%stack_size</tt> directive and Lemon will generate a parse
|
||||
with a stack of the requested size. The default value is 100.</p>
|
||||
|
||||
<p><pre>
|
||||
@ -886,25 +895,40 @@ with a stack of the requested size. The default value is 100.</p>
|
||||
<a name='start_symbol'></a>
|
||||
<h4>The <tt>%start_symbol</tt> directive</h4>
|
||||
|
||||
<p>By default, the start-symbol for the grammar that Lemon generates
|
||||
<p>By default, the start symbol for the grammar that Lemon generates
|
||||
is the first non-terminal that appears in the grammar file. But you
|
||||
can choose a different start-symbol using the %start_symbol directive.</p>
|
||||
can choose a different start symbol using the
|
||||
<tt>%start_symbol</tt> directive.</p>
|
||||
|
||||
<p><pre>
|
||||
%start_symbol prog
|
||||
</pre></p>
|
||||
|
||||
<a name='syntax_error'></a>
|
||||
<h4>The <tt>%syntax_error</tt> directive</h4>
|
||||
|
||||
<p>See <a href='#error_processing'>Error Processing</a>.</p>
|
||||
|
||||
<a name='token_class'></a>
|
||||
<h4>The <tt>%token_class</tt> directive</h4>
|
||||
|
||||
<p>Undocumented. Appears to be related to the MULTITERMINAL concept.
|
||||
<a href='http://sqlite.org/src/fdiff?v1=796930d5fc2036c7&v2=624b24c5dc048e09&sbs=0'>Implementation</a>.</p>
|
||||
|
||||
<a name='token_destructor'></a>
|
||||
<h4>The <tt>%token_destructor</tt> directive</h4>
|
||||
|
||||
<p>The %destructor directive assigns a destructor to a non-terminal
|
||||
symbol. (See the description of the %destructor directive above.)
|
||||
This directive does the same thing for all terminal symbols.</p>
|
||||
<p>The <tt>%destructor</tt> directive assigns a destructor to a non-terminal
|
||||
symbol. (See the description of the
|
||||
<tt><a href='%destructor'>%destructor</a></tt> directive above.)
|
||||
The <tt>%token_destructor</tt> directive does the same thing
|
||||
for all terminal symbols.</p>
|
||||
|
||||
<p>Unlike non-terminal symbols which may each have a different data type
|
||||
for their values, terminals all use the same data type (defined by
|
||||
the %token_type directive) and so they use a common destructor. Other
|
||||
than that, the token destructor works just like the non-terminal
|
||||
the <tt><a href='#token_type'>%token_type</a></tt> directive)
|
||||
and so they use a common destructor.
|
||||
Other than that, the token destructor works just like the non-terminal
|
||||
destructors.</p>
|
||||
|
||||
<a name='token_prefix'></a>
|
||||
@ -913,8 +937,9 @@ destructors.</p>
|
||||
<p>Lemon generates #defines that assign small integer constants
|
||||
to each terminal symbol in the grammar. If desired, Lemon will
|
||||
add a prefix specified by this directive
|
||||
to each of the #defines it generates.
|
||||
So if the default output of Lemon looked like this:
|
||||
to each of the #defines it generates.</p>
|
||||
|
||||
<p>So if the default output of Lemon looked like this:
|
||||
<pre>
|
||||
#define AND 1
|
||||
#define MINUS 2
|
||||
@ -931,7 +956,7 @@ to cause Lemon to produce these symbols instead:
|
||||
#define TOKEN_MINUS 2
|
||||
#define TOKEN_OR 3
|
||||
#define TOKEN_PLUS 4
|
||||
</pre>
|
||||
</pre></p>
|
||||
|
||||
<a name='token_type'></a><a name='ptype'></a>
|
||||
<h4>The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
|
||||
@ -952,7 +977,7 @@ token structure. Like this:</p>
|
||||
is "void*".</p>
|
||||
|
||||
<p>Non-terminal symbols can each have their own data types. Typically
|
||||
the data type of a non-terminal is a pointer to the root of a parse-tree
|
||||
the data type of a non-terminal is a pointer to the root of a parse tree
|
||||
structure that contains all information about that non-terminal.
|
||||
For example:</p>
|
||||
|
||||
@ -973,14 +998,15 @@ and able to pay that price, fine. You just need to know.</p>
|
||||
<a name='pwildcard'></a>
|
||||
<h4>The <tt>%wildcard</tt> directive</h4>
|
||||
|
||||
<p>The %wildcard directive is followed by a single token name and a
|
||||
period. This directive specifies that the identified token should
|
||||
match any input token.
|
||||
<p>The <tt>%wildcard</tt> directive is followed by a single token name and a
|
||||
period. This directive specifies that the identified token should
|
||||
match any input token.</p>
|
||||
|
||||
<p>When the generated parser has the choice of matching an input against
|
||||
the wildcard token and some other token, the other token is always used.
|
||||
The wildcard token is only matched if there are no other alternatives.
|
||||
The wildcard token is only matched if there are no alternatives.</p>
|
||||
|
||||
<a name='error_processing'></a>
|
||||
<h3>Error Processing</h3>
|
||||
|
||||
<p>After extensive experimentation over several years, it has been
|
||||
@ -988,19 +1014,20 @@ discovered that the error recovery strategy used by yacc is about
|
||||
as good as it gets. And so that is what Lemon uses.</p>
|
||||
|
||||
<p>When a Lemon-generated parser encounters a syntax error, it
|
||||
first invokes the code specified by the %syntax_error directive, if
|
||||
first invokes the code specified by the <tt>%syntax_error</tt> directive, if
|
||||
any. It then enters its error recovery strategy. The error recovery
|
||||
strategy is to begin popping the parsers stack until it enters a
|
||||
state where it is permitted to shift a special non-terminal symbol
|
||||
named "error". It then shifts this non-terminal and continues
|
||||
parsing. But the %syntax_error routine will not be called again
|
||||
parsing. The <tt>%syntax_error</tt> routine will not be called again
|
||||
until at least three new tokens have been successfully shifted.</p>
|
||||
|
||||
<p>If the parser pops its stack until the stack is empty, and it still
|
||||
is unable to shift the error symbol, then the %parse_failed routine
|
||||
is unable to shift the error symbol, then the
|
||||
<tt><a href='#parse_failure'>%parse_failure</a></tt> routine
|
||||
is invoked and the parser resets itself to its start state, ready
|
||||
to begin parsing a new file. This is what will happen at the very
|
||||
first syntax error, of course, if there are no instances of the
|
||||
first syntax error, of course, if there are no instances of the
|
||||
"error" non-terminal in your grammar.</p>
|
||||
|
||||
</body>
|
||||
|
13
manifest
13
manifest
@ -1,5 +1,5 @@
|
||||
C Add\sthe\ssqlite3_mmap_warm()\sfunction\sas\san\sextension\sin\sthe\sext/misc/mmapwarm.c\ssource\sfile.
|
||||
D 2017-09-18T18:17:01.889
|
||||
C Updates\sto\sthe\s"lemon.html"\sdocument\sreceived\sfrom\sAndy\sGoth.
|
||||
D 2017-09-20T09:09:34.192
|
||||
F Makefile.in 4bc36d913c2e3e2d326d588d72f618ac9788b2fd4b7efda61102611a6495c3ff
|
||||
F Makefile.linux-gcc 7bc79876b875010e8c8f9502eb935ca92aa3c434
|
||||
F Makefile.msc 6033b51b6aea702ea059f6ab2d47b1d3cef648695f787247dd4fb395fe60673f
|
||||
@ -33,7 +33,7 @@ F config.sub 9ebe4c3b3dab6431ece34f16828b594fb420da55
|
||||
F configure e691ad9b505f1f47bc5d99be9e1d49b1be9037e9cb3821c9b14c63c3d413d055 x
|
||||
F configure.ac bb85c1c53e952c8c7078a2f147eba613e0128b8b6e7780d64758d8fb29bcc695
|
||||
F contrib/sqlitecon.tcl 210a913ad63f9f991070821e599d600bd913e0ad
|
||||
F doc/lemon.html 1f8b8d4c9f5cfe40e679fee279cc9eb2da8e6eb74ad406028538d7864cc4b6cb
|
||||
F doc/lemon.html 278113807f49d12d04179a93fab92b5b917a08771152ca7949d34e928efa3941
|
||||
F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
|
||||
F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a
|
||||
F ext/README.md fd5f78013b0a2bc6f0067afb19e6ad040e89a10179b4f6f03eee58fac5f169bd
|
||||
@ -1655,8 +1655,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
|
||||
F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
|
||||
F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
|
||||
F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
|
||||
P a944719314e0ac2f1954b65668815769eba3ab3e39a74666293b8dea52a184b2 3235835babb49b4dd1acaabd1aa6cfb0b7fe19a914db1cb511e8cc872d3c0c39
|
||||
R 62da41337307696798754c70fb4a4da8
|
||||
T +closed 3235835babb49b4dd1acaabd1aa6cfb0b7fe19a914db1cb511e8cc872d3c0c39
|
||||
P 1b2de41453ac33de82f9cd6cbb92eee4fe184fb282c27e5efa5243c8cb239630
|
||||
R e0e8cf7279386534b02109d8fa18ad99
|
||||
U drh
|
||||
Z 9c4b90490d8e7ae619a7569445f78dbf
|
||||
Z 86920861ac015f347841db4c53c64a7b
|
||||
|
@ -1 +1 @@
|
||||
1b2de41453ac33de82f9cd6cbb92eee4fe184fb282c27e5efa5243c8cb239630
|
||||
5b2002f3df1902aaa571a0efd01ab8bae7f4d37ac4819cc51595277f4de93433
|
Loading…
Reference in New Issue
Block a user