This is an edge case that doesn't occur in practice since pretty much
nobody dares to use variable names that contain an actual '$' in their
name. This is not about the fairly common VAR.${param} (as written in
the makefile), but instead about the variable whose name is literally
'VAR.${param}'.
The test demonstrates that after the fix, the variable name is taken
exactly as-is for the simple assignment modifier '::='. There are no
such tests for the modifiers '::+=', '::!=' and '::?=', but that's ok.
The code in ApplyModifier_Assign would look assymetrical and suspicious
enough if one of these modifiers would expand its variable name and the
others wouldn't.
In CLEANUP mode, was originally meant to track memory allocations but is
useful during debugging as well, initialize the list. There is no
distinct constant representing an invalid pointer, otherwise that would
have been an even better choice.
An enum with 32 bits would lead to signed integer overflow anyway, so
that definition is not worth keeping even if it works on typical
2-complement platforms.
The definitions for 2, 4 and 8 enum have been unused for several months
now.
No functional change.
This makes the code easier to read, especially in var.c. It also makes
debugging sessions easier since some debuggers don't show enum
bit-fields symbolically as soon as more than one bit is set.
The code outside var.c is basically unchanged, except that instead of
passing the individual flags, there are 4 predefined evaluation modes.
These suffice for all practical use cases. Only in the implementation
deep inside var.c, the value of the flags keepDollar and keepUndef
differs.
There is no way of passing the struct to EnumFlags_ToString, which means
the ToString function has to be spelled out explicitly. This allows for
fine-tuning the representation in the debug log, to reduce the amount of
uppercae letters.
No functional change.
The name 'NONE' described the bit pattern, which was not useful to
understand its meaning. Omitting VARE_WANTRES only parses the
expression, without evaluating any part of it.
No functional change, not even in debug mode since Enum_FlagsToString
always returns "none" for all-bits-unset.
This is just to keep the code consistent among the various variable
modifiers. The performance gain is negligible.
The actual assignment to the variable had already been skipped
previously.
No functional change.
No functional change in practical usage. Theoretically this change can
be observed by looking at the generated random numbers for the ':Ox'
modifier, but the quality or exact sequence of these random numbers is
not guaranteed anyway.
In parse-only mode, variable expressions in the argument to that
modifier are not resolved. This led to the error message about the 'Bad
modifier' in var-eval-short.mk.
This affects the modifiers ':E', ':H', ':P', ':Q', ':R', ':T', ':hash',
':q', ':range', ':tl', ':ts', ':tu', and ':u'. All these modifiers are
side-effect free.
Skipping the evaluation for these modifiers is purely for code
consistency and performance.
No functional change.
No functional change, just a tiny bit of performance improvement,
probably not even measurable. Having the code nevertheless serves as a
copy-and-paste template for implementing other modifiers that might
perform more costly tasks.
The test 'var-eval-short' had produced the output 'unexpected' before,
on stderr. It had been generated by '${:Uword:@${FAIL}@expr@}' by
combining the following obscure "features" of make:
1. the ':@' modifier loops over the words of the variable. This
modifier is not really obscure, it still takes some time to get used
to it.
2. the ':@' modifier allows a '$' sign in the variable name, which is
useless in practice.
3. the ':@' modifier creates a temporary loop variable in the global
namespace. Luckily there are only few collisions with other
variable names since their naming conventions differ.
4. after looping over the words of the expression, the temporary global
loop variable is deleted, and at that point the '$' is expanded,
being interpreted as the start of a variable expression.
5. The ':@' modifier deleted the global variable even when it was
called in parse-only mode (without VARE_WANTRES).
When the modifier ':@' was initially added to make in var.c 1.40 from
2000-04-29, Var_Delete didn't expand the variable name. That feature
was added in var.c 1.174 from 2013-05-18, probably without thinking of
this very edge-casey combination of features.
This commit fixes item 5 from the above list. The other obscurities
remain for now.
This way, parsing and evaluating the modifier is only written once in
the code. The downside is that the variable name is allocated even if
VARE_WANTRES is not set, but since this modifier is so obscure and
seldom used this doesn't matter in practice.
This edge case had been so obscure that even discovering this takes
quite some time and requires reading the source code of make.
The manual page doesn't document whether the variable name is expanded
or not, it doesn't even give an example. When this obscure modifier was
initially added in var.c 1.210 from 2017-01-30, Var_Set always expanded
the variable name once, and there was no way around it. Therefore this
expansion has probably been unintentional.
See var-eval-short.mk:46 for the test demonstrating this change.
Previously, the expression ${:Uword:_=VAR} was evaluated including all
its side effects even though it was in an irrelevant branch of the
condition.
No functional change since the only caller of TryParseIntBase0 already
handles all possible parse errors. Without this check, the code just
looked wrong though.
TryParseIntBase0 wrongly returns successful for a string that does not
start with a number at all. Its only caller, ApplyModifier_Words,
already handles all error cases properly.
No functional change.
This aligns the implementation of these modifiers with the requirements
in the long comment starting with 'The ApplyModifier functions'.
No functional change.
Right now, when a variable expression cannot be parsed, the result of
calling Var_Subst is a string containing garbage, and no error is
reported. In addition, there are some silent errors that are not
reported at all. This combination makes it difficult to change the
error handling without introducing subtle breakage in some edge cases.
An example for garbage output is in varmod-subst-regex.mk, in target
mod-regex-compile-error.
No functional change.
The modifier ':C' now only compiles the regular expression if the result
of the expression is actually needed.
Several other modifiers have the same bug of evaluating the expression
in cases where this is not needed. It just doesn't show up because they
don't have any noticeable side effects, other than wasting CPU time.
This affects irrelevant conditions as well.
The additional parameter last_bl_ptr was only necessary because the last
blank was stored as a pointer into the buffer. By storing the index in
the buffer instead, it doesn't need to be updated all the time.
No functional change.
The '?:' operator computing the factor was too hard to read. When
quickly scanning the code, the 1 in the expression looked too much like
it would be added to the indentation, which would turn the indentation
length into a column number, and that again would smell like an
off-by-one error.
No functional change.
The manual page says that the default maximum length of a comment line
is 78. The test 'comments.0' wrongly assumed that this 78 would refer
to the maximum _column_ allowed, which is off by one.
Fix the wording in the test 'comments.0' and remove the (now satisfied)
expectation comments in the test 'token-comment.0'.
Several other tests just happened to hit that limit, fix these as well.
Formatting indent.h required the following manual corrections
afterwards:
The first tab in the comment in line 1 was replaced with a space but
shouldn't be.
The spacing around the '...' in function prototypes was completely
wrong. It looked like 'const char *,...)__printflike', without any
spaces.
The '*' of the return type 'const char *' was tied to the function name,
even though this declaration was only for a single function. In such a
case, it's more appropriate to line up the function names.
The function-like macros were not indented to -di. This is something
that I would not expect from indent, so it's ok to do that manually.
column == 1 + indentation.
In addition, indentation is a relative distance while column is an
absolute position. Therefore, don't confuse these two concepts, to
prevent off-by-one errors.
No functional change.
Previously, the '/*' in the string literal had been interpreted as the
beginning of a comment, which was wrong. Because of that, the variable
declaration in the following line was still interpreted as part of the
comment. The comment even continued until the end of the file.
Due to indent's forgiving nature, it neither complained nor even
mentioned that anything had gone wrong. The decision of rather
producing wrong output than failing early is a dangerous one.
At least, there should have been an error message that at the end of the
file, the parser was still in a a comment, expecting the closing '*/'.
Since several years (maybe even decades) compilers know how to inline
static functions that are only used once. Therefore there is no need to
have overly long functions anymore, especially not 'main', which is only
called a single time and thus does not add any noticeable performance
degradation.
No functional change.
The word 'col' should only be used for the 1-based column number. This
name is completely inappropriate for a line length since that provokes
off-by-one errors. The name 'cols' would be acceptable although
confusing since it sounds so similar to 'col'.
Therefore, rename variables that are related to the maximum line length
to 'line_length' since that makes for obvious code and nicely relates to
the description of the option in the manual page.
No functional change.
These two functions operated on column numbers instead of indentation,
which required adjustments of '+ 1' and '- 1'. Their names were
completely wrong since these functions did not count anything, instead
they computed the column.
No functional change.
The goal is to only ever be concerned about the _indentation_ of a
token, never the _column_ it appears in. Having only one of these
avoids off-by-one errors.
No functional change.
Using the invariant 'column == 1 + indent'. This removes several overly
complicated '+ 1' from the code that are not needed conceptually.
No functional change.
Together with the results of the tokenizer and the 4 buffers for token,
label, code and comment, the debug log now provides a good high-level
view on how the indentation happens and where to look for the many
remaining bugs.
Whenever the code to be output contained the magic byte 0x80, instead of
writing this byte, indent wrote the column number at the beginning of
the code snippet, times 7. Especially the 'times 7' does not make any
sense at all.
In ISO-8859-1, this character position is not assigned. In Microsoft
Codepage 1252 it is the Euro sign. In UTF-8 (which was probably not on
the author's list when the code was originally written) it occurs as the
middle byte for code points like U+2026 (horizontal ellipsis) from the
block General Punctuation.
Remove this strange code, thereby fixing indent for UTF-8 code. The
code had been there since at least 1993-04-09, when it was first
imported to NetBSD.
Calculating the indentation is simpler than calculating the column,
since that saves the constant addition and subtraction of the 1.
No functional change.
Since C90, there is no need to repeat the type of the function
parameters.
In the whole code of indent, there is a lot of confusion between the
concepts of a 'column' (which is a position on the screen, counting
starts at 1) and 'indentation' (which is a length, not a position). To
avoid this confusion, the code will be rewritten anyway very soon.
Repeatedly adding and subtracting 1 from the 'current column' is not
elegant, this should rather be done by consistently measuring only the
indentation from the left border (at offset 0), as a distance, not as an
absolute position.
Column counting starts at 1. This 1 should rather be at the beginning
of the formula since it is thought of being added at the very beginning
of the line, not at the end.
When adding a tab, the newly added tab is added at the end of the
string, therefore that '+ 1' should be at the end of the formula as
well.
No functional change.
This allows to add debug logging to these few functions instead of all
other places that might output something.
Reducing the possible output formats to a few primitives makes dump_line
simpler, especially the fprintf calls. It also removes the non-constant
printf string.
The call to output_int may be meant for debugging, as the character 0x80
is unlikely to appear in any real-world code.
No functional change.
Having it directly below the table makes it easier understandable.
I also tried to omit this function entirely by moving the code into the
initializer itself, but that made the code redundant and furthermore
increased the size of the resulting binary, probably because of the new
relocation records.
No functional change.
This check has been too quick and broke the lint build. Among others,
lib/libpuffs has -w included in LINTFLAGS, which means that the build
can fail even for new warnings, not only for errors.
libpuffs compares a uint16_t with constants from an unnamed enum type.
Since the enum type is completely unnamed (neither a tag nor a typedef),
there is no way to define a struct member having this type. This was a
scenario that I just didn't consider when I added the check to lint.
For now, disable the new check completely. The previously existing lint
checks stay enabled, including the one that warns about mismatched
anonymous enum types in the '==' operator, which is very similar to the
now disabled check.
The previous 'casestmt' was wrong since a case label is not a statement
at all.
The previous 'swstmt' was overly short, and wrong as well, since it
represents only the 'switch (expr)' part, which is not a complete switch
statement. Same for 'ifstmt', 'whilestmt', 'forstmt'.
The previous word 'head' was not precise enough since it didn't specify
exactly where the head ends and the body starts. Especially for
handling the dangling else, this distinction is important.
No functional change.
This refactoring reduces the indentation of the code, as well as
removing any ambiguity as to which 'switch' statement a 'break' belongs,
as there are no more nested 'switch' statements.
No functional change.
It's strange that indent's own code is not formatted by indent itself,
which would be a good demonstration of its capabilities.
In its current state, I don't trust indent to get even the tokenization
correct, therefore the only safe way is to format the code manually.
It may have been a clever trick to use the same memory layout for struct
templ and a string pointer, but it's not worth the extra comment and
difficulty in understanding the code.
No functional change.
No functional change since neither rw_jump nor rw_inline_or_restrict is
mentioned in any switch statement, and lint didn't find any other
suspicious enum operations.
This reduces the magic numbers in the code. Most of these had their
designated constant name written in a nearby comment anyway.
The one instance where arithmetic was performed on this new enum type
(in indent.c) was a bit tricky to understand.
The combination rw_continue_or_inline_or_restrict looks strange, the
'continue' should intuitively belong to the other control flow keywords
in rw_break_or_goto_or_return.
No functional change.
It's a funny idea to do something like 'case a = 13:', but since any
compiler will reject this code, there is no point in lint supporting it.
No functional change since everywhere the grammar allows a constant
expression, there is no ambiguity where an assignment could be
interpreted differently.
C99 6.4.4 already defines a grammar rule named 'constant' for an number
literal or an enum constant, so don't use that name for something else.
No functional change.
According to the GCC documentation[1], the high end of the range is
inclusive as well, which makes sense since otherwise there would be no
way of specifying a range that includes the maximum representable
number.
Since the range is not used at all in the code, none of the tests could
possibly fail.
[1] https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html
No functional change.
There was no point in having a separate grammar rule for the '3...' part
of a range expression, it just made the code more complicated than
necessary.
No functional change.
before: array of unsigned int[4]
now: array[4] of unsigned int
Listing the array dimension first keeps it in contact with the keyword
'array'. This reduces confusion, especially for nested arrays.
The main ingredient for understanding how indent works is the tokenizer
and the 4 buffers in which the text is collected.
Inspecting this debug log for the test comment-line-end makes it obvious
why indent messes up code that contains '//' comments. The cause is
that indent interprets '//' as an operator, just like '&&' or '||'. The
sequence '/////' is interpreted as a single operator as well, by the
way.
Since '//' is interpreted as an ordinary operator, any words following
it are plain identifiers, usually several of them in a row, which is a
syntax error. Depending on the context, the operator '//' is either a
unary operator (no space around) or a binary operator (space around).
This explains why the word 'line-end' is expanded to 'line - end'.
No functional change outside of debug mode.
This is a prerequisite for converting the token types to an enum instead
of a preprocessor define, since the return type of lexi will become
token_type. Having the enum will make debugging easier.
There was a single naming collision, which forced the variable in
scan_profile to be renamed. All other token names are used nowhere
else.
No change to the resulting binary.
This is something that neither GCC 10 nor Clang 8 do, even though it
seems useful. Lint didn't do it up to now, but that was probably an
oversight since it is easy to miss the implicit '==' operator in the
switch statement.
assuming that everything that isn't a list is a tailq. Fixes random
reads from kmem that either fail or return incorrect data for the vcache
hash table.
Previously, the test msg_056.c warned twice about the integer literal,
but only on 32-bit platforms. On 64-bit platforms, there was only a
single warning since the integer constant was converted to type
__uint128_t, and this prevented the second warning. On 32-bit targets,
there is no __uint128_t though.
Fixes part of PR bin/55976.
The broad type of a value is indeed stored in the value itself, in the
member v_tspec. For nodes that refer to this value, it is redundantly
stored, it always equals tn->tn_type->t_tspec.
After initialization, neither tn->tn_type nor val->v_tspec are modified.
This is not ensured by the compiler but has to be analyzed manually.
No functional change.
Having the measurement unit in the variable name prevents accidental
confusion between bits and bytes, especially since usually the word
'size' means the size in bytes, while 'width' means the size in bits, at
least for integer types.
No functional change.
Each of the tests named msg_*.c repeats the template of the message, to
make the test somewhat self-contained when viewed in isolation.
This creates a redundancy, and keeping track of this manually is next to
impossible. I tried it and failed in 9 cases, even though it has just
been 2 months since I myself created the initial files and I knew all
the time that this redundancy exists.
Be fool-proof for the future by checking this automatically.
These expressions are indeed constant for a specific platform, but on
another platform their value may change. This makes them unsuspicious
and legitimate for portable code.
Seen in rump_syscalls.c, as 'sizeof(int) > sizeof(register_t)'.
Previously, 'typedef enum { E } name' was output as 'name', which
omitted the information that this was an enum type. Now it is output as
'enum typedef name'.
Previously, 'typedef struct { int member; } name' was output as 'struct
<unnamed>', which omitted the typedef name. Now it is output as 'struct
typedef name'.
Message 153 didn't state obviously which of the pointer types was the
one before conversion (or cast) and which was the resulting type.
Message 229 didn't have any type information at all.
This warning occurs more than 7400 times in a regular NetBSD build, and
without giving any type information, leaves the reader clueless about
what the underlying issue might be. Add type information since that is
a no-brainer to implement.
For performance reasons, the implementation of the simple rule "cmdline
overrides global" grew into code that is much more complicated than a
straight-forward implementation. This added complexity made it easy for
bugs to sneak in.
The extra condition had been necessary before FStr made memory
management simpler.
The Coverity annotation got out-of-date when the parameter was converted
to FStr since that type is not allocated on the heap, only its inner
members are.
No functional change.
Breaking the loop once for depth == 0 and once for depth == 1 was
unnecessarily confusing, as was the nested 'if'. Start counting with 0
since there is no reason to start at 1.
Evaluating the common subexpression '*p == endc' is left as an exercise
to the compiler.
No functional change.
In the expression ${:U}, the variable name is empty. Since these
expressions are generated by .for loops, the error messages for them
must not end with a trailing space. Putting the variable name in quotes
helps against that.
Replace "variable specification" with the more modern "variable
expression", reduce the number of parentheses, output more than a single
character for modifiers, make it obvious that in expressions such as
${:Serror}, the "" means a variable name.
Back in 1995, the modifiers were all single-character, and it made sense
to print only the first character. Nowadays, with ':S', ':@var@...@',
'::=' and several others, a little more context is useful to see where
the exact error is. The actual modifier is still guessed, and the guess
may be wrong as soon as backslashes get involved, but it is still better
than before.
For a very long time now, I had thought that it would be impossible to
undefine global variables during the evaluation of variable expressions.
This is something that the memory management in Var_Parse relies upon,
see the comment 'the value of the variable must not change'.
After several unsuccessful attempts at referring to an already freed
previous value of a variable, today I discovered how to unset a global
variable while evaluating an expression, which has the same effect. To
demonstrate that this use-after-free can reliably crash make, it would
need a memory allocator with a debug mode that never re-allocates the
same memory block after it has been used once. This is something that
jemalloc cannot do at the moment. Valgrind would be another idea, but
that has not been ported to NetBSD.
Undefining a global variable while evaluating an expression is made
possible by an implementation detail of the modifier ':@'. That
modifier undefines the loop variable, without restoring its previous
value, see ApplyModifier_Loop.
By the very old conventions of ODE Make, these loop variables are named
'.V.' and thus do not conflict with variables from other naming
conventions. In NetBSD and pkgsrc, these loop variables are typically
called 'var', sometimes '_var' with a leading underscore, which also
doesn't conflict with the typical form 'VAR' of variables in the global
namespace. Therefore, in practice these loop variables don't interfere
with other variables.
One case that can practically arise is when an outer variable has a
modifier ':@word@${VAR.${word}}@' and one of the referenced variables
uses the same variable name in the modifier, see varmod-loop.mk 1.10
line 91 for a detailed explanation.
By using the ${:@VAR@@} modifier in a place that is evaluated with
cmdline scope, it is not only possible to undefine global variables, it
is possible to undefine cmdline variables as well. When evaluated in a
specific make target, the expression ${:@\@@@} can even be used to
undefine the variable '.TARGET', which will probably crash make with an
assertion failure.
The variable name 'arg' was misleading since after a successful
TryParseTime, it would no longer point to the argument of the variable
modifier, but to the _end_ of the argument. To reduce confusion, use p
instead, like everywhere else. This name is less specific, which is
still better than a wrong name.
Addition is easier than subtraction, and the expression 'word + wordLen'
obviously means 'the end of the word', which was not as easy to spot
before.
No functional change.
These variables all belong to a string variable. Connect them using
FStr, which reduces the number of variables to keep track of.
No functional change.
There was only a single case where this parameter was false. Inline
that case. That was the only case that needed the return value, so
remove that as well.
When
1. there is a global variable containing a dollar in its expanded name
(very unlikely since there are lots of undocumented edge cases that
make variable names containing dollar signs fragile), and
2. after that (unlikely since that requires .MAKEFLAGS instead of a
normal command line)
3. there is a command line variable of the same name (again very
unlikely since that variable name would contain a dollar sign as
well in the expanded form),
the global variable would not be undefined as promised by the comments
since its name was expanded once more than intended.
Because of the two 'very unlikely' above, this edge case hopefully does
not affect any practical use cases.
Note that this is not about VAR.${param} (which has a dollar sign in its
unexpanded form), but about the case where param itself would expand to
a dollar sign, such as after param=$$.
This is a preparation to extract the code for exporting a cmdline
variable. That code differs in several details from the other code in
ExportVar.
No functional change.