A to-be-closed variable is constant and it must have a close metamethod
when it is created. A program has to go out of its way (e.g., by
changing the variable's metamethod) to invalidate that check. So,
it is not worth to test that again. If the program tampers with the
metamethod, Lua will raise a regular error when attempting to call it.
Undo commit c220b0a5d0: '__close' is not called again in case of
errors. (Upvalue is removed from the list before the call.) The
common error that justified that change was C stack overflows, which
are much rarer with the stackless implementation.
* Avoids multiple definitions of 'lua_assert' in test file.
* Smaller C-stack limit in test mode.
* Note in the manual about the use of false
* Extra test for constant reuse.
Fixed the bug that a key removed from a table might not be found
again by 'next'. (This is needed to allow keys to be removed during a
traversal.) This bug was introduced in commit 73ec04fc.
- After converting a generic GCObject to a specific type ('gco2*'),
avoid using the original GCObject (to reduce aliasing).
- Small corrections in comments in 'lopcodes.h'
- Added tests about who calls __close metamethods
Shrink a stack only when the final stack size can be at most 2/3 the
previous size with half of its entries empty. This commit also
improves the clarity of 'luaD_growstack'.
A "with stack" implementation gains too little in performance to be
worth all the noise from C-stack overflows.
This commit is almost a sketch, to test performance. There are several
pending stuff:
- review control of C-stack overflow and error messages;
- what to do with setcstacklimit;
- review comments;
- review unroll of Lua calls.
When "undumping" a long string, the function 'loadVector' can call the
reader function, which can run the garbage collector, which can collect
the string being read. So, the string must be anchored during the call
to 'loadVector'.
This commit fixes a bug introduced in commit 9cf3299fa. TOUCHED2
objects are always black while the mutator runs, but they can become
temporarily gray inside a minor collection (e.g., if the object is a
weak table).
Tables were using this bit to indicate their array sizes were real
('isrealasize'), but this bit can be useful for tests. Instead, they
can use bit 7 of their 'flag' field for that purpose. (There are only
six fast-access metamethods.) This 'flag' field only exists in tables,
so this use does not affect other types.
OLD1 objects can be potentially anywhere in the 'allgc' list (up
to 'reallyold'), but frequently they are all after 'old1' (natural
evolution of survivals) or do not exist at all (when all objects die
young). So, instead of 'markold' starts looking for them always
from the start of 'allgc', the collector keeps an extra pointer,
'firstold1', that points to the first OLD1 object in the 'allgc' list,
or is NULL if there are no OLD1 objects in that list.
Barriers cannot be active during sweep, even in generational mode.
(Although gen. mode is not incremental, it can hit a barrier when
deleting a thread and closing its upvalues.) The colors of objects are
being changed during sweep and, therefore, cannot be trusted.
When entering a coroutine, the computation of nCcalls added 'from->nci'
to correct for preallocated CallInfos, but 'nci' includes also the
Callinfos already used.
Allow memory errors to be raised through the API (throwing the
error with the memory error message); error in external allocations
raises a memory error; memory errors in coroutines are re-raised
as memory errors.
To allow their use in memory tests, some functions in 'ltests.c'
should never allocate memory. To avoid this allocation, the
library registers the strings used for status codes, and keeps
the variable '_WARN' always defined (with false instead of nil).
In 'lundump.c', when loading the upvalues of a function, there can be
a read error if the chunk is truncated. In that case, the creation
of the error message can trigger an emergency collection while the
prototype is still anchored. So, the prototype must be GC consistent
before loading the upvales, which implies that it the 'name' fields
must be filled with NULL before the reading.
The parser were mixing compiler indices of variables with stack indices,
so that when a to-be-closed variable was used inside the scope of
compile-time constants (which may be optimized away), it might be closed
in the wrong place. (See new tests for examples.)
Besides fixing the bugs, this commit also changed comments and variable
names to avoid that kind of confusion and added tests.
(Undoing part of commit f53eabeed8.) It is better to keep this encoding
stable, so that all Lua versions can read at least the version of a
binary file.
Instead of an explicit value (field 'b'), true and false use different
tag variants. This avoids reading an extra field and results in more
direct code. (Most code that uses booleans needs to distinguish between
true and false anyway.)
The initial "\n\t" to properly indent a searcher message is being added
by 'findloader' when building the error message, instead of being
included in the original message by each searcher itself.
- Several details in 'lcode.c'
- A few more tests for code generation
- Bug in assert in 'lcode.c' ("=" x "==")
- Comments in 'lopcodes.h' and 'ltable.c'
The difference in performance between immediate operands and K operands
does not seem to justify all those extra opcodes. We only keep OP_ADDI,
due to its ubiquity and because the difference is a little more relevant.
(Later, OP_SUBI will be implemented by OP_ADDI, negating the constant.)
In arithmetic/bitwise operators, the call to metamethods is made
in a separate opcode following the main one. (The main
opcode skips this next one when the operation succeeds.) This
change reduces slightly the size of the binary and the complexity
of the arithmetic/bitwise opcodes. It also simplfies the treatment
of errors and yeld/resume in these operations, as there are much
fewer cases to consider. (Only OP_MMBIN/OP_MMBINI/OP_MMBINK,
instead of all variants of all arithmetic/bitwise operators.)
The family of opcodes OP_ADDK (arithmetic operators with K constant)
were not being handled in 'luaV_finishOp', which completes their
task after an yield.
When using warn-mode '@store', from the test library, the tests ensure
not only that the expected warnings were issued, but also that there was
no extra warnings.
The macro 'luaL_pushfail' documents all places in the standard libraries
that return nil to signal some kind of failure. It is defined as
'lua_pushnil'. The manual also got a notation (@fail) to document those
returns. The tests were changed to be agnostic regarding whether 'fail'
is 'nil' or 'false'.
Added the concept of control messages to the warning system, plus the
implementation of the controls "@on"/"@off" to turn warnings on/off.
Moreover, the warning system in the test library adds some other
controls to ease the test of warnings.
When initializing a to-be-closed variable, check whether it has a
'__close' metamethod (or is a false value) and raise an error if
if it hasn't. This produces more accurate error messages. (The
check before closing still need to be done: in the C API, the value
is not constant; and the object may lose its '__close' metamethod
during the block.)
The test "to-be-closed variables in main chunk" was broken,
as it used the removed feature of functions as to-be-closed values.
The error was not detected because its expected result had no lines
to be checked (due to missing new lines).
Attributes changed to posfixed ('x <const>', instead of '<const> x'),
and "toclose" renamed to "close". Posfixed attributes seem to make it
clearer that it applies to only one variable when there are multiple
variables.
Instead of updating 'L->top' in every place that may call a
metamethod, the metamethod functions themselves (luaT_trybinTM and
luaT_callorderTM) correct the top. (When calling metamethods from
the C API, however, the callers must preserve 'L->top'.)
- OP_NEWTABLE can use 'ra + 1' to set top (instead of ci->top);
- OP_CLOSE doesn't need to set top ('Protect' already does that);
- OP_TFORCALL must use 'ProtectNT', to preserve the top already set.
(That was a small bug, because iterators could be called with
extra parameters besides the state and the control variable.)
- Comments and an extra test for the bug in previous item.
As an example, 'print(string.format("%.99f", 1e70))' may have a
lot of garbage after the number.
The old test to ensure that 'string.format("%.99f", n)' was not too
large, 'fabs(n) < 1e100', assumes that the number will fit in the 99
bytes; but the 99 is not the space for the number, it is the added
extra zeros. The option worked for smaller numbers because of the
extra space added to MAX_ITEM.
The addresses of static variables may be different for different
instances of Lua, making these instances incompatible if they use
these addresses as unique keys in the registry (or other tables).
- Macro 'checkliveness' (for debug) always uses 'L', to avoid warnings.
- Some old 'while' changed to 'for' in 'testes/gc.lua'.
- In 'testes/libs/makefile', do not make files depend on 'ltests.h',
which may not even exist.
This commit brings a new implementation for HARDMEMTESTS, which forces
an emergency GC whenever possible. It also fixes some issues detected
with this option:
- A small bug in lvm.c: a closure could be collected by an emergency
GC while being initialized.
- Some tests: a memory address can be immediatly reused after a GC;
for instance, two consecutive '{}' expressions can return exactly the
same address, if the first one is not anchored.
Constants directly assigned to other constants were not propagating:
For instance, in
local <const> k1 = 10
local <const> k2 = k1
'k2' were not treated as a compile-time constant.
String literal expressions have their own kind VKSTR, instead of the
generic VK. This allows strings to "cross" functions without entering
their constant tables (e.g., if they are used only by some nested
function).
An error in a closing method may be caused by a lack of resources,
such as memory or stack space, and the error may free enough resources
(by unwinding the stack) to allow the method to work if called again.
If the closing method is already running after some error (including
its own), it is not called again.
Opcodes OP_NEWTABLE and OP_SETLIST use the same representation to
store the size of the array part of a table. This new representation
can go up to 2^33 (8 + 25 bits).
OP_NEWTABLE is followed by an OP_EXTRAARG, so that it can keep
the exact size of the array part of the table to be created.
(Functions 'luaO_int2fb'/'luaO_fb2int' were removed.)
In the generic for loop, it is simpler for OP_TFORLOOP to use the
same 'ra' as OP_TFORCALL. Moreover, the internal names of the loop
temporaries "(for ...)" don't need to leak internal details (even
because the numerical for loop doesn't have a fixed role for each of
its temporaries).
VLOCAL expressions keep a reference to their corresponding 'Vardesc',
and 'Upvaldesc' (for upvalues) has a field 'ro' (read-only). So, it is
easier to check whether a variable is read-only. The decoupling in
VLOCAL between 'vidx' ('Vardesc' index) and 'sidx' (stack index)
should also help the forthcoming implementation of compile-time
constant propagation.
The syntax for local attributes ('const'/'toclose') was unified with
the regular syntax for local variables, so that we can have variables
with attributes in local definitions with multiple names; for instance:
local <toclose> f, <const> err = io.open(fname)
This new syntax does not implement constant propagation, yet.
This commit also has some small improvements to the manual.
- Better documentation in 'testes/cstack.lua' about using
'debug.setCstacklimit' to find a good limit.
- Constant LUAI_MAXCSTACK gets added CSTACKERR (extra stack for
error handling), so that it is compatible with the argument to
'debug.setCstacklimit'.
- Added a test for calling 'debug.traceback' after yields inside
hooks. (Lua 5.3 seems to have a bug there.)
- Removed test "repeat test with '__open' metamethod instead of a
function", as the previous test already uses the '__open' metamethod.
(It changed when functions were removed as possible to-be-closed
variables).
When there are multiple errors when closing objects, the error
reported by the protected call is the first one, for two reasons:
First, other errors may be caused by this one;
second, the first error is handled in the original execution context,
and therefore has the full traceback.
Instead of a 'tocont' flag, the function 'warn' in Lua now receives all
message pieces as multiple arguments in a single call. Besides being
simpler to use, this implementation ensures that Lua code cannot create
unfinished warnings.
- tests show progress in real time, so that we can see maximum
stack levels even if test crashes.
- new test for recursion continuing into message handler.
- new error message for "attempt to assign to const variable"
- note in the manual about compatibility options
- comments
- small changes in 'read_line' and 'pushstr'
The flag for to-be-closed variables was changed from '*toclose'
to '<toclose>'. Several people found confusing the old syntax and
the new one has a clear terminator, making it more flexible for
future changes.
Back to how it was, a coroutine does not unwind its stack in case of
errors (and therefore do not close its to-be-closed variables). This
allows the stack to be examined after the error. The program can
use 'coroutine.kill' to close the variables.
The function created by 'coroutine.wrap', however, closes the
coroutine's variables in case of errors, as it is impossible to examine
the stack any way.
- 'L' added to the 'BuffFS' structure
- '%c' does not handle control characters (it is not its business.
This now is done by the lexer, who is the one in charge of that
kind of errors.)
- avoid the direct use of 'l_sprintf' in the Lua kernel
Both when setting a path and searching for a file ('searchpath'),
this commit reduces the number of intermediate strings created
in Lua.
(For setting a path the change is not relevant, because this is
done only twice when loading the module. Anyway, it is a nice example
of how to use auxlib buffers to manipulate strings in the C API.)
The function 'luaO_pushvfstring' now uses an internal buffer to
concatenate small strings, instead of pushing all pieces on the
stack. This avoids the creation of several small Lua strings for each
piece of the result. (For instance, a format like "n: '%d'" used to
create three intermediate strings: "n: '", the numeral, and "'".
Now it creates none.)
Avoid creating extra strings when possible:
- avoid creating new resulting string when subject was not modified
(instead, return the subject itself);
- avoid creating strings representing the captured substrings when
handling replacements like '%1' (instead, add the substring directly
to the buffer).
Checks of the form '1 <= x && x <= M' were rewritten in the form
'(unsigned)x - 1 < (unsigned)M', which is usually more efficient.
(Other similar checks have similar translations.) Although
some compilers do these optimizations, that does not happen
for all compilers or all cases.
When calling metamethods for things like 'a < 3.0', which generates
the opcode OP_LTI, the C register tells that the operand was
converted to an integer, so that it can be corrected to float when
calling a metamethod.
This commit also includes some other stuff:
- file 'onelua.c' added to the project
- opcode OP_PREPVARARG renamed to OP_VARARGPREP
- comparison opcodes rewritten through macros
Changed some implementation details; in particular, it is back using
an internal variable to keep the index, with the control variable
being only a copy of that internal variable. (The direct use of
the control variable demands a check of its type for each access,
which offsets the gains from the use of a single variable.)
- LUAC_VERSION is equal to LUA_VERSION_NUM, and it is stored
as an int.
- 'sizeof(int)' and 'sizeof(size_t)' removed from the header, as
the binary format does not depend on these sizes. (It uses its
own serialization for unsigned integer values.)
The numerical 'for' loop over integers now uses a precomputed counter
to control its number of iteractions. This change eliminates several
weird cases caused by overflows (wrap-around) in the control variable.
(It also ensures that every integer loop halts.)
Also, the special opcodes for the usual case of step==1 were removed.
(The new code is already somewhat complex for the usual case,
but efficient.)
All UTF-8 encoding functionality (including the escape
sequence '\u') accepts all values from the original UTF-8
specification (with sequences of up to six bytes).
By default, the decoding functions in the UTF-8 library do not
accept invalid Unicode code points, such as surrogates. A new
parameter 'nonstrict' makes them accept all code points up to
(2^31)-1, as in the original UTF-8 specification.
- The warning functions get an extra parameter that tells whether
message is to be continued (instead of using end-of-lines as a signal).
- The user data for the warning function is a regular value, instead
of a writable slot inside the Lua state.
When called with no arguments, 'math.randomseed' uses time and ASLR
to generate a somewhat random seed. the initial seed when Lua starts
is generated this way.
Several small improvements (code style, warnings, comments, more tests),
in particular:
- 'lua_topointer' extended to handle strings
- raises an error in 'string.format("%10q")' ('%q' with modifiers)
- in the manual for 'string.format', the term "option" replaced by
"conversion specifier" (the term used by the C standard)
After a major bad collection (one that collects too few objects),
next collection will be major again. In that case, avoid switching
back to generational mode (as it will have to switch again to
incremental to do next major collection).
The function 'string.gmatch' now has an optional 'init' argument,
similar to 'string.find' and 'string.match'. Moreover, there was
some reorganization in the manipulation of indices in the string
library.
This commit also includes small janitorial work in the manual
and in comments in the interpreter loop.
To-be-closed variables must contain objects with '__toclose'
metamethods (or nil). Functions were removed for several reasons:
* Functions interact badly with sandboxes. If a sandbox raises
an error to interrupt a script, a to-be-closed function still
can hijack control and continue running arbitrary sandboxed code.
* Functions interact badly with coroutines. If a coroutine yields
and is never resumed again, its to-be-closed functions will never
run. To-be-closed objects, on the other hand, will still be closed,
provided they have appropriate finalizers.
* If you really need a function, it is easy to create a dummy
object to run that function in its '__toclose' metamethod.
This comit also adds closing of variables in case of panic.
* unification of the 'nny' and 'nCcalls' counters;
* external C functions ('lua_CFunction') count more "slots" in
the C stack (to allow for their possible use of buffers)
* added a new test script specific for C-stack overflows. (Most
of those tests were already present, but concentrating them
in a single script easies the task of checking whether
'LUAI_MAXCCALLS' is adequate in a system.)
The script 'all', to run all tests, automatically ensures that the
Lua interpreter and the test C libraries (in 'testes/libs/') are
updated with any changes in 'luaconf.h'.
This file is not part of the regular tests. It tests error conditions
that demand too much memory or too much time to create:
* string with too many characters
* control structure with body too large
* chunk with too many lines
* identifier with too many characters
* chunks with too many instructions
* function with too many constants
* too many strings internalized
* table with too many entries
In machines with limited memory (less than 150 GB), many tests run up
to a "not enough memory" error. We need some memory (~256 GB) to
run all tests up to their intrinsic limits.
New functions to reset/kill a thread/coroutine, mainly (only?) to
close any pending to-be-closed variable. ('lua_resetthread' also
allows a thread to be reused...)
The call 'math.rand()' converts the higher bits of the internal unsigned
integer random to a float, instead of its lower bits. That ensures that
Lua compiled with different float precisions always generates equal (up
to the available precision) random numbers when given the same seed.
A to-be-closed variable must be closed when a block ends, so even
a 'return foo()' cannot directly returns the results of 'foo'; the
function must close the scope before returning.
It is an error for a to-be-closed variable to have a non-closable
non-nil value when it is being closed. This situation does not seem to
be useful and often hints to an error. (Particularly in the C API, it is
easy to change a to-be-closed index by mistake.)
To remove a to-be-closed variable from the stack in the C API a
function must use 'lua_settop' or 'lua_pop'. Previous implementation of
'luaL_pushresult' was not closing the box. (This commit also added
tests to check that box is being closed "as soon as possible".)
Added opcodes for all seven arithmetic operators with K operands
(that is, operands that are numbers in the array of constants of
the function). They cover the cases of constant float operands
(e.g., 'x + .0.0', 'x^0.5') and large integer operands (e.g.,
'x % 10000').
Sometimes it is useful to mark to-be-closed an index that is not
at the top of the stack (e.g., if the value to be closed came from
a function call returning multiple values).
The variable to be closed in a generic 'for' loop now is the
4th value produced in the loop initialization, instead of being
the loop state (the 2nd value produced). That allows a loop to
use a state with a '__toclose' metamethod but do not close it.
(As an example, 'f:lines()' might use the file 'f' as a state
for the loop, but it should not close the file when the loop ends.)
The new syntax is <local *toclose x = f()>. The mark '*' allows other
attributes to be added later without the need of new keywords; it
also allows better error messages. The API function was also renamed
('lua_tobeclosed' -> 'lua_toclose').
The mechanism of "caching the last closure created for a prototype to
try to reuse it the next time a closure for that prototype is created"
was removed. There are several reasons:
- It is hard to find a natural example where this cache has a measurable
impact on performance.
- Programmers already perceive closure creation as something slow,
so they tend to avoid it inside hot paths. (Any case where the cache
could reuse a closure can be rewritten predefining the closure in some
variable and using that variable.)
- The implementation was somewhat complex, due to a bad interaction
with the generational collector. (Typically, new closures are new,
while prototypes are old. So, the cache breaks the invariant that
old objects should not point to new ones.)
The implicit variable 'state' in a generic 'for' is marked as a
to-be-closed variable, so that the state will be closed as soon
as the loop ends, no matter how.
Taking advantage of this new facility, the call 'io.lines(filename)'
now returns the open file as a second result. Therefore,
an iteraction like 'for l in io.lines(name)...' will close the
file even when the loop ends with a break or an error.
"Better" and similar to error messages for invalid function arguments.
*old message: 'for' limit must be a number
*new message: bad 'for' limit (number expected, got table)
Statements like 'if cond then goto label' generate code so that the
jump in the 'if' goes directly to the given label. This optimization
cannot be done when the jump is backwards leaving the scope of some
variable, as it cannot add the needed 'close' instruction. (The jumps
were already generated by the 'if'.)
This commit also added 'likely'/'unlikely' for tests for errors in
the parser, and it changed the way breaks outside loops are detected.
(Now they are detected like other goto's with undefined labels.)
Added restriction that, when a label is created, there cannot be
another label with the same name visible. That allows backward goto's
to be resolved when they are read. Backward goto's get a close if
they jump out of the scope of some variable; labels get a close only
if previous goto to it jumps out of the scope of some upvalue.
A closing method cannot be called in its own stack slot, as there may
be returning values in the stack after that slot, and the call would
corrupt those values. Instead, the closing method must be copied to the
top of the stack to be called.
Moreover, even when a function returns no value, its return istruction
still has to have its position (which will set the stack top) after
the local variables, otherwise a closing method might corrupt another
not-yet-called closing method.
