them whenever the user tries to step on one, we can change our behaviour
back to what the kernel considers to be that of a well behaved shell
(wrt file descriptor usage). If our user causes problems, we will soon
move into recalcitrant process territory, but that should normally be
rare. This should reduce kernel overheads a little.
parser tracing is useful when debugging the parser (which admittedly is
fairly often...) but there is a lot of it, and it gets in the way when
looking at something else. Now we can turn it off when not wanted.
option sorting (no longer required option.list to be manually
sorted by long option name) and properly handles conditional
options. Cleaner output format as well.
This allows option.list to be reordered to group related options
together ... also added more comments to it.
Unless the shell is compiled with the (compilation time) option
BOGUS_NOT_COMMAND (as in CFLAGS+=-DBOGUS_NOT_COMMAND) which it
will not normally be, the ! command (reserved word) will only
be permitted at the start of a pipeline (which includes the
degenerate pipeline with no '|'s in it of course - ie: a simple cmd)
and not in the middle of a pipeline sequence (no "cmd | ! cmd" nonsense.)
If the latter is really required, then "cmd | { ! cmd; }" works as
a standard equivalent.
In POSIX mode, permit only one ! ("! pipeline" is ok. "! ! pipeline" is not).
Again, if needed (and POSIX conformance is wanted) "! { ! pipeline; }"
works as an alternative - and is safer, some shells treat "! ! cmd" as
being identical to "cmd" (this one did until recently.)
inheritance when a variable is declared local, but instead leave
the local var unset (if not given a value) in the function.
Only ash derived shells do inheritance it seems.
So, to compensate for that, and get one step closer to making
"local" part of POSIX, so we can really rely upon it, a compromise
has been suggested, where "local x" is implementation defined
when it comes to this issue, and we add "local -I x" to specify
inheritance, and "local -N x" to specify "not" (something...)
(not inherited, or not set, or whatever you prefer to imagine!)
The option names took a lot of hunting to find something reasonable
that no shell (we know of) had already used for some other purpose...
The I was easy, but 'u' 'U' 'X' ... all in use somewhere.
This implements that (well, semi-) agreement.
While here, add "local -x" (which many other shells already have)
which causes the local variable to be made exported. Not a lot
of gain in that (since "export x" can always be done immediately
after "local x") but it is very cheap to add and allows more other
scripts to work with out shell.
Note that while 'local x="${x}"' always works to specify inheritance
(while making the shell work harder), "local x; unset x" does not
always work to specify the alternative, as some shells have
"re-interpreted" unset of a local variable to mean something that
would best be described as "unlocal" instead - ie: after the unset
you might be back with the variable from the outer scope, rather
than with an unset local variable.
Also add "unset -x" to allow unsetting a variable without removing
any exported status it has.
There are gazillions of other options that are not supported here!
Originally, MKCRYPTO was introduced because the United States
classified cryptography as a munition and restricted its export. The
export controls were substantially relaxed fifteen years ago, and are
essentially irrelevant for software with published source code.
In the intervening time, nobody bothered to remove the option after
its motivation -- the US export restriction -- was eliminated. I'm
not aware of any other operating system that has a similar option; I
expect it is mainly out of apathy for churn that we still have it.
Today, cryptography is an essential part of modern computing -- you
can't use the internet responsibly without cryptography.
The position of the TNF board of directors is that TNF makes no
representation that MKCRYPTO=no satisfies any country's cryptography
regulations.
My personal position is that the availability of cryptography is a
basic human right; that any local laws restricting it to a privileged
few are fundamentally immoral; and that it is wrong for developers to
spend effort crippling cryptography to work around such laws.
As proposed on tech-crypto, tech-security, and tech-userlevel to no
objections:
https://mail-index.netbsd.org/tech-crypto/2017/05/06/msg000719.htmlhttps://mail-index.netbsd.org/tech-security/2017/05/06/msg000928.htmlhttps://mail-index.netbsd.org/tech-userlevel/2017/05/06/msg010547.html
P.S. Reviewing all the uses of MKCRYPTO in src revealed a lot of
*bad* crypto that was conditional on it, e.g. DES in telnet... That
should probably be removed too, but on the grounds that it is bad,
not on the grounds that it is (nominally) crypto.
sh +c "command string" no longer works (it must be -c)
sh +o and sh -o no longer work (if you could call what they did
before working.) nb: this is without an option name.
-ooo Opt1 Opt2 Opt3 no longer works (set & cmd line), this should be
-o Opt1 -o Opt2 -o Opt3 (same with +ooo of course).
-oOpt is now supported - option value (name of option in
this case) immediately following -o (or +o).
(as with other commands that use std opt parsing)
Both set comamnd and command line.
In addition, the output from "set +o" has shrunk dramatically, by borrowing
a trick from ksh93 (but implemented in a more traditional syntax).
"set +o" is required to produce a command (or commands) which when executed
later, will return all options to the state they were in when "set +o"
was done. Previously that was done by generating a set command, with
every option listed (set -o opt +o other-opt ...) to set them all back
to their current setings. Now we have a new "magic option" ("default")
which sets all options to their default values, so now set +o output
need only be "set -o default -o changed-opt ..." (only the options that
have been changed from their default values need be explicitly mentioned.)
The definition of "default value" for this is the value the shell set the
option to, after startup, after processing the command line (with any
flags, or -o option type settings), but before beginning processing any
user input (incuding startup files, like $ENV etc).
Anyone can execute "set -o default" of course, but only from a "set"
command (it makes no sense at all as a -o option to sh). This also
causes "set +o" to be slightly more useful as a general command, as
ignoring the "set -o default" part of the result, it lists just those
options that have been altered after sh startup. There is no +o default.
There isn't an option called "default" at all...
This causes some of the commented out text from sh.1 to become uncommented.
levels for debug output. This change accidentally omitted earlier (only
effect is incorrect nesting levels shown in trace output when the option
to show them is enabled.) NFC for any normal shell build.
output from "sh -x" only (tracing execution), not quoting + is better,
as it makes tracing commands with + and - options, or numbers, more
consistent.
Also one minor white space change (excess indentation removed).
compiled for DEBUG.)
Add debug builtin command, and corresponding -D command line option.
As usual, for DEBUG related stuff, read the source for info, that's
all there is about this.
This completes the infrastructure changes for the updated DEBUG TRACE
mechanism, so now converting the rest of the shell's internal tracing
can happen as desired - piecemeal.
upgrade a while ago (this should make no difference to anything
other than a minor - very minor - build time speedup, ld is
smart enough to relaise that nothing from the lex library was
needed, and the executable contains no reference to it, even
befor ethis change.)
Document the (slightly) enhanced NETBSD_SHELL.
Fix a typo (one of my typos...)
Move a commented out section to align with current planned changes
(and fix its commented out markup.)
MKREPRO_TIMESTAMP (as an additional word in the value, with a "BUILD:" prefix)
if it is set during the build. (Trailing 00 pairs in the time are removed).
While here, throw in some extra words that list the compilation
options used which alter sh behaviour (mostly by removing stuff.)
Usually that will only be noticed in a SMALL shell compiled for
install media, or similar - none of the others (not that there
are many) are ever changed from the default in a normal build
(default settings are just omitted.) This also allows scripts
to tell if they are running in a DEBUG shell, which can sometimes
make debugging easier.
First, be aware that the DEBUG spoken of here has nothing whatever to
do with MKDEBUG=true type builds of NetBSD. The only way to get a
DEBUG shell is to build it yourself manually.
That said, for non-DEBUG shells, this change makes only one slight
(trivial really) difference, which should affect nothing.
Previously some code was defined like ...
function(args)
{
#ifdef DEBUG
/* function code goes here */
#endif
}
and called like ...
#ifdef DEBUG
function(params);
#endif
resulting in several empty functions that are never called being
defined in non-DEBUG shells. Those are now gone. If you can detect
the difference any way other than using "nm" or similar, I'd be very
surprised...
For DEBUG shells, this introduces a whole new TRACE() setup to use
to assist in debugging the shell.
I have had this locally (uncommitted) for over a year... it helps.
By itself this change is almost useless, nothing really changes, but
it provides the framework to allow other TRACE() calls to be updated
over time. This is why I had not committed this earlier, my previous
version required a flag day, with all the shell's internal tracing
being updated a once - which I had done, but that shell version has
bit-rotted so badly now it is almost useless...
Future updates will add the mechanism to allow the new stuff to actually
be used in a productive way, and following that, over time, gradual
conversion of all the shell tracing to the updated form (as required,
or when I am bored...)
The one useful change that we do get now is that the fd that the shell
uses for tracing (which was usually 3, but not any more) is now protected
from user/script interference, like all the other shell inernal fds.
There is no doc (nor will there be) on any of this, if you are not reading
the source code it is useless to you, if you are, you know how it works.
arg list processing), and the set command in general.
Also add some (new) commented out text related to options which may
be commented back in sometime soon...
${#VAR:-foo} (or any other modifier on ${#VAR} is a syntax error.
On the other hand ${##} is not, nor is ${##13} though they mean
quite different things (the latter is an idiom everyone should learn,
... $# except we refuse to admit the possibility that it is 13...
Even I cannot explain what ${#-foo} used to do, but it wasn't sane!
(It should be just $# as $# is never unset, but ...)
Shell syntax is truly a wondrous thing!
one of the words happens to contain ${#var}. (This is the command
string shown by the "jobs" command, and when a background job completes)
While here, undo the LINENO hack when building that string.
And one ot two other foibles...
POSIX requires that the output of the "set" command (with no args -- it
gives a list of variables, and their values) be sorted according to
the collating sequence defined by the current locale.
Now I'm not aware of any locale where the collating sequence order of
ascii letters, digits, and '_' are any different than they are in the
C locale (and those are the only characters that can occur in variable
names - unless there is perhaps a locale that defines "dictionary" order
as the sort order) but never mind, that isn't the bug...
What "collating sequence order" does mean however, if not "collating
sequence order, except when we happen to have two variable names, where
one name is a prefix of the other (say X and XY) and the first character
of the 'Y' part of the longer name happens to be a digit..."
"set" is not a frequently used command (particularly in scripts where
it matters - that is, the no args form, nothing here alters anything
about any use of set with args) and is already a bit slow (sluggish...)
because of the sort requirement, so let's make it fractionally even
slower, but correct.
! ! pipeline
(And for now the other places where ! is permitted)
we should at least generate the logically correct exit
status:
! ! (exit 5); echo $?
should print 1, not 5. ksh and bosh do it this way - and it makes sense.
bash and the FreeBSD sh echo "5" (as did we until now.)
dash, zsh, yash all enforce the standard syntax, and prohibit this.
by FreeBSD sh (though different, for other reasons) - but the bug discovered
while searching for why a (nonsense) attempted test of the forthcoming
code to handle "! ! pipeline" properly wasn't working... (it was how I was
testing it that was broken, but until I achieved enlightenment, I was bug
hunting, and found this...)
Most likely the bugs here wouldn't have affected any real code (no bug
reports anyway), but ...
(even if no shell in existence, that I am aware of, does that).
That is, POSIX says ... [of the trap command with no args]
The shell shall format the output, including the proper use of
quoting, so that it is suitable for re-input to the shell as commands
that achieve the same trapping results. For example:
save_traps=$(trap)
...
eval "$save_traps"
It is obvious what the intent is there. But no shell makes it work.
An example using bash (as the NetBSD shell, still does not do the save_traps=
stuff correctly - but that is a problem for a different time and place...)
Given this script
printf 'At start: '; trap
printf '\n'
traps=$(trap)
trap 'echo hello' INT
printf 'inside : '; trap
printf '\n'
eval "${traps}"
printf 'At end : '; trap
printf '\n'
One would expect that (assuming no traps are set at the start, and
there aren't) that the first trap will print nothing, then the inside
trap will show the trap that was set, and then when we get to the
end everything will be back to nothing again.
But:
At start:
inside : trap -- 'echo hello' SIGINT
At end : trap -- 'echo hello' SIGINT
And of course. when you think about it, it is obvious why this happens.
The first "trap" command prints nothing ... nothing has changed when we
get to the "traps=$(trap)" command ... that trap command also prints
nothing. So this does traps=''. When we do eval "${traps}" we are
doing eval "", and it is hardly surprising that this accomplishes nothing!
Now we cannot rationally change the "trap" command without args to
behave in a way that would make it useful for the posix purpose (and
here, what they're aiming for is good, it should be possible to
accomplish that objective) so is there some other way?
I think I have seen some shell (but I do not remember which one) that
actually has "trap -" that resets all traps to the default, so with that,
if we changed the 'eval "${traps}"' line to 'trap -; eval "${traps}"'
then things would actually work - kind of - that version has race conditions,
so is not really safe to use (it will work, most of the time...)
But, both ksh93 and bash have a -p arg to "trap" that allows information
about the current trap status of named signals to be reported. Unfortunately
they don't do quite the same thing, but that's not important right now,
either would be usable, and they are, but it is a lot of effort, not
nearly as simple as the posix example.
First, while "trap -p" (with no signals specified) works, it works just
the same (in both bash and ksh93, aside from output format) as "trap".
That is, that is useless. But we can to
trap_int=$(trap -p int)
trap_hup=$(trap -p hup)
...
and then reset them all, one by one, later...
(bash syntax)
test -n "${trap_int}" && eval "${trap_int}" || trap - int
test -n "${trap_hup}" && eval "${trap_hup}" || trap - hup
(ksh93 syntax)
trap "${trap_int:-}" int
trap "${trap_hup:-}" hup
the test (for bash) and variable with default for ksh93, is needed
because they both still print nothing if the signal action is the default.
So, this modification attempts to fix all of that...
1) we add trap -p, but make it always output something for every signal
listed (all of the signals if none are given) even if the signal
action is the default.
2) choose the bash output format for trap -p, over the ksh93 format,
even though the simpler usage just above makes the ksh93 form seem
better. But it isn't. Consider:
ksh93$ trap -p int hup
echo hello
One of the two traps has "echo hello" as its action, the other is
still at the default, but which?
From bash...
bash$ trap -p int hup
trap -- 'echo hello' SIGINT
And now we know! Given the bash 'trap -p' format, the following function
produces ksh93 format output (for use with named signals only) instead...
ksh93_trap_p() {
for _ARG_ do
_TRAP_=$(trap -p "${_ARG_}") || return 1
eval set -- "${_TRAP_}"
printf '%s' "$3${3:+
}"
done
return 0
}
[ It needs to be entered without the indentation, that '}"' line has to be
at the margin. If the shell running that has local vars (bash does) then
_ARG_ and _TRAP_ should be made local. ]
So the bash format was chosen (except we do not include the "SIG" on the
signal names. That's irrelevant.)
If no traps are set, "trap -p" will say (on NetBSD of course)...
trap -- - EXIT HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS
trap -- - PIPE ALRM TERM URG STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ
trap -- - VTALRM PROF WINCH INFO USR1 USR2 PWR RT0 RT1 RT2 RT3 RT4 RT5
trap -- - RT6 RT7 RT8 RT9 RT10 RT11 RT12 RT13 RT14 RT15 RT16 RT17 RT18
trap -- - RT19 RT20 RT21 RT22 RT23 RT24 RT25 RT26 RT27 RT28 RT29 RT30
Obviously if traps are set, the relevant signal names will be removed from
that list, and additional lines added for the trapped signals.
With args, the signals names are listed, one line each, whatever
the status of the trap for that signal is:
$ trap -p HUP INT QUIT
trap -- - HUP
trap -- 'echo interrupted' INT
trap -- - QUIT
3) we add "trap -" to reset all traps to default. (It is easy, and seems
useful.)
4) While here, lots of generic cleanup. In particular, get rid of the
NSIG+1 nonsense, and anything that ever believes a signo == NSIG
is in any way rational. Before there was a bunch of confusion,
as we need all the signals for traps, plus one more for the EXIT
trap, which looks like we then need NSIG+1. But EXIT is 0, NSIG
includes signals from 0..NSIG-1 but there is no signal 0, EXIT
uses that slot, so we do not need to add and extra one, NSIG is
enough. (To see the effect of this, use a /bin/sh from before
this fix, and compare the output from
trap '' 64
and trap '' 65
both invalid signal numbers.
Then try just "trap" and watch your shell drop core...)
Eventually NSIG needs to go away completely (from user apps), it
is not POSIX, it isn't really useful (unless we make lots of
assumptions about how signals are numbered, which are not guaranteed,
so even if apps, like this sh, work on NetBSD, they're not portable,)
and it isn't necessary (or will not be, soon.)
But that is for another day...
5) As is kind of obvious above, when listing "all" traps, list all the
ones still at their defaults, and all the ignored signals, on as
few lines as possible (it could all be on one line - technically it
would work as well, but it would have made this cvs log message
really ugly...) Signals with a non-null action still get listed
one to a line (even if several do have the exact same action.)
6) Man page updates as well.
After this change, the following script:
printf 'At start: '; trap
printf '\n'
trap -p >/tmp/out.$$
trap 'echo hello' INT
printf 'inside : '; trap
printf '\n'
. /tmp/out.$$; rm /tmp/out.$$
printf 'At end : '; trap
printf '\n'
which is just the example from above,
using "trap -p" instead of just "trap" to save the traps,
and modified to a form that will work with the NetBSD shell today
produces:
At start:
inside : trap -- 'echo hello' INT
At end :
[Do I get a prize for longest commit log message of the year?]
(by which they mean > 0). We were checking for negative numbers, but
not for 0. More by chance of the implementation than any specific design
(I suspect) "break 0" was being treated the same as "break" or "break 1".
Since 3 ways to achieve the same thing is overkill, let's do what posix
wants and forbid "break 0" and "continue 0".
which causes fall through the to command list of the following pattern
(wuthout evaluating that pattern). This has been approved for inclusion
in the next major version of the POSIX standard (Issue 8), and is
implemented by most other shells.
Now all form a circle and together attempt to summon the great wizd
in the hopes that his magic spells can transform the poor attempt
at documenting this feature into something rational...
shells (anything made by build.sh) there is no change at all.
In DEBUG shells, when tree dumping, remember to include NNOT (same
omission as was just corrected in jobs.c :1.81) - of course, here there
are lots of other node types not handled as well.
ALso, avoid a core dump bug when doing a tree dump of a pieline
where the commands are not all simple commands (which can only
happen with a command string like " cmd | ! cmd | ... ". The "!"
in the middle is utter nonsense, and should be forbidden, but
for now, at least avoid a core dump.
"jobs" output (or other places where the cmd string is shown - like
when reporting status when a background job completes.)
Without this fix, try
! sleep 5 &
jobs
wait
and try not to wonder at the '???" that appears instead of "! sleep 5"
fixed (there might be no way) - but it "feels right"!
When popping an (exhausted) input string off the input stack, allow
for the possibility that the previous string might also just happened
to have run out of steam as well, so keep poppin' along until we
run out of pop, or find something to consume.
If there are no arguments, or only null arguments,
eval shall return a zero exit status;
Make it so. Now:
false; eval; echo $?
produces 0 instead of 1.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...
This fix from FreeBSD (thanks again folks).
To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).
All this because I wanted to see what number would come from
echo $\
{\
L\
I\
N\
E\
N\
O\
}
and was surprised at the result! That works now...
The bug would also affect stuff like
true &\
& false
and all kinds of other uses where the \newline occurred in the
"wrong" place.
An ATF test for sh syntax is coming... (sometime.)
arg (struct alias *) rather than using void * and then casting it
when used. For callers, the arg either is a struct alias *, or is NULL,
so nothing to adjust there.
NB: This change untested by itself, it was going to be a part of the next
change (coming in a few minutes) but is logically unrelated, so ...
