- committed separately to make it easier to swap back if desired.
This version avoids open coding is_name() and isdigit() - measurements
show it is perhaps fractionally faster (though the difference is so small
as to probably be statistically insignifigant - if I were a statician
I might understand that) and just a little larger than the previous version.
closing PR bin/50958
That meant adding the assignment operators ('=', and all of the +=, *= ...)
Currently, ++, --, and ',' are not implemented (none of those are required
by posix) but support for them (most likely ',' first) might be added later.
To do this, I removed the yacc/lex arithmetic parser completely, and
replaced it with a hand written recursive descent parser, that I obtained
from FreeBSD, who earlier had obtained it from dash (Herbert Xu).
While doing the import, I cleaned up the sources (changed some file names
to avoid requiring a clean build, or signifigant surgery to the obj
directories if "build.sh -u" was to be used - "build.sh -u" should work
fine as it is now) removed some dashisms, applied some KNF, ...
Remove some unnecessary cuteness that limited error reporting.
Permit just one -s arg to fdflags
Be deterministic in the case of fdflags -s +cloexec,-cloexec 0
(and similar) - use the last specified, always.
Allow:
FD_0_FLAGS=$( fdflags -v 0 )
# do stuff, manipulating the flags
fdflags -s "FD_0_FLAGS" 0
to save/restore flags for a fd.
Correctly mask result of fcntl(fd, F_GETFD) with FD_CLOEXEC as the
specs require before deciding close on exec is set.
Improve portability as a tool, don't assume strtoi(), nor __arraycount()
and avoid needlessly requiring recent C versions (ie: there's no need to
sprinkle declarations in the middle of the code, it just makes them hard
to find, and benefits nothing.)
Still to do: As currently implemented, both user, and shell internal fds
are reported, and can be manipulated. Allowing users to touch the shell's
internal fds is bogus, and providing this easy way to allow users to
discover which values they have is poor. Fixing this means getting rid
of the use of fcntl(F_MAXFD) and replacing it with a shell maintained
memory of what fds the user (script) has allocated. The shell's fd
manipulation really still needs major work (including properly fixing
bin/48875)
From the POSIX time.h man page:
"The formal definition of UTC does not permit double leap seconds, so all mention of double leap seconds has been removed, and the range shortened from the former [0,61] seconds seen in previous versions of POSIX."
Move email addresses to same line as author name.
Do not split the third author entry on to a new line.
Public domain is not licensed by definition, heads up by Robert Elz.
Unfortunately only the source code for the compiler is present for that
release. The v2 archive is missing the manuals & only contains a some of the
commands.
Switch to v3 and reference manual category eight.
Unfortunately only the source code for the compiler is present for that
release. The v2 archive is missing the manuals & only contains a some of the
commands.
Switch to v3 and reference manual category eight, as per r1.32 of
src/bin/ps/ps.1 from OpenBSD.
Heads up from Ingo Schwarze.
Redo earlier fix to only prohibit sourcing directories and block special files.
char specials (/dev/tty, /dev/null, ... incl /dev/rwd0a) and fifos are OK.
Posix actually requires that we find only readable files - that is not yet
implemented (doing it sanely, without opening the file twice, is going to
take some more modifications to code elsewhere).
More fixes to the shell parser to prevent empty simple commands (where
empty means no significant text at all) - as discussed on tech-userlevel
this still allows { } (which can be useful in function definitions, not
really anywhere else though) except in posix mode. ( ) now generates
a syntax error, as should any other place where commands are required but
nothing is present. (nb, redirections, var assignments, even var expansions
that expand to nothing, are all OK, and avoid the error - just comments, or
whits space, are not.) This is (aside from allowing { } at all) all in
accordance with the posix spec.
buffer. If snprintf truncated the output, the length returned will be
greater and we'll write trash. Just call strlen instead. (And since
what we're doing is writing progress messages to the user, checking
carefully for truncation isn't really worthwhile either.)
Spotted when attending to PR 50998 from David Binderman; the issue
there (computation of an unused value) popped up because one of the
prints was already calling strlen.
code reading, rather than any actual real use case failing.
With this script
f()
{
echo hello $1
}
exec 3>&1
echo $(
for i in a b c
do
echo @$i
f >&3
done >/tmp/foo
)
echo foo= $(cat /tmp/foo)
what should be output is
hello
hello
hello
foo= @a @b @c
but since the (my) 48875 fix the other day, we've been getting
hello
@b
hello
@c
hello
foo= @a
This fixes that. I think (hope) this is the last of these fixes...
copyfd() duplicates file descriptors - it used to be widely used,
but these days has seen its popularity dwindle. Strip it of an
option that ceased to be variable (simplifying code) and cause all
its users to check its result, so it does not need to handle errors
itself (simplifying code further), and make it become a private inernal
routine in redir.c (all callers from other places have switched to a
more modern interface.) Make sure we error() if N>&N fails (if N is closed.)
(where N is a decimal fd number) either when used as
some-command N>&N
(where fd N is passed, open, to some-command - which is obviously what is
wanted)
Or as
exec N>&N
which effects fd N for all future commands.
Note that this means
exec N>foo N>&N
returns to the old behaviour of leaving the file descriptor open
when commands are run (as do most shells, other than ksh) and works for
both new and old NetBSD shells (old ones never set close-on-exec, and treat
N>&N as a rather meaingless no-op request, and just ignore it), new ones
set close-on-exec on the first redirection, then disable it again on the
second.
Everything here about >& for output fds applies to <& for input ones.
OK christos@
must be assumed to have something following, even if the loop itself doesn't,
so redirected fd's around func calls need to be saved. Should fix etcupdate
command in the current shell (so they can be restored for the next command)
in cases where it is obvious that there is not going to be a following
command to use them. This fixes the problem reported in the PR (though
there are still plenty of situations where a FD could be closed but isn't,
we do not do full fd flow eveluation to determine whether a fd will be
used or not).
This is the change that was just committed and then backed out again...
OK christos@
a fd to an unspecified high fd number, we certainly do not want to hand
that high fd off to other processes after an exec, so always set close-on-exec
on the result (even if lack of fd's means no fd alteration happens.)
This will (eventually) allow some other code that sets close-on-exec to
be removed, but for now, doing it twice won't hurt. Also, in a N>&M
type redirection, do not set close-on-exec if we don't want it.
OK christos@
completely empty, they must have something (var assignment, redirect,
or command, or multiple of those) to avoid a syntax error. This
matches the requirements of the grammar in the standard. Correct the
parser (using FreeBSD's sh as a guide) and update the man page to
remove text that noted a couple of places this was previously OK.
OK christos@
This allows anything that could be a filesystem command to be
implemented as a function instead. The restriction on '/'
is because of the way that functions are (required to be) searched
for relative to PATH searching - a function with a name containing '/'
could never be executed, so simply prohibit defining such a thing.
ok christos@
Also, move (most of) the shell's internal use fd's to much
higher values (depending upon what ulimit -n allows) so they
are less likely to clash with user supplied fd numbers. A future
patch will (hopefully) avoid this problem completely by dynamically
moving the shell's internal fds around as needed. (From kre@)
POSIX just says "here docs begin after the next newline [token]".
Nothing about "provided it is inside any `` the redirect operator
appears in... As best I can tell, NetBSD now has the only shell to
handle this "correctly" (which raises the question whether it is
correct - but if not, only erroneous scripts are affected.)
This is required by some (probably broken) autoconfigure related
scripts. (from kre@)
following a suggestion from him, the way the fix to PR bin/50993
was implemented has changed a little. There are three steps involved
in processing a here document, reading it, parsing it, and then
evaluating it before applying it to the correct file descriptor for
the command to use. The third of those is not related to this
problem, and has not changed. The bug was caused by combining the
first two steps into one (and not doing it correctly - which would be
hard that way.) The fix is to split the first two stages into
separate events. The original fix moved the 2nd stage (parsing)
to just immediately before the 3rd stage (evaluation.) Jilles
pointed out some unwanted side effects from doing it that way, and
suggested moving the 2nd stage to immediately after the first.
This commit makes that change. The effect is to revert the changes
to expand.c and parser.h (which are no longer needed) and simplify
slightly the change to parser.c. (from kre@)
variable (with its current value set at 20160401) as discussed on
current-users and tech-userlevel. This also includes the necessary
support to implement it properly (particularly the unexportable
part) and adds options to the export command to support unexportable
variables. Also implement the "posix" option (no single letter
equivalent) which gets its default value from whether or not
POSIXLY_CORRECT is set in the environment when the shell starts
(but can be changed just like any other option using -o and +o on
the command line, or the set builtin command.) While there, fix
all uses of options so it is possible to have options that have a
short (one char) name, and no long name, just as it has been possible
to have options with a long name and no short name, though there
are currently none (with no long name). For now, the only use of
the posix option is to control whether ${ENV} is read at startup
by a non-interactive shell, so changing it with set is not usful
- that might change in the future. (from kre@)
when given without braces (ie: $2 etc). Only the first 9 shell
parameters ($1 .. $9) and the special parameter ($0) can be
referenced this way, $10 is ${1}0 not ${10}. Make it so.
This bug brought to notice by Sven Mascheck's web pages which
discuss (among other things) the history of this (and other ash
based) shells .. see http://www.in-ulm.de/~mascheck/ (from kre@)
user is expected. With the previous commits to parser.c, we no
longer need to handle reading here documents in the (massive)
readtoken1() function. That allows its code to be simplified and
made easier to read and understand (several goto's goto goto heaven.
RIP) (from kre@)
a suggestion from him, the way the fix to PR bin/50993 was implemented
has changed a little. There are three steps involved in processing
a here document, reading it, parsing it, and then evaluating it
before applying it to the correct file descriptor for the command
to use. The third of those is not related to this problem, and
has not changed. The bug was caused by combining the first two
steps into one (and not doing it correctly - which would be hard
that way.) The fix is to split the first two stages into separate
events. The original fix moved the 2nd stage (parsing) to just
immediately before the 3rd stage (evaluation.) Jilles pointed out
some unwanted side effects from doing it that way, and suggested
moving the 2nd stage to immediately after the first. This commit
makes that change. The effect is to revert the changes to expand.c
and parser.h (which are no longer needed) and simplify slightly
the change to parser.c. (from kre@)
documents are processed. Now, when first detected, they are
simply read (the only change made to the text is to join lines
ended with a \ to the subsequent line, otherwise end marker detection
does not work correctly (for here docs with an unquoted endmarker
only of course.) This patch also moves the "internal subroutine"
for looking for the end marker out of readtoken1() (which had to
happen as readtoken1 is no longer reading the here doc when it is
needed) - that uses code mostly taken from FreeBSD's sh (thanks!)
and along the way results in some restrictions on what the end
marker can be being removed. We still do not allow all we should.
(from kre@)
Any redirect (or redirects) before a function definition were
allowed by the parser, but otherwise totally ignored. The standard
syntax does not permit redirects there, now, neither do we. (from kre@)
magic string " \t\n" all over the place, slightly improved
syntax error messages, restructured some of the code for
clarity, don't allow IFS to be imported through the environment,
and remove the (never) conditionally compiled ATTY option.
Apart from one or two syntax error messages, and ignoring IFS
if present in the environment, this is intended to have no
user visible changes. (from kre@)
readtoken1() into a real function of its own (inspired by a
similar change made by FreeBSD - the other internal routines
they moved out are expected to move out here as well soon.)
This change helps avoid gcc 5.3 demanded (not required!) volatile
noise which would slow down parsing. (from kre)
(undoing the effect of that commit on syntax.h when it was
being dynamically generated) from 1996. This means that the shell
parser is now locale independent, so scripts that work anywhere will
work consistently everywhere. Inspired by a similar change in
FreeBSD's sh (from 2010) - the original change in the other direction
came from FreeBSD as well.... Note that this does not in any way
add any kind of support for locales to sh (which is a whole different
problem.) (from kre)
ran; i.e. we want this to work:
$ cat succ1
#!/bin/sh
./succ2 6>out
$ cat succ2
#!/bin/sh
echo succ2 >&6
$ ./succ1
And this to fail:
$ cat fail1
#!/bin/sh
exec 6> out
echo "fail1" >&6
./fail2
exec 6>&-
$ cat fail2
#!obj.amd64/sh
echo "fail2" >&6
$ ./fail1
./fail2: 6: Bad file descriptor
XXX: Do we want a -k (keep flag on exec to make redirections not close-on-exec?
quotes ('_' and '.' do not need quoting) and never quote the '=' in
an assignment (or it would not be one.) From kre, with some refactoring
to be blamed to me.
a terminal. The Open Group notes this historic behavior and correctly
notes that it doesn't make much sense. Note also, that mv(1) has
always respected its '-i' regardless of whether the standard input is
a terminal.
From Timo Buhrmester.
disabling noexec, if the shell is interactive, each time that
a new command is about to be read. Also correct the -I
(ignoreeof) option so that it only applies to interactive shells,
as required by posix. (from kre)
and also added doc for some other stuff that was missing.
Take the opportunity to clean up the way the flags are set in the
man page, so every new flag doesn't have to be added 6 times!
(Some of the lists were different from others, in ordering, and
content, for no good reason at all.)
Make a few other cleanups ... Add text about AND-OR lists,
This can be also used to justify closing an open PR:
(that "sh -c 'command &&'" is not a syntax error...).
Add doc for -F, which should default to set if the shell somehow
gets compiled without DO_SHAREDVFORK defined, (to be committed
separately)
XXX: Consider disabling DO_SHAREDVFORK if SMALL is defined?
From kre
handle the token syntax it really should be handling (including
some that posix does not require, but is right anyway.) This is
quite similar to, and to some extent inspired by the way the FreeBSD
sh parser.c works, but the actual implementation is quite different.
(from kre)
For df -G, print the block and fragment size instead of the iosize
and the blocksize. If we need to print the iosize, it should be done
in a different field. Nevertheless printing the blocksize in the fragment
size field is just wrong.
XXX: pullup-6, pullup-7
Like GNU dd(1) similar operands, iflag and oflag allow specifying the
O_* flags given to open(2) for the input and the output file. The values
are comma-sepratated, lower-case, O_ prefix-stripped constants documented
in open(2).
Since iflag and oflag override default values, specifying oflag means
O_CREATE is not set by default and must be specified explicitely.
Some values do not make sense (e.g.: iflag=directory) but are still used
and will raise a warning. For oflag, values rdonly, rdwr and wronly are
filtered out with a warning (dd(1) attempts open(2) with O_RDWR and
then O_WRONLY on failure).
Specifying oflag=trunc along with (seek, oseek or conv=notrunc) is
contradictory and will raise an error.
iflag and oflag are disabled if building with -DMALLPROG
One of motivation of this change is to make the behavior of test(1)
-nt/ot with preserved copy (like cp -p) closer to the NetBSD 6.
Of course whether full timestamps are kept or not depends also on
underlying file system.
The ifdef added in mv(1) since existing ifdefs was our local change
to compile it on solaris (though I couldn't test it):
http://mail-index.netbsd.org/tech-userlevel/2014/11/28/msg008831.html
Rename the following reference documents to match their programs:
shell -> sh
viref -> vi
and rename the following to match their topic better:
ipctut -> sockets
ipc -> sockets-advanced
Also, the old "timed" and "timedop" docs are now ref5/timed and
ref8/timed respectively, as the first of these documented the
protocol.
Move all the reference manuals to subdirs of /usr/share/doc/reference.
We have subdirs ref1-ref9, corresponding to man page sections 1-9.
Everything that's the reference manual for a program (sections 1, 6,
8), C interface (sections 2, 3), driver or file system (section 4),
format or configuration (section 5), or kernel internal interface
(section 9) belongs in here.
Section 7 is a little less clear: some things that might go in section
7 if they were a man page aren't really reference manuals. So I'm only
putting things in reference section 7 that are (to me) clearly
reference material, rather than e.g. tutorials, guides, FAQs, etc.
This obviously leaves some room for debate, especially without first
editing the docs with this distinction in mind, but if people hate
what I've done things can always be moved again.
Note also that while roff macro man pages traditionally go in section
7, I have put all the roff documentation (macros, tools, etc.) in one
place in reference/ref1/roff. This will make it easier to find and
also easier to edit it into some kind of coherent form.
Update the <bsd.doc.mk> infrastructure, and update the docs to match
the new infrastructure.
- Build and install text, ps, pdf, and/or html, not roff sources.
- Don't wire the chapter numbers into the build system, or use them in
the installed pathnames. This didn't matter much when the docs were a
museum, but now that we're theoretically going to start maintaining
them again, we're going to add and remove documents periodically and
having the chapter numbers baked in creates a lot of thrashing for no
purpose.
- Specify the document name explicitly, rather than implicitly in a
path. Use this name (instead of other random strings) as the name
of the installed files.
- Specify the document section, which is the subdirectory of
/usr/share/doc to install into.
- Allow multiple subdocuments. (That is, multiple documents in one
output directory.)
- Enumerate the .png files groff emits along with html so they can be
installed.
- Remove assorted hand-rolled rules for running roff and roff widgetry
and add enough variable settings to make these unnecessary. This
includes support for
- explicit use of soelim
- refer
- tbl
- pic
- eqn
- Forcibly apply at least minimal amounts of sanity to certain
autogenerated roff files.
- Don't exclude USD.doc, SMM.doc, and PSD.doc directories from the
build, as they now actually do stuff.
Note: currently we can't generate pdf. This turns out to be a
nontrivial problem with no immediate solution forthcoming. So for now,
as a workaround, install compressed .ps as the printable form.
Evaluation of commands goes completely haywire if a file containing
a break/continue/return command outside its "intended" scope is sourced
using a dot command inside its "intended" scope. The main symptom is
not exiting from the sourced file when supposed to, leading to evaluation
of commands that were not supposed to be evaluated. A secondary symptom
is that these extra commands are not evaluated correctly, as some of them
are skipped. Some examples are listed in the How-To-Repeat section.
According to the POSIX standard, this is how it should work:
dot:
The shell shall execute commands from the file in the current
environment.
break:
The break utility shall exit from the smallest enclosing for, while,
or until loop, [...]
continue:
The continue utility shall return to the top of the smallest
enclosing for, while, or until loop, [...]
return:
The return utility shall cause the shell to stop executing
the current function or dot script. If the shell is not currently
executing a function or dot script, the results are unspecified.
It is clear that return should return from a sourced file, which
it does not do. Whether break and continue should work from the sourced
file might be debatable. Because the dot command says "in the current
environment", I'd say yes. In any case, it should not fail in weird
ways like it does now!
The problems occur with return (a) and break/continue (b) because:
1) dotcmd() does not record the function nesting level prior to
sourcing the file nor does it touch the loopnest variable,
leading to either
2 a) returncmd() being unable to detect that it should not set
evalskip to SKIPFUNC but SKIPFILE, or
b) breakcmd() setting evalskip to SKIPCONT or SKIPBREAK,
leading to
3) cmdloop() not detecting that it should skip the rest of
the file, due to only checking for SKIPFILE.
The result is that cmdloop() keeps executing lines from the file
whilst evalskip is set, which is the main symptom. Because
evalskip is checked in multiple places in eval.c, the secondary
symptom appears.
>How-To-Repeat:
Run the following script:
printf "break\necho break1; echo break2" >break
printf "continue\necho continue1; echo continue2" >continue
printf "return\necho return1; echo return2" >return
while true; do . ./break; done
for i in 1 2; do . ./continue; done
func() {
. ./return
}
func
No output should be produced, but instead this is the result:
break1
continue1
continue1
return1
The main symptom is evident from the unexpected output and the secondary
one from the fact that there are no lines with '2' in them.
>Fix:
Here is patch to src/bin/sh to fix the above problems. It keeps
track of the function nesting level at the beginning of a dot command
to enable the return command to work properly.
I also changed the undefined-by-standard functionality of the return
command when it's not in a dot command or function from (indirectly)
exiting the shell to being silently ignored. This was done because
the previous way has at least one bug: the shell exits without asking
for confirmation when there are stopped jobs.
Because I read the standard to mean that break and continue should have
an effect outside the sourced file, that's how I implemented it. For what
it's worth, this also seems to be what bash does. Also laziness, because
this way required no changes to loopnesting tracking. If this is not
wanted, it might make sense to move the nesting tracking to the inputfile
stack.
The patch also does some clean-up to reduce the amount of global
variables by moving the dotcmd() and the find_dot_file() functions from
main.c to eval.c and making in_function() a proper function.
We have written a diff to our ls(1) to recover the traditional behaviour
of -f implying -a. This change does not only accommodates POSIX.1
but also matches traditional UNIX.
OpenBSD commit message:
CVSROOT: /cvs
Module name: src
Changes by: sobrado@cvs.openbsd.org 2014/03/31 14:54:37
Modified files:
bin/ls : ls.1 ls.c
Log message:
restore the traditional behavior of -f implying -a; apparently Keith Bostic
forgot to restore it when the -f flag was put back on 2nd of September 1989,
after being removed on 16th of August as a consequence of issues getting it
working over NFS, so deviation from traditional UNIX behavior in all BSDs
looks like an historical accident; as a side effect, this change accommodates
behavior of this option to IEEE Std 1003.1-2008 (``POSIX.1'').
joint work with jmc@ (who found the inaccuracy in our implementation),
schwarze@ (who provided a detailed tracking of historical facts) and millert@
ok millert@, schwarze@
