Multiple arguments to atf_set are joined by spaces, there's no need for
an extra space.
The exit status on success must be 0, so don't ignore it.
Remove the unnecessary shell wrapper, as no redirection is going on.
the pathname matching tests ... this is not used in any way in a normal
test run, but if there are problems, one needs a mechanism to discover
whether the issue is with the code that creates the directory tree in the
test, or with the matching (either what the test expects to happen, or
what the shell actually does). The couple of lines being fixed here make
that mechanism work --- I have had this in my private sources for years
now, but never got around to committing it. It is time. NFC for any
normal test run.
to what the shell produces for tilde expansions, when the expansion
of the tilde prefix ends with '/' and the character immediately following
is another '/' - previously /bin/sh (and most other shells, but not all)
retained both slashes, and this test expected that behaviour.
No longer, now only one of the two will appear. Adjust the expected
test results accordingly, and add an extra loop iteration to make sure
this is thoroughly tested (one more tilde expansion value).
Also, add two new test cases that test for the new (explicit - though
it was always stated this way, but not as explicitly) that if HOME is
an empty string (not unset - that remains an unspecified case, as it
was) then the expansion of ~ must generate "", and not nothing.
The current test was unable to distinguish those two, since it
simply looked for characters in the output, so add a new test cases
explicitly to test for this particular case. This also means (also
previously in the standard, but not as explicitly) that shells are
not permitted to decide "The ~ expansion produces an empty string, that
is weird, let's generate something else instead" which some shells were
doing.
/bin/sh's processing of here doc expansions has changed. Now it happens
in the context of the parent shell, so side effects are visible there,
just like all other redirection expansions.
We need to stop testing that that doesn't happen, and instead test
that it does. This is that change.
Add another test case which is testing exactly the example from the PR
(well, with a different exit status, 1 is too generic and could happen
by accident) to make sure we don't reintroduce that bug sometime.
which allows the atf_check to fail without causing the test to fail
(unless this is the (very) last command in the test case, in which case it
will fail with what can be interpreted as an internal error)/
Check for this failing and explicitly atf_fail whwn it does.
changed to exit(1) when it detects a write error to stdout.
Running printf with stdout closed is guaranteed to generate such a
condition.
Until the previous commit, while the test case was actually failing
(stderr was expected to be empty. and was not) this was unnoticed.
We don't want the output (obviously), we also don't want the error
message, so just direct the latter to /dev/null.
atf_check and simply expect it to work - the shell is permitted to,
and our shell currently does, run all commands in a pipeline in subshell
environments - when atf_check attempts to exit to indicate failure, it
only exits from that subshell, and the rest of the test continues, usually
to indicate success
Instead, when it is necessary (or just convenient) to pipe into atf_check
check the exit status of the pipeline (if atf_check is not last, which it
would usually be, then we would need the pipefail option set - there are
currently no such cases), and explicitly fail if atf_check did not exit(0).
of a few bizarre here document usages, such as
: << do | for x in xxx
do
do echo $x
done
which should work, but never have done. (See the source for others).
This test case will currently fail, until the bug is fixed
(already done, to be committed very soon).
The first verifies that echo exits >0 when it encounters an I/O error on
its output (this part would have succeeded for a long time). It also
verifies the POSIX requirement that when most standard utilities (or
perhaps many rather than most) exit(>0) they must write a message to stderr.
Our sh's built in echo did not do that (nor does /bin/echo but that's not
relevant here).
The second demonstrates (on an unfixed built-in echo) a bug reported in
private e-mail by Oguz <oguzismailuysal@gmail.com> where once an instance of
the built-in echo has detected an I/O error, all later invocations of
the built-in echo, with no I/O errors of their own, also exit(1) (the error
status on stdout is not cleared, each echo sees the "I/O error occurred" and
does exit(1)).
In this second sub-test, the "2>&-" on the first echo command is simply
an artifact caused by the test harness - the "check" function verifies
that exit((>0) requires a message on stderr (and vice versa), but that
only applies to most (or many) utilities, echo is one, but sh is not.
In the second test, the exit status comes from sh - sh is permitted to
write to stderr (via the echo command it runs in this case) and still
exit(0). But the check function in the test does not understand that
subtlety. So, we simply suppress the stderr message by closing stderr
(the first of these two new sub-tests has verified that the message exists)..
Add a sh ATF test to demonstrate a bug in the way that \0 characters
are dropped from scripts. This test will eventually be extended to
test other potential sh script input related issues.
When initially committed, this test should fail. It should succeed
when the fix for the PR is committed (soon).
Nb: this tests only the \0 related issues from the PR, the MSAN
detected uninitialised variable (struct field) can only be detected
by MSAN, as it has no visible impact on the operation of the shell
when running on any real (or even emulated) hardware.
(It will, however, also be fixed).
(if date fails, because of bad or out of range input
to parsedate, it cannot be expected to output the
expected result).
Remove the atf-expect-fail now that parsedate() should
detect overflows, and fail, causing date to fail, rather
than simply producing bogus results.
in the past, but managed to re-surface...
The expression "${0+\}}" should expand to "}" not "\}"
Almost all other shells handle it that way (incl FreeBSD & dash).
Issue pointed out by Martijn Dekker.
Add ATF sub-tests for the 4 old var expand operators (${var+word}
${var-word} ${var-word} and ${var?word} - including the forms
with the ':' included) and amongst those tests include test cases
for this issue, so if the bug tries to appear again, we can squash
it quicker. (The newer pattern matching operators are already
well tested as part of testing patterns.)
Fix handling of "$@" (that is, double quoted dollar at), when it
appears in a string which will be subject to field splitting.
Eg:
${0+"$@" }
More common usages, like the simple "$@" or ${0+"$@"} end up
being entirely quoted, so no field splitting happens, and the
problem was avoided.
See the PR for more details.
This ends up making a bunch of old hack code (and some that was
relatively new) vanish - for now it is just #if 0'd or commented out.
Cleanups of that stuff will happen later.
That some of the worst $@ hacks are now gone does not mean that processing
of "$@" does not retain a very special place in every hackers heart.
RIP extreme ugliness - long live the merely ordinary ugly.
Added a new bin/sh ATF test case to verify that all this remains fixed.
bug (and its fix (depending upon whether the test is run against
an unfixed, or fixed, shell).
An obvious indication of the failure is the following (one of the
new sub-tests)
p=A
cat <<EOF
${p+\%$p\%}
${p+%$p%}
EOF
which should output
\%A\%
%A%
as a here doc is treated as a double quoted string, except
that the " character is just a character. In such a string,
the \ is only an escape character when the following character
is special, otherwise it represents a literal \ (which is the
case here).
An unfixed shell will omit the backslashes in the output.
It gets even more wrong if the % chars are replaced by "
(double quote) chars, which should make no difference, other
than the corresponding change, in the output. But doesn't
(it doesn't even produce output broken in a similar way).
This one is a harder case to be specific about however,
as while the fixed (and expected in the test) output is what
is technically correct, only a few shells actually produce
it, most generate something different (but not all the same.)
when the locale is ru_RU.UTF-8 (which has ',' as its decimal radix char).
Inspired by a problem with rc.subr experiened by ru_RU.UTF-8 reported on
netbsd-users. These are in the "fraction" test case.
While here, add some more (sub-)tests of invalid input, to make sure they
fail (in the "nonnumeric" test case), and for fun, a couple of subtests
testing hex input fractional delays (in the "hex" test case).
including testing correct handling of error cases.
One of these new tests located a bug which will be fixed
within minutes of this commit ...
While doing this, move the test that was used only in the
echo builtin test case (testing if the NetBSD shell was being
tested) into a utility function, and also call it in the eval
test, so NetBSD specific behaviour of what is unspecified in
the standard can be checked to ensure it is not accidentally
altered.
In particular, add a readonly test to detect the bug that was just fixed...
(but there is more than that one added here).
Also, allow zsh to run more of these tests than it did, what is a builtin
command most places (including in POSIX) can be a reserved word in zsh!
utilities. That is, not the low level ones that look like syntax,
but aren't: break/continue/return; not those which are really
just external programs that are built in for efficiency (printf,
test, and kill - though kill really needs to be built in) - those
should all have separate test programs (there is a test here for the
built-in echo, as that is an entirely different thing to /bin/echo);
and also not those for which there are other tests because of the
nature of the built-in (like exit, wait, shift, ...). Lastly not
"times" either as that just seems to be too hard to test rationally.
There is a test (well, framework) for ulimit and there's also t_ulimit.sh
one of those should go, but I am not yet sure which is the best way
to reconcile things.
Note: many (in fact) most of the test cases added here are either
entirely empty (no tests at all, beyond testing that the built-in is
in fact a shell built-in) or only very rudimentary tests - assistance
in fleshing those out would be welcome (the boilerplate is all here,
all that is needed is some actual tests to run...)
(and a few more subtests in an existing test case).
The two new test cases currently fail, because of issues with
expanding "${1+$@}" which will (hopefully) be fixed soon.
(This looks to have been broken sometime during 2013 ... then I
made it even worse with some of the parser changes a while ago,
though the end result is that it appears less broken than it
really is.)
includion a test to make sure that the file doesn't get truncated.
Add new subtests to the "incorrect redirections" test case, to
validate correct behaviour of the shell when redirections fail in
various scenarios, including when the redirect is the whole command.
More along these lines are really needed, but this is better than nothing.
All the added tests pass on the /bin/sh currently in netbsd HEAD.
was not being done properly (too many different code paths inside sh)
so add even more subtests to the case_matching test case to verify
that all (that I can think of for now anyway) get fixed when this
gets cleaned up. The case_matching test case still fails, but now
6 of its subtests should fail (until sh is corrected ... soon).
Martijn Dekker (private e-mail.) Variable expansions that are
double quoted result in literal characters (nver pattern matching
meta chars.) This includes '\' (that one was the bug.)
[On the other hand, a variable in a case pattern expansion that is
unquoted, produces a pattern, and in that the \ character can be
used to excape other pattern meta-chars (and itself.]
This addition will cause the case_matching test case to fail (two
of the newly added sub-tests fail) until fixes to /bin/sh are made.
(That is comiung soon, the code exists already.)
In the filename expansion test, don't insist on testing cases
of [^a] anything not an a) (etc) - while almost all shells allow
^ there (as in regular expressions) this is not standard sh/glob syntax
(! is used for that, not ^). Use of ^ as first char after '[' in a []
match has unspecified behaviour. So, check if the shell being
tested supports ^ used this way before running the tests of ^ in [].
This makes no difference at all when testing the NetBSD sh which
does allow ^ for that use case.
Fix an obvious (in hindsight) stupidity in the case pattern tests
(a cut/paste/forget-to-fix type error).
Many cleanups, including some additional sub-tests, for the var
substring matching (just a couple of actual fixes to the tests).
(This was the last added, and least polished part - still is.)
This makes no difference to the filename expansion tests (11 of 167
still fail), the fix reduces the failing case match tests from
14 of 261 to 13 of 261. For var substring tests, the failure
has changed from 6 of 87 to 8 of 99. (Some) Fixes to sh will appear
soon.
For comparison, using this version of the test, dash passes all case
and var substring tests, and (aside from the [^...] tests which it
does not implement) fails 4 of the filename tests. bash fails one
case test (a bash oddity in interpretation of the spec, which is unique
to bash amongst shells I have tested) and one filename expansion (all
var substring tests pass). Other shells are much worse (though the
freebsd shell results were coloured by the version of their shell I
tested still having the [[:xxx:]] always matches '[' bug, which is
fixed in later versions of their shell)
patterns, and variable expansion substring matching)
Currently (2018-07-10) all 3 sub-tests fail (sh bugs...)
Expect to see 14 (of 261) case matching sub-tests fail, 11 (of 167) filename
expansion (glob) sub-tests fail, and 6 (of 87) var substring sub-tests fail.
Also expect those numbers to reduce as sh bugs are fixed.
Assert that -9223372036854775808 % -1 and -9223372036854775808 / -1 return
message about overflow / underflow detected.
These tests pass correctly.
Sponsored by <The NetBSD Foundation>
Currently this test case will fail, a fix is coming soon (not worth
marking this as an expected failure.)
This test case and the initial bug report comes from
Martijn Dekker's modernish (shell/test set).
Currently (or when testing any shell that does not support -X) the
test will be skipped (also for [m]ksh (but not ksh93 etc) where there
is an absurdly badly named -X option, skip the new test for them as well.)
When -X appears in /bin/sh, this will verify that it is probably working
(the test is MUCH more gruelling than any rational use of -X would be.)
rillig