use -j; all make's in a recursive build cooperate to limit the total
number of jobs, using a token-passing scheme.
The current token passing algorithm is similar to the one implemented
by gmake; there is a single pipe which is inherited through the entire
process hierarchy; tokens are obtained by reading a byte from the
"read end" of the pipe, and are returned by writing them to the "write
end". This exact algorithm is likely to change in the future.
Implementation details:
- Use the new trace facility to allow measurement of the
effectiveness of different token-passing schemes
- Get a token in MakeStartJobs(), return it in Make_Update()
- Eliminate Job_Full() and the jobFull global since they are
redundant with token system.
- Add an "internal" -J option (to pass the token pipe fd's down to
submakes) and a -T option for tracing.
- Change how compatMake is forced so that -j means something when
inherited by submakes.
- When waiting for a token, poll the token-passing pipe as well as
the output pipes of existing jobs.
poll() for parallel make:
- Make the poll() code behave more like the select() code: sleep for
a bit waiting for output rather than busy-wait (eww).
- Install a no-op SIGCHLD handler so that poll/select wake up early
(with -1/EINTR) when a child exits.
- Change the default sleep time from 500ms to 5 seconds since we now
wake up promptly when a child exits.
try to open tfile many times, passing bad strings to mkstemp
2. remove extra semicolon after the MESSAGE macro
3. more error checking
4. be more careful about setting things to NULL after freeing.
5. fix a comment that does not apply anymore
arguments names on one function being swapped (by a previous author).
Do not do any duplicate suppression when a source list is created. Instead:
* OP_MADE protects against trying to make the source multiple times.
* A new OP_MARK flag is introduced to suppress duplicates while expanding
the .ALLSRC variable and .USE targets.
This turns the O(n^2) insertion into O(n) in most cases.
This is tested with a `make build' and some special test cases.
To avoid that, we now do ::[+?!]*= but the SysV = modifier can
conflict with any new modifier. At there are currently no Makefiles
in our tree that use ${FOO::=bar}
problems such as using modifiers on .for loop iterators derived from
local variables (eg .TARGET).
Unless the variable already exists in a global context, these assignments are
local to the current context (this is usually what is wanted).
is an empty list, use DEFSYSPATH.
The current behaviour may have been useful when DEFSYSMK was an
absolute path (hasn't been the case since 1996), but right now
make -m /no/such/dir will fail to find sys.mk and die.
These allow some very useful magic in the makefiles.
The comment in var.c describing their behaviour is mostly lifted
from ODE make, but the implementation of the modifiers themselves
is quite different (much simpler) due to divergence of our code base.
Firstly, we ignore getenv("PWD") if MAKEOBJDIRPREFIX is set so that we always
get the same value for .CURDIR regardless of how make was invoked.
Second, when executing a command we check if it is ${.MAKE} or ${.MAKE:T}
without a preceeding chdir, if so we insert a chdir(${.CURDIR}) so that
the Makefile will be found by the child make. Note that this behaviour is
dissabled if MAKEOBJDIRPREFIX is not set or if NOCHECKMAKECHDIR is set.
See the comments in main.c for more detail.
With these two changes, one can successfully build usr/src using MAKEOBJDIRPREFIX
allowing the src to be mounted from a CD-ROM.
someone wants to do freaky stuff with $MAKEOBJDIR.
Initialize the Var system and set .CURDIR, MACHINE and MACHINE_ARCH
before chdir_verify_path() is called.
list of target nodes returned by Make_ExpandUse(). We have to search the
cohorts explicitly while iterating through the tree. So, tweak CompatMake()
to do this.
can (i.e., everything except environment variables, which aren't
stored in hash tables).
While we're here, inline the body of VarDelete into Var_Delete since
it's the only caller and it's just simpler that way when v->name can
share storage with the hash entry and may not need to be freed
separately.
Speeds up the infamous libc build benchhmark maybe 1% on PIII, 4% on
alpha pc164
Suggested by Perry Metzger.
Build a list of `cohorts' as before, but do *not* link each one into all the
parent nodes; instead, copy the `cohort' lists into the stream of targets to
be built inside Make_ExpandUse(). Also do the attribute propagation as a
separate pass after parsing.
This eliminates several O(n^2) algorithms.
passing in the LstNode of the child being inspected. Shaves off another few %,
particularly when there are long child lists containing $ expansions (e.g. in
libc).
in the presence of .PATH directives by specifying:
.PATH: .DOTLAST
This will be used to fixup the build system to work with both crypto-us
and crypto-intl sub-trees.
Make(1) changes by Christos Zoulas, after much badgering by me :-)
parents would be get remade, even if children were not really updated
by the commands executed for them. It also makes all the children have
the real modification time set if possible, so it should fix some other
timing weirdnesses...
- collapse childMade and make fields into flags and convert them to bits
CHILDMADE and REMAKE
- introduce FORCE flag that gets set in all the parents of a child that
has no sources and does not exist.
- set oodate if the FORCE flag is set, and not if CHILDMADE
- centralize the RECHECK into Make_Recheck() and use this in make.c and compat.c
- use Make_TimeStamp for all child -> parent timestamp propagations
a union mount.
eg.
src: FORCE
FORCE is a fake target that does not have sources. When FORCE is
considered made it gets updated with the current timestamp. If the
directory happens to have the same timestamp too, then it will not
be made because it is considered to be up-to-date with respect to
the child. This can happen because the time resolution is only in
seconds. It is more likely to happen on a union filesystem where
the timestamps take longer to update.
The fix is to consider the parent unmade when children have been
updated.
the previous garbage bytes may be read.
Example: "make n" --- target = n, suffix = .ln
Changing interface of SuffSuffIsSuffix() is required to fix this bug.
This revealed another long standing problem with pmake's port to bsd.
.MAKE was not set as the manual page states. Set it and remove another
typo in my last commit.
Unfortunately this revealed a deeper problem with the brk_string code.
To fix it:
- remove sharing of the buffer between brk_string invocations
- change the semantics of brk_string so that the argument array
starts with 0, and return the buffer where the strings are
stored
Actually there were two bugs:
- Add REG_NOTBOL after the first substitution.
- Handle the rm_so == rm_eo == 0 case, where in a substitution such
as 's/bzzzt/z*/g' the first time z* matches nothing.
directory specified, and add it to sysIncPath only if it exists.
However, afterwards make tested for the presence of a -m option by
checking to see if sysIncPath was an empty list, and assumed that
the -m option was not used if it was empty. This obviously breaks
if -m specified a non-existent directory. So I have added a flag
that is set if the -m option is used, and I test that instead.
Make used to only use the search path for nodes that were pure
sources (not targets of other sources). This has been corrected
and now gnu-autoconf generated Makefiles work in directories other
than the source one.
- Suffix transformation rescanning:
Suffix transformations (.c.o:; cc ...) were only recognized in
the past when both suffixes were members of the suffix list.
Thus a sequence like:
.z.b:
echo ${.TARGET}
.SUFFIXES: .z
would cause .z.b: to be inserted as a regular target (and the main
target in this case). Other make programs always add rules that
start with a period in the transformation list and never consider
them as targets. We cannot do that (consider .depend files) so we
resort to scanning the list of the current targets every time a
suffix gets added, and we mutate existing targets that are now
valid transformation rules into transformation rules. If the
transformed target was also the main target, we set the main target
to be the next target in the targets list.
I.e. if you had a line in your Makefile:
../foo.o: foo.c
`..' would be added in the search path. The addition of such paths has
been now disabled. If a pathname contains a slash, then the directory
where such a file is found is not added to the search path. Of course
this eliminates most (all?) use of this function.
- Fix globbing so that patterns that don't have a matching number of [] or {}
don't get expanded. (before the [ case got expanded to nothing!) This is
disabled.
1. ${.*} variables did not get expanded in dependencies.
2. expanded ${.*} variables in .USE dependencies can cause tree
restructuring; handle it.
3. in compat mode, expand .USE before evaluating the list of targets,
instead of doing .USE expansions on demand, because they can cause
tree restructuring.
- fix the variable substitution code in make [PR/2748]
1. change s/a/b/ so that it substitutes the first occurance of the
pattern on each word, not only the first word.
2. add flag '1' to the variable substitution so that the substitutions
get performed only once.
***THIS IS AN INCOMPATIBLE CHANGE!***
Unfortunately there was no way to make things consistent without
modifying the current behavior. Fortunately none of our Makefiles
depended on this.
OLD:
VAR = aa1 aa2 aa3 aa4
S/a/b/ = ba1 aa2 aa3 aa4
S/a/b/g = bb1 bb2 bb3 bb4
NEW:
VAR = aa1 aa2 aa3 aa4
S/a/b/ = ba1 ba2 ba3 ba4
S/a/b/1 = ba1 aa2 aa3 aa4
S/a/b/g = bb1 bb2 bb3 bb4
S/a/b/1g = bb1 aa2 aa3 aa4
- add regexp variable substitution via 'C/foo/bar/' [PR/2752]
- add variable quoting via the ${VAR:Q} modifier. This is useful when running
recursive invocations of make(1):
make VAR=${VAR:Q}
will always work... (This may prove useful in the kernel builds...) [PR/2981]
referenced only by their basename and not by their full pathname. This
breaks when .PATH or MAKEOBJDIR are used. There might be Makefiles around
that try to work around this bug by prepending ${.CURDIR} to the sources,
and they should be found and fixed. Also a lot of the gunk in suff.c that
was attempting to work around the same problem could be removed.
arch.c: Don't require ranlib stuff. Not everybody has it.
dir.c: SunOS-4 != Solaris; change #ifdef sun to #if sun && !__svr4__
job.c, compat.c: Don't use 'union wait', use int and the W*() macros.
main.c: Check for uname() == -1; some unames return > 0...
util.c, job.c: Add signal() with BSD semantics for svr4, don't use bsd
sigmask and friends.
- ${.PREFIX} should never contain a full pathname
- Fixed gcc -Wall warnings
Major:
- compatMake is now FALSE. This means that we are now running in
full pmake mode:
* rules on dependency lines can be executed in parallel and or
out of sequence:
foo: bar baz
can fire the rule for baz before the rule for bar is fired.
To enforce bar to be fired before baz, another rule needs to be
added. [bar: baz]
* adjacent shell commands in a target are now executed by a single
invocation of the shell, not one invocation of the shell per line
(compatMake can be turned off using the -B flag)
- The -j flag now works... I.e. make -j 4 will fork up to four jobs in
parallel when it can. The target name is printed before each burst
of output caused by the target execution as '--- target ---', when j > 1
- I have changed all the Makefiles so that they work with make -j N, and
I have tested the whole netbsd by:
'make -j 4 cleandir; make -j 4 depend; make -j 4; make -j 4 install'
- I have not compiled or tested this version of make with -DREMOTE.
Dependency rules with `=' in the lhs are parsed as variable assignments.
E.g., the following Makefile fails:
A=a b c d
all: $(A:%=%b)
$(A:%=%b):
@echo $@
var.c: Minor memory leak plugged.
suff.c: Don't add extra sources on the null suffix if it has dependency
lines or commands attached to it [POSIX says so]
This is not the perfect fix. I think that the whole __SYMDEF
checking code should be removed. [I should also teach myself
how to deal with the PR stuff to mark this closed :-)]
1. Honor environment variable $MACHINE if set over uname.machine
2. archives with :: are always out of date, even when they have no children.
3. VAR= a b c # comment, gets the trailing blanks trimmed, unless
escaped by \. I'll have to read the posix manul to make sure that it
is ok to handle escapes here.
sommerfeld