NetBSD/doc/BUILDING.mdoc

.\"	$NetBSD: BUILDING.mdoc,v 1.149 2023/07/22 18:50:04 lukem Exp $
.\"
.\" Copyright (c) 2001-2023 The NetBSD Foundation, Inc.
.\" All rights reserved.
.\"
.\" This code is derived from software contributed to The NetBSD Foundation
.\" by Todd Vierling and Luke Mewburn.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
.\" PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.\" NOTE: After changing this file, run "make regen" in the src/doc
.\" directory, and check in both src/BUILDING and src/doc/BUILDING.mdoc.
.\"
.\" Toolchain prefix for commands
.ds toolprefix nb
.
.Dd July 21, 2023
.Dt BUILDING 8
.Os NetBSD
.\" turn off hyphenation
.hym 999
.
.\" if this file is processed with real roff (doc.tmac)
.\" now that the tmac files have been lazily read,
.\" we can override ``quotes'' with "quotes" for ascii output
.ie '\*[Lq]'``' \{\
.ds Lq \*q
.ds Rq \*q
.\}
.\" and disable hyphenation
.nr hyphen-flags 0
.
.Sh NAME
.
.Nm BUILDING
.Nd Procedure for building
.Nx
from source code
.
.Sh REQUIREMENTS
.
.Nx
is designed to be buildable on most POSIX-compliant host systems.
The basic build procedure is the same whether compiling
.Em natively
(on the same
.Nx
architecture) or
.Em cross compiling
(on another architecture or OS).
.Pp
This source tree contains the
.Sy build.sh
shell script which supports both native and cross builds of
.Nx .
.Pp
This source tree contains a special subtree,
.Dq tools ,
which uses the host system to create a build toolchain for the target
architecture.
The host system must have at least C and C++
compilers in order to create the toolchain
.Xr ( make 1
is not required); all other tools
(including
.Xr make 1
as
.Sy \*[toolprefix]make )
are created as part of the
.Nx
build process.
(See the
.Sx Environment variables
section below if you need
to override or manually select your compilers.)
.Pp
.Em Note :
Within this document, cross-references to manual pages are to the
.Nx
manual pages, not the host system manual pages.
The
.Xr mdoc 7
source
to the
.Nx
manual pages can be found within the source tree,
and these and can be formatted with
.Xr mandoc 1
or
.Xr nroff 1
if those are available on the host system.
The
.Nx
manual pages are also available at
.Lk https://man.netbsd.org
.
.Sh FILES
.
.Ss Source tree layout
.
.Bl -tag -width "BUILDING.mdoc"
.
.It Pa BUILDING
This document (in plaintext).
Generated from
.Pa doc/BUILDING.mdoc .
.
.It Pa Makefile
The main Makefile for
.Nx ;
should only be run for native builds with an appropriately up-to-date
version of
.Nx
.Xr make 1 .
Intended for expert use with knowledge of its shortcomings,
it has been superseded by the
.Sy build.sh
shell script as the recommended means for building
.Nx .
.
.It Pa UPDATING
Special notes for updating from an earlier revision of
.Nx .
It is important to read this file before every build of an updated
source tree.
.
.It Pa build.sh
Bourne-compatible shell script used for building the host build tools
and the
.Nx
system from scratch.
Can be used for both native and cross builds, and should be used instead of
.Xr make 1
as it performs additional checks to prevent common issues going undetected, such
as building with an outdated version of
.Xr make 1 .
.
.It Pa crypto/dist/ , dist/ , gnu/dist/
Sources imported verbatim from third parties, without mangling the
existing build structure.
Other source trees in
.Pa bin
through
.Pa usr.sbin
use the
.Nx
.Xr make 1
.Dq reachover
Makefile semantics when building these programs for a native host.
.
.It Pa distrib/ , etc/
Sources for items used when making a full release snapshot, such as
files installed in
.Sy DESTDIR Ns Pa /etc
on the destination system, boot media, and release notes.
.
.It Pa doc/BUILDING.mdoc
The source to this document, in
.Xr mdoc 7
format.
Used to generate
.Pa BUILDING .
.
.It Pa external/ , sys/external/
Sources and build infrastructure for components imported (mostly) unchanged
from upstream maintainers, sorted by applicable license.
This is (slowly) replacing the
.Pa crypto/dist ,
.Pa dist ,
and
.Pa gnu/dist
directories.
.
.It Pa external/mit/xorg/
.Dq Reachover
build structure for modular Xorg; the source is in
.Sy X11SRCDIR .
.
.It Pa mk.conf
Optional source tree specific
.Xr mk.conf 5 ,
used (if present) instead of
.Pa /etc/mk.conf
unless
.Sy MAKECONF
is defined.
.Pp
.Em Note :
Not part of the
.Nx
source repository.
.
.It Pa regress/ , tests/
Regression test harness.
Can be cross-compiled, but only run natively.
.Pa tests/
uses the
.Xr atf 7
test framework;
.Pa regress/
contains older tests that have not yet been migrated to
.Xr atf 7 .
.
.It Pa sys/
.Nx
kernel sources.
.
.It Pa tools/
.Dq Reachover
build structure for the host build tools.
This has a special method of determining out-of-date status.
.
.It Pa tools/compat/README
Special notes for cross-hosting a NetBSD build on non-NetBSD platforms.
.
.It Other directories including Pa bin/ ... usr.sbin/
Sources to the
.Nx
userland (non-kernel) programs.
If any of these directories are missing, they will be skipped during the build.
.
.El
.
.Ss Build tree layout
.
The
.Nx
build tree is described in
.Xr hier 7
(whose
.Xr mdoc 7
source is in
.Pa share/man/man7/hier.7 ) ,
and the release layout is described in
.Xr release 7
(whose
.Xr mdoc 7
source is in
.Pa share/man/man7/release.7 ) .
.
.Sh CONFIGURATION
.
.Ss Environment variables
.
.de DFLT
.Pp
.Em Default :
..
.de DFLTn
.DFLT
.Dq no .
..
.de DFLTu
.DFLT
Unset.
..
.de DFLTy
.DFLT
.Dq yes .
..
.de NODEF
.Pp
Forced to
.Dq no
if
.Sy \\$*
is defined,
usually in the Makefile before any
.Xr make 1
.Cm \&.include
directives.
..
.de NOVAR
.Pp
Forced to
.Dq no
if
.Sy \\$* .
..
.de YorN
Can be set to
.Dq yes
or
.Dq no .
..
.
Several environment variables control the behaviour of
.Nx
builds.
.
.Bl -tag -width 14n
.
.\" These entries are sorted alphabetically.
.
.It Sy HOST_CC
Path name to C compiler used to create the toolchain.
.DFLT
.Dq cc .
.
.It Sy HOST_CFLAGS
Flags passed to the host C compiler.
.DFLT
.Dq \&-O .
.
.It Sy HOST_CPPFLAGS
Flags passed to the host C/C++ pre-processor.
.DFLTu
.
.It Sy HOST_CXX
Path name to C++ compiler used to create the toolchain.
.DFLT
Unset, but defaults to
.Dq c++
where required.
.
.It Sy HOST_CXXFLAGS
Flags passed to the host C++ compiler.
.DFLTu
.
.It Sy HOST_SH
Path name to a shell available on the host system
and suitable for use during the build.
The
.Nx
build system requires a modern Bourne-like shell
with POSIX-compliant features,
and also requires support for the
.Dq local
keyword to declare local variables in shell functions
(which is a widely-implemented but non-standardised feature).
.Pp
Depending on the host system, a suitable shell may be
.Pa /bin/sh ,
.Pa /usr/xpg4/bin/sh ,
.Pa /bin/ksh
(provided it is a variant of ksh that supports the
.Dq local
keyword,
such as ksh88, but not ksh93),
or
.Pa /usr/local/bin/bash .
.Pp
Most parts of the build require
.Sy HOST_SH
to be an absolute path; however,
.Sy build.sh
allows it to be a simple command name, which will be converted
to an absolute path by searching the
.Ev PATH .
.DFLT
.Dq sh .
.
.It Sy INSTALLBOOT_UBOOT_PATHS
A colon-separated list of search paths used by
.Xr installboot 8
to find U-Boot packages.
.DFLTu
.
.It Sy MACHINE
Machine type, e.g.,
.Dq macppc .
.DFLTu
.
.It Sy MACHINE_ARCH
Machine architecture, e.g.,
.Dq powerpc .
.DFLTu
.
.It Sy MAKE
Path name to invoke
.Xr make 1
as.
.DFLT
.Dq make .
.
.It Sy MAKECONF
The name of the
.Xr make 1
configuration file.
See
.Sx make variables
and
.Xr mk.conf 5 .
.Pp
.Em Note :
Only settable in the process environment.
.DFLT
.Dq Pa /etc/mk.conf ,
although
.Sy build.sh
will set the default to the full path to
.Pa mk.conf
if the latter is present in the same directory as
.Sy build.sh .
.
.It Sy MAKEFLAGS
Flags to invoke
.Xr make 1
with.
.Pp
.Em Note :
.Sy build.sh
ignores the value of
.Sy MAKEFLAGS
passed in the environment, but allows
.Sy MAKEFLAGS
to be set via the
.Fl V
option.
.DFLT
.Dq \&-X
on systems with a small
.Dv ARG_MAX
(Cygwin, Darwin, FreeBSD);
otherwise unset.
.
.It Sy MAKEOBJDIR
Directory to use as the
.Sy .OBJDIR
for the current directory.
The value is subjected to variable expansion by
.Xr make 1 .
Typical usage is to set this variable to a value involving the use of
.Sq ${.CURDIR:S...}
or
.Sq ${.CURDIR:C...} ,
to derive the value of
.Sy .OBJDIR
from the value of
.Sy .CURDIR .
Used only if
.Sy MAKEOBJDIRPREFIX
is not defined.
.Pp
.Em Note :
.Sy MAKEOBJDIR
can be provided only in the environment or via the
.Fl O
flag of
.Sy build.sh ;
it cannot usefully be set inside a Makefile, including in
.Xr mk.conf 5
or
.Sy MAKECONF .
.DFLTu
.
.It Sy MAKEOBJDIRPREFIX
Top level directory of the object directory tree.
The value is subjected to variable expansion by
.Xr make 1 .
.Sy build.sh
will create the
${MAKEOBJDIRPREFIX}
directory if necessary, but if
.Xr make 1
is used without
.Sy build.sh ,
then rules in
.Aq bsd.obj.mk
will abort the build if the
${MAKEOBJDIRPREFIX}
directory does not exist.
If the value is defined and valid, then
${MAKEOBJDIRPREFIX}/${.CURDIR}
is used as the
.Sy .OBJDIR
for the current directory.
The current directory may be read only.
.Pp
.Em Note :
.Sy MAKEOBJDIRPREFIX
can be provided only in the environment or via the
.Fl M
flag of
.Sy build.sh ;
it cannot usefully be set inside a Makefile, including in
.Xr mk.conf 5
or
.Sy MAKECONF .
.DFLTu
.
.It Ev TMPDIR
Top-level directory to store temporary directories used by
.Sy build.sh
before paths to other directories such as
.Sy .OBJDIR
can be determined.
.Pp
.Em Note :
Must support execution of binaries.
I.e., without
.Xr mount 8 Ns 's
.Fl o
.Cm noexec
option.
.DFLT
.Dq Pa /tmp .
.
.El
.
.Ss make variables
.
Variables that control the behavior of
.Nx
builds are documented in
.Xr mk.conf 5
(whose
.Xr mdoc 7
source is in
.Pa share/man/man5/mk.conf.5 ) .
.Pp
Unless otherwise specified, these variables may be set in
either the process environment or the
.Xr make 1
configuration file
.Xr mk.conf 5
specified by
.Sy MAKECONF .
.Pp
.Em Note :
Variables set in the environment, either directly or via
.Sy build.sh
options to set specific values in the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script do not override variables set in the
.Xr mk.conf 5
file.
To allow variables in
.Xr mk.conf 5
to be overridden by the environment or
.Sy build.sh
options, define the variables using the
.Dq Li ?=
.Xr make 1
variable assignment operator.
For example,
.Bd -unfilled -offset indent
.Li MAKEVERBOSE?=1
.Ed
.Pp
The supported
.Xr mk.conf 5
make variables are:
.\" Keep in sync with share/man/man5/mk.conf.5 section NETBSD SYSTEM VARIABLES
.Bd -ragged -offset indent
.Sy BSDOBJDIR ,
.Sy BSDSRCDIR ,
.Sy BUILD ,
.Sy BUILDID ,
.Sy BUILDINFO ,
.Sy BUILDSEED ,
.Sy CDEXTRA ,
.Sy CONFIGOPTS ,
.Sy COPTS ,
.Sy CPUFLAGS ,
.Sy DESTDIR ,
.Sy EXTERNAL_TOOLCHAIN ,
.Sy INSTALLBOOT_BOARDS ,
.Sy INSTALLWORLDDIR ,
.Sy KERNARCHDIR ,
.Sy KERNCONFDIR ,
.Sy KERNEL_DIR ,
.Sy KERNOBJDIR ,
.Sy KERNSRCDIR ,
.Sy LOCALTIME ,
.Sy MAKEVERBOSE ,
.Sy MKAMDGPUFIRMWARE ,
.Sy MKARGON2 ,
.Sy MKARZERO ,
.Sy MKATF ,
.Sy MKBINUTILS ,
.Sy MKBSDGREP ,
.Sy MKBSDTAR ,
.Sy MKCATPAGES ,
.Sy MKCLEANSRC ,
.Sy MKCLEANVERIFY ,
.Sy MKCOMPAT ,
.Sy MKCOMPATMODULES ,
.Sy MKCOMPATTESTS ,
.Sy MKCOMPATX11 ,
.Sy MKCOMPLEX ,
.Sy MKCROSSGDB ,
.Sy MKCTF ,
.Sy MKCVS ,
.Sy MKCXX ,
.Sy MKDEBUG ,
.Sy MKDEBUGKERNEL ,
.Sy MKDEBUGLIB ,
.Sy MKDEBUGTOOLS ,
.Sy MKDEPINCLUDES ,
.Sy MKDOC ,
.Sy MKDTB ,
.Sy MKDTC ,
.Sy MKDTRACE ,
.Sy MKDYNAMICROOT ,
.Sy MKFIRMWARE ,
.Sy MKGCC ,
.Sy MKGCCCMDS ,
.Sy MKGDB ,
.Sy MKGROFF ,
.Sy MKGROFFHTMLDOC ,
.Sy MKHESIOD ,
.Sy MKHOSTOBJ ,
.Sy MKHTML ,
.Sy MKIEEEFP ,
.Sy MKINET6 ,
.Sy MKINFO ,
.Sy MKIPFILTER ,
.Sy MKISCSI ,
.Sy MKKERBEROS ,
.Sy MKKMOD ,
.Sy MKKYUA ,
.Sy MKLDAP ,
.Sy MKLIBCSANITIZER ,
.Sy MKLIBCXX ,
.Sy MKLIBSTDCXX ,
.Sy MKLINKLIB ,
.Sy MKLINT ,
.Sy MKLLVM ,
.Sy MKLLVMRT ,
.Sy MKLVM ,
.Sy MKMAKEMANDB ,
.Sy MKMAN ,
.Sy MKMANDOC ,
.Sy MKMANZ ,
.Sy MKMDNS ,
.Sy MKNLS ,
.Sy MKNOUVEAUFIRMWARE ,
.Sy MKNPF ,
.Sy MKNSD ,
.Sy MKOBJ ,
.Sy MKOBJDIRS ,
.Sy MKPAM ,
.Sy MKPCC ,
.Sy MKPF ,
.Sy MKPIC ,
.Sy MKPICINSTALL ,
.Sy MKPICLIB ,
.Sy MKPIE ,
.Sy MKPIGZGZIP ,
.Sy MKPOSTFIX ,
.Sy MKPROFILE ,
.Sy MKRADEONFIRMWARE ,
.Sy MKRELRO ,
.Sy MKREPRO ,
.Sy MKREPRO_TIMESTAMP ,
.Sy MKRUMP ,
.Sy MKSANITIZER ,
.Sy MKSHARE ,
.Sy MKSKEY ,
.Sy MKSLJIT ,
.Sy MKSOFTFLOAT ,
.Sy MKSTATICLIB ,
.Sy MKSTATICPIE ,
.Sy MKSTRIPIDENT ,
.Sy MKSTRIPSYM ,
.Sy MKTEGRAFIRMWARE ,
.Sy MKTPM ,
.Sy MKUNBOUND ,
.Sy MKUNPRIVED ,
.Sy MKUPDATE ,
.Sy MKX11 ,
.Sy MKX11FONTS ,
.Sy MKX11MOTIF ,
.Sy MKXORG_SERVER ,
.Sy MKYP ,
.Sy MKZFS ,
.Sy NETBSDSRCDIR ,
.Sy NETBSD_OFFICIAL_RELEASE ,
.Sy NOCLEANDIR ,
.Sy NODISTRIBDIRS ,
.Sy NOINCLUDES ,
.Sy OBJMACHINE ,
.Sy RELEASEDIR ,
.Sy RUMPUSER_THREADS ,
.Sy RUMP_CURLWP ,
.Sy RUMP_DEBUG ,
.Sy RUMP_DIAGNOSTIC ,
.Sy RUMP_KTRACE ,
.Sy RUMP_LOCKDEBUG ,
.Sy RUMP_LOCKS_UP ,
.Sy RUMP_NBCOMPAT ,
.Sy RUMP_VIRTIF ,
.Sy RUMP_VNODE_LOCKDEBUG ,
.Sy TOOLCHAIN_MISSING ,
.Sy TOOLDIR ,
.Sy USETOOLS ,
.Sy USE_FORT ,
.Sy USE_HESIOD ,
.Sy USE_INET6 ,
.Sy USE_JEMALLOC ,
.Sy USE_KERBEROS ,
.Sy USE_LDAP ,
.Sy USE_LIBCSANITIZER ,
.Sy USE_PAM ,
.Sy USE_PIGZGZIP ,
.Sy USE_SANITIZER ,
.Sy USE_SKEY ,
.Sy USE_SSP ,
.Sy USE_XZ_SETS ,
.Sy USE_YP ,
.Sy X11MOTIFPATH ,
.Sy X11SRCDIR .
.Ed
.Pp
The obsolete
.Xr mk.conf 5
make variables are:
.\" Keep in sync with share/man/man5/mk.conf.5 section OBSOLETE VARIABLES
.Bd -ragged -offset indent
.Sy EXTSRCSRCDIR ,
.Sy MKBFD ,
.Sy MKCRYPTO ,
.Sy MKEXTSRC ,
.Sy MKKDEBUG ,
.Sy MKKERBEROS4 ,
.Sy MKLLD ,
.Sy MKLLDB ,
.Sy MKMCLINKER ,
.Sy MKPERFUSE ,
.Sy MKTOOLSDEBUG ,
.Sy NBUILDJOBS ,
.Sy SHAREDSTRINGS ,
.Sy USE_COMBINE ,
.Sy USE_NEW_TOOLCHAIN .
.Ed
.
.Sh BUILDING
.
.Ss make command line options
This is not a summary of all the options available to
.Xr make 1 ;
only the options used most frequently with
.Nx
builds are listed here.
.
.Bl -tag -width "var=value"
.
.It Fl j Ar njob
Run up to
.Ar njob
.Xr make 1
subjobs in parallel.
Makefiles should use .WAIT or have explicit dependencies
as necessary to enforce build ordering.
.
.It Fl m Ar dir
Specify the default directory for searching for system Makefile
segments, mainly the
.Aq bsd.*.mk
files.
When building any full
.Nx
source tree, this should be set to the
.Dq Pa share/mk
directory in the source tree.
This is set automatically when building from the top level, or when using
.Sy build.sh .
.
.It Fl n
Show the commands that would have been executed, but do not
actually execute them.
This will still cause recursion to take place.
.
.It Fl V Ar var
Show
.Xr make 1 Ns 's
idea of the value of
.Ar var .
Does not build any targets.
.
.It Ar var Ns Sy = Ns Ar value
Set the variable
.Ar var
to
.Ar value ,
overriding any setting specified by the process environment, the
.Sy MAKECONF
configuration file, or the system Makefile segments.
.
.El
.
.Ss make targets
.
These default targets may be built by running
.Xr make 1
in any subtree of the
.Nx
source code.
It is recommended that none of these be used from the top
level Makefile; as a specific exception,
.Dq make obj
and
.Dq make cleandir
are useful in that context.
.
.Bl -tag -width "dependall"
.
.It Sy all
Build programs, libraries, and preformatted documentation.
.
.It Sy clean
Remove program and library object code files.
.
.It Sy cleandir
Same as
.Sy clean ,
but also remove preformatted documentation, dependency files generated
by
.Dq make depend ,
and any other files known to be created at build time.
.
.It Sy depend
Create dependency files
.Pa ( .depend )
containing more detailed information about the dependencies of source
code on header files.
Allows programs to be recompiled automatically when a dependency changes.
.
.It Sy dependall
Does a
.Dq make depend
immediately followed by a
.Dq make all .
This improves cache locality of the build since both passes read the source
files in their entirety.
.
.It Sy distclean
Synonym for
.Sy cleandir .
.
.It Sy includes
Build and install system header files.
Typically needed before any system libraries or programs can be built.
.
.It Sy install
Install programs, libraries, and documentation into
.Sy DESTDIR .
Few files will be installed to
.Sy DESTDIR Ns Pa /dev ,
.Sy DESTDIR Ns Pa /etc ,
.Sy DESTDIR Ns Pa /root
or
.Sy DESTDIR Ns Pa /var
in order to prevent user supplied configuration data from being overwritten.
.It Sy lint
Run
.Xr lint 1
against the C source code, where appropriate, and generate
system-installed lint libraries.
.
.It Sy obj
Create object directories to be used for built files, instead of
building directly in the source tree.
.
.It Sy tags
Create
.Xr ctags 1
searchable function lists usable by the
.Xr ex 1
and
.Xr vi 1
text editors.
.
.El
.
.Ss make targets for the top level
.
Additional
.Xr make 1
targets are usable specifically from the top source level to facilitate
building the entire
.Nx
source tree.
.
.Bl -tag -width "distribution"
.
.It Sy build
Build the entire
.Nx
system (except the kernel).
This orders portions of the source tree such that prerequisites
will be built in the proper order.
.
.It Sy distribution
Do a
.Dq make build ,
and then install a full distribution (which does not include a kernel) into
.Sy DESTDIR ,
including files in
.Sy DESTDIR Ns Pa /dev ,
.Sy DESTDIR Ns Pa /etc ,
.Sy DESTDIR Ns Pa /root
and
.Sy DESTDIR Ns Pa /var .
.
.It Sy buildworld
As per
.Dq make distribution ,
except that it ensures that
.Sy DESTDIR
is not the root directory.
.
.It Sy installworld
Install the distribution from
.Sy DESTDIR
to
.Sy INSTALLWORLDDIR ,
which defaults to the root directory.
Ensures that
.Sy INSTALLWORLDDIR
is not the root directory if cross compiling.
.Pp
The
.Sy INSTALLSETS
environment variable may be set to a space-separated list of
distribution sets to be installed.
By default, all sets except
.Dq etc
and
.Dq xetc
are installed, so most files in
.Sy INSTALLWORLDDIR Ns Pa /etc
will not be installed or modified.
.Pp
.Em Note :
Before performing this operation with
.Sy INSTALLWORLDDIR Ns = Ns Pa / ,
it is highly recommended that you upgrade your kernel and reboot.
After performing this operation,
it is recommended that you use
.Xr etcupdate 8
to update files in
.Sy INSTALLWORLDDIR Ns Pa /etc ,
and
.Xr postinstall 8
to check for or fix inconsistencies.
.It Sy sets
Create distribution sets from
.Sy DESTDIR
into
.Sy RELEASEDIR/RELEASEMACHINEDIR Ns Pa /binary/sets .
Should be run after
.Dq make distribution ,
as
.Dq make build
alone does not install all of the required files.
.
.It Sy sourcesets
Create source sets of the source tree into
.Sy RELEASEDIR Ns Pa /source/sets .
.
.It Sy syspkgs
Create syspkgs from
.Sy DESTDIR
into
.Sy RELEASEDIR/RELEASEMACHINEDIR Ns Pa /binary/syspkgs .
Should be run after
.Dq make distribution ,
as
.Dq make build
alone does not install all of the required files.
.
.It Sy release
Do a
.Dq make distribution ,
build kernels, distribution media, and install sets
(this as per
.Dq make sets ) ,
and
then package the system into a standard release layout as described by
.Xr release 7 .
This requires that
.Sy RELEASEDIR
be set (see above).
.
.It Sy iso-image
Create a
.Nx
installation CD-ROM image in the
.Sy RELEASEDIR Ns Pa /images
directory.
The CD-ROM file system will have a layout as described in
.Xr release 7 .
.Pp
For most machine types, the CD-ROM will be bootable, and will automatically
run the
.Xr sysinst 8
menu-based installation program, which can be used to install or upgrade a
.Nx
system.
Bootable CD-ROMs also contain tools that may be useful in
repairing a damaged
.Nx
installation.
.Pp
Before
.Dq make iso-image
is attempted, RELEASEDIR must be populated by
.Dq make release
or equivalent.
.Pp
.Em Note :
Other, smaller, CD-ROM images may be created in the
.Sy RELEASEDIR/RELEASEMACHINEDIR Ns Pa /installation/cdrom
directory by
.Dq "make release" .
These smaller images usually contain the same tools as the larger images in
.Sy RELEASEDIR Ns Pa /images ,
but do not contain additional content such as the distribution sets.
.Pp
.Em Note :
The mac68k port still uses an older method of creating
CD-ROM images.
This requires the
.Xr mkisofs 1
utility, which is not part of
.Nx ,
but which can be installed from
.Pa pkgsrc/sysutils/cdrtools .
.
.It Sy iso-image-source
Create a
.Nx
installation CD-ROM image in the
.Sy RELEASEDIR Ns Pa /images
directory.
The CD-ROM file system will have a layout as described in
.Xr release 7 .
It will have top level directories for the machine type and source.
.Pp
For most machine types, the CD-ROM will be bootable, and will automatically
run the
.Xr sysinst 8
menu-based installation program, which can be used to install or upgrade a
.Nx
system.
Bootable CD-ROMs also contain tools that may be useful in
repairing a damaged
.Nx
installation.
.Pp
Before
.Dq make iso-image-source
is attempted, RELEASEDIR must be populated by
.Dq make sourcesets release
or equivalent.
.Pp
.Em Note :
Other, smaller, CD-ROM images may be created in the
.Sy RELEASEDIR/RELEASEMACHINEDIR Ns Pa /installation/cdrom
directory by
.Dq make release .
These smaller images usually contain the same tools as the larger images in
.Sy RELEASEDIR Ns Pa /images ,
but do not contain additional content such as the distribution sets.
.Pp
.Em Note :
The mac68k port still uses an older method of creating
CD-ROM images.
This requires the
.Xr mkisofs 1
utility, which is not part of
.Nx ,
but which can be installed from
.Pa pkgsrc/sysutils/cdrtools .
.
.It Sy install-image
Create a bootable
.Nx
installation disk image in the
.Sy RELEASEDIR Ns Pa /images
directory.
The installation disk image is suitable for copying to
bootable USB flash memory sticks, etc., for machines which
are able to boot from such devices.
The file system in the bootable disk image will have a layout
as described in
.Xr release 7 .
.Pp
The installation image is bootable, and will automatically
run the
.Xr sysinst 8
menu-based installation program, which can be used to install or upgrade a
.Nx
system.
The image also contains tools that may be
useful in repairing a damaged
.Nx
installation.
.Pp
Before
.Dq make install-image
is attempted,
.Sy RELEASEDIR
must be populated by
.Dq make release
or equivalent.
The build must have been performed with
.Sy MKUNPRIVED=yes
because
.Dq make install-image
relies on information in
.Sy DESTDIR Ns Pa /METALOG .
.
.It Sy live-image
Create
.Nx
live images in the
.Sy RELEASEDIR/images
directory.
The live image contains all necessary files
to boot
.Nx
up to multi-user mode, including all files
which should be extracted during installation,
.Nx
disklabel, bootloaders, etc.
.Pp
The live image is suitable for use as a disk image in
virtual machine environments such as QEMU,
and also useful to boot
.Nx
from a USB flash memory stick
on a real machine, without the need for installation.
.Pp
Before
.Dq make live-image
is attempted, RELEASEDIR must be populated by
.Dq make release
or equivalent.
The build must have been performed with
.Sy MKUNPRIVED=yes
because
.Dq make install-image
relies on information in
.Sy DESTDIR Ns Pa /METALOG .
.
.It Sy regression-tests
Can only be run after building the regression tests in the directory
.Dq regress .
Runs those compiled regression tests on the local host.
.Pp
.Em Note :
Most tests are now managed instead using
.Xr atf 7 ;
this target should probably run those as well but currently does not.
.
.El
.
.Ss The build.sh script
.
This script file is a shell script designed to build the
entire
.Nx
system on any host with a suitable modern shell and some common
utilities.
The required shell features are described under the
.Sy HOST_SH
variable.
.Pp
If a host system's default shell does support the required
features, then we suggest that you explicitly specify
a suitable shell using a command like
.Bd -unfilled -offset indent
.Li /path/to/suitable/shell build.sh Op Ar options
.Ed
.Pp
The above command will usually enable
.Sy build.sh
to automatically set
.Sy HOST_SH Ns Cm = Ns Pa /path/to/suitable/shell ,
but if that fails, then the following set of commands may be used instead:
.Bd -unfilled -offset indent
.Li HOST_SH= Ns Va /path/to/suitable/shell
.Li export HOST_SH
.Li ${HOST_SH} build.sh Op Ar options
.Ed
.Pp
If
.Sy build.sh
detects that it is being executed under an unsuitable shell, it attempts
to exec a suitable shell instead, or shows an error message.
If
.Sy HOST_SH
is not set explicitly, then
.Sy build.sh
sets a default using heuristics dependent on the host platform,
or from the shell under which
.Sy build.sh
is executed (if that can be determined),
or using the first copy of
.Pa sh
found in
.Ev PATH .
.Pp
All cross-compile builds, and most native builds, of the entire system
should make use of
.Sy build.sh
rather than just running
.Dq make .
This way, the
.Xr make 1
program will be bootstrapped properly, in case the host system has an
older or incompatible
.Dq make
program.
.Pp
When compiling the entire system via
.Sy build.sh ,
many
.Xr make 1
variables are set for you in order to help encapsulate the build
process.
In the list of options below, variables that are automatically set by
.Sy build.sh
are noted where applicable.
.
.Pp
The following operations are supported by
.Sy build.sh :
.
.Bl -tag -width "distribution"
.
.It Sy build
Build the system as per
.Dq make build .
Before the main part of the build commences, this command runs the
.Sy obj
operation (unless the
.Fl o
option is given),
.Dq make cleandir
(unless the
.Fl u
option is given),
and the
.Sy tools
operation.
.
.It Sy distribution
Build a full distribution as per
.Dq make distribution .
This command first runs the
.Sy build
operation.
.
.It Sy release
Build a full release as per
.Dq make release .
This command first runs the
.Sy distribution
operation.
.
.It Sy help
Show a help message, and exit.
.
.It Sy makewrapper
Create the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script.
This operation is automatically performed for any of the other
operations.
.
.It Sy cleandir
Perform
.Dq make cleandir .
.
.It Sy obj
Perform
.Dq make obj .
.
.It Sy tools
Build and install the host tools from
.Pa src/tools .
This command will first run
.Dq make obj
and
.Dq make cleandir
in the
.Pa tools
subdirectory unless the
.Fl o
or
.Fl u
options (respectively) are given.
.
.It Sy install Ns = Ns Ar idir
Install the contents of
.Sy DESTDIR
to
.Ar idir ,
using
.Dq make installworld .
.Pp
.Em Note :
Files that are part of the
.Dq etc
or
.Dq xetc
sets will not be installed, unless overridden by
the INSTALLSETS environment variable.
.
.It Sy kernel Ns = Ns Ar kconf
Build a new kernel.
The
.Ar kconf
argument is the name of a configuration file suitable
for use by
.Xr config 1 .
If
.Ar kconf
does not contain any
.Sq /
characters, the configuration file is expected to be found in the
.Sy KERNCONFDIR
directory, which is typically
.Pa sys/arch/MACHINE/conf .
The new kernel will be built in a subdirectory of
.Sy KERNOBJDIR ,
which is typically
.Pa sys/arch/ Ns Sy MACHINE Ns Pa /compile
or an associated object directory.
.Pp
This command does
.Em not
imply the
.Sy tools
command; run the
.Sy tools
command first unless it is
.Em certain
that the tools already exist and are up to date.
.Pp
This command will run
.Dq make cleandir
on the kernel in question first unless the
.Fl u
option is given.
.
.It Sy kernel.gdb Ns = Ns Ar kconf
Build a new kernel with debug information.
Similar to the above
.Sy kernel Ns = Ns Ar kconf
operation, but creates a
.Pa netbsd.gdb
file alongside of the kernel
.Pa netbsd ,
which contains a full symbol table and can be used for debugging
(for example with a cross-gdb built by
.Sy MKCROSSGDB ) .
.It Sy kernels
This command will build all kernels defined in port specific release build
procedure.
.Pp
This command internally calls the
.Sy kernel Ns = Ns Ar kconf
operation for each found kernel configuration file.
.It Sy modules
This command will build kernel modules and install them into
.Sy DESTDIR .
.
.It Sy releasekernel Ns = Ns Ar kconf
Install a
.Xr gzip 1 Ns ed
copy of the kernel previously built by
.Sy kernel Ns = Ns Ar kconf
into
.Sy RELEASEDIR Ns Pa / Ns Sy RELEASEMACHINEDIR Ns Pa /binary/kernel ,
usually as
.Pa netbsd- Ns Ar kconf Ns Pa .gz ,
although the
.Dq Pa netbsd
prefix is determined from the
.Dq Sy config
directives in
.Ar kconf .
.
.It Sy sets
Perform
.Dq make sets .
.
.It Sy sourcesets
Perform
.Dq make sourcesets .
.
.It Sy syspkgs
Perform
.Dq make syspkgs .
.
.It Sy iso-image
Perform
.Dq make iso-image .
.
.It Sy iso-image-source
Perform
.Dq make iso-image-source .
.
.It Sy install-image
Perform
.Dq make install-image .
.
.It Sy live-image
Perform
.Dq make live-image .
.
.It Sy list-arch
Show a list of valid
.Sy MACHINE
and
.Sy MACHINE_ARCH
settings, the default
.Sy MACHINE_ARCH
for each
.Sy MACHINE ,
and aliases for
.Sy MACHINE Ns No / Ns Sy MACHINE_ARCH
pairs, and then exits.
The
.Fl m
or
.Fl a
options (or both) may be used to specify glob patterns
that will be used to narrow the list of results;
for example,
.Dq Li build.sh -m 'evb*' -a '*arm*' list-arch
will list all known
.Sy MACHINE Ns No / Ns Sy MACHINE_ARCH
values in which either
.Sy MACHINE
or
.Sy ALIAS
matches the pattern
.Ql evb* ,
and
.Sy MACHINE_ARCH
matches the pattern
.Ql *arm* .
.
.El
.
.Pp
The following command line options alter the behaviour of the
.Sy build.sh
operations described above:
.
.Bl -tag -width "-T tools"
.
.It Fl a Ar arch
Set the value of
.Sy MACHINE_ARCH
to
.Ar arch .
See the
.Fl m
option for more information.
.
.It Fl B Ar buildid
Set the value of
.Sy BUILDID
to
.Ar buildid .
This will also append the build identifier to the name of the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script so that the resulting name is of the form
.Dq Sy \*[toolprefix]make- Ns Sy MACHINE Ns Sy -BUILDID .
.
.It Fl C Ar cdextras
Append
.Ar cdextras
to the
.Sy CDEXTRA
variable,
which is a space-separated list of files or directories that will be
added to the CD-ROM image that may be create by the
.Dq iso-image
or
.Dq iso-image-source
operations.
Files will be added to the root of the CD-ROM image, whereas directories
will be copied recursively.
If relative paths are specified, they will be converted to
absolute paths before being used.
Multiple paths may be specified via multiple
.Fl C
options, or via a single option whose argument contains multiple
space-separated paths.
.
.It Fl c Ar compiler
Select the compiler for the toolchain to build
.Nx
and for inclusion in the
.Nx
distribution.
Supported choices:
.Bl -inset -offset indent
.It clang
.It gcc
[default]
.El
.Pp
The compiler used to build the toolchain can be different; see
.Sy HOST_CC
and
.Sy HOST_CXX .
.It Fl D Ar dest
Set the value of
.Sy DESTDIR
to
.Ar dest .
If a relative path is specified, it will be converted to an
absolute path before being used.
.
.It Fl E
Set
.Sq expert
mode.
This overrides various sanity checks, and allows:
.Sy DESTDIR
does not have to be set to a non-root path for builds,
and
.Sy MKUNPRIVED=yes
does not have to be set when building as a non-root user.
.Pp
.Em Note :
It is highly recommended that you know what you are doing when
you use this option.
.
.It Fl h
Show a help message, and exit.
.
.It Fl j Ar njob
Run up to
.Ar njob
.Xr make 1
subjobs in parallel;
passed through to
.Xr make 1 .
If you see failures for reasons other than running out of memory
while using
.Sy build.sh
with
.Fl j ,
please save complete build logs
so the failures can be analyzed.
.Pp
To achieve the fastest builds,
.Fl j
values between (1 + the number of CPUs) and (2 * the number of CPUs)
are recommended.
Use lower values on machines with limited memory or I/O bandwidth.
.
.It Fl M Ar obj
Set
.Sy MAKEOBJDIRPREFIX
to
.Ar obj .
Unsets
.Sy MAKEOBJDIR .
See
.Dq Fl O Ar obj
for more information.
.Pp
For instance, if the source directory is
.Pa /usr/src ,
a setting of
.Dq Fl M Pa /usr/obj
will place build-time files under
.Pa /usr/obj/usr/src/bin ,
.Pa /usr/obj/usr/src/lib ,
.Pa /usr/obj/usr/src/usr.bin ,
and so forth.
.Pp
If a relative path is specified, it will be converted to an
absolute path before being used.
.Sy build.sh
imposes the restriction that the argument to the
.Fl M
option must not begin with a
.Dq \&$
(dollar sign)
character; otherwise it would be too difficult
to determine whether the value is an absolute or a relative path.
If the directory does not already exist,
.Sy build.sh
will create it.
.
.It Fl m Ar mach
Set the value of
.Sy MACHINE
to
.Ar mach ,
unless the
.Ar mach
argument is an alias that refers to a
.Sy MACHINE Ns No / Ns Sy MACHINE_ARCH
pair, in which case both
.Sy MACHINE
and
.Sy MACHINE_ARCH
are set from the alias.
Such aliases are interpreted entirely by
.Sy build.sh ;
they are not used by any other part of the build system.
The
.Sy MACHINE_ARCH
setting implied by
.Ar mach
will override any value of
.Sy MACHINE_ARCH
in the process environment, but will not override a value set by the
.Fl a
option.
All cross builds require
.Fl m ,
but if unset on a NetBSD host, the host's value of MACHINE will be
detected and used automatically.
.Pp
See the
.Cm list-arch
operation for a way to get a list of valid
.Sy MACHINE
and
.Sy MACHINE_ARCH
settings.
.
.It Fl N Ar noiselevel
Set the
.Dq noisyness
level of the build, by setting
.Sy MAKEVERBOSE
to
.Ar noiselevel .
.
.It Fl n
Show the commands that would be executed by
.Sy build.sh ,
but do not make any changes.
This is similar in concept to
.Dq make -n .
.
.It Fl O Ar obj
Create an appropriate transform macro for
.Sy MAKEOBJDIR
that will place the built object files under
.Ar obj .
Unsets
.Sy MAKEOBJDIRPREFIX .
.Pp
For instance, a setting of
.Dq Fl O Pa /usr/obj
will place build-time files under
.Pa /usr/obj/bin ,
.Pa /usr/obj/lib ,
.Pa /usr/obj/usr.bin ,
and so forth.
.Pp
If a relative path is specified, it will be converted to an
absolute path before being used.
.Sy build.sh
imposes the restriction that the argument to the
.Fl O
option must not contain a
.Dq \&$
(dollar sign)
character.
If the directory does not already exist,
.Sy build.sh
will create it.
.Pp
In normal use, exactly one of the
.Fl M
or
.Fl O
options should be specified.
If neither
.Fl M
nor
.Fl O
is specified, then a default object directory will be chosen
according to rules in
.Aq bsd.obj.mk .
Relying on this default is not recommended because
it is determined by complex rules that are influenced
by the values of several variables and
by the location of the source directory.
.Pp
.Em Note :
Placing the
.Ar obj
directory location outside of the default source tree hierarchy makes
it easier to manually clear out old files in the event the
.Dq make cleandir
operation is unable to do so.
(See
.Sx CAVEATS
below.)
.Pp
.Em Note :
The use of one of
.Fl M
or
.Fl O
is the only means of building multiple machine architecture userlands
from the same source tree without cleaning between builds (in which
case, one would specify distinct
.Ar obj
locations for each).
.
.It Fl o
Set the value of
.Sy MKOBJDIRS
to
.Dq no .
Otherwise, it will be automatically set to
.Dq yes .
This default is opposite to the behaviour when not using
.Sy build.sh .
.
.It Fl P
Set the value of
.Sy MKREPRO
and
.Sy MKREPRO_TIMESTAMP
to the latest source CVS timestamp for reproducible builds.
.
.It Fl R Ar rel
Set the value of
.Sy RELEASEDIR
to
.Ar rel .
If a relative path is specified, it will be converted to an
absolute path before being used.
.
.It Fl r
Remove the contents of
.Sy DESTDIR
and
.Sy TOOLDIR
before building (provides a clean starting point).
This will skip deleting
.Sy DESTDIR
if building on a native system to the root directory.
.
.It Fl S Ar seed
Change the value of
.Sy BUILDSEED
to
.Ar seed .
This should rarely be necessary.
.
.It Fl T Ar tools
Set the value of
.Sy TOOLDIR
to
.Ar tools .
If a relative path is specified, it will be converted to an
absolute path before being used.
If set, the bootstrap
.Dq make
will only be rebuilt if the source files for
.Xr make 1
have changed.
.
.It Fl U
Set
.Sy MKUNPRIVED=yes .
.
.It Fl u
Set
.Sy MKUPDATE=yes .
.
.It Xo
.Fl V
.Sm off
.Ar var
.Li =
.Op Ar value
.Sm on
.Xc
Set the environment variable
.Ar var
to an optional
.Ar value .
This is propagated to the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script.
.
.It Fl w Ar wrapper
Create the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script (see below) in a custom location,
specified by
.Ar wrapper .
This allows, for instance, to place the wrapper script in
.Ev PATH
automatically.
.Pp
.Em Note :
.Ar wrapper
is the full name of the file, not just a directory name.
If a relative path is specified, it will be converted to an
absolute path before being used.
.
.It Fl X Ar x11src
Set the value of
.Sy X11SRCDIR
to
.Ar x11src .
If a relative path is specified, it will be converted to an
absolute path before being used.
.
.It Fl x
Set
.Sy MKX11=yes .
.
.It Fl Z Ar var
Unset ("zap") the environment variable
.Ar var .
This is propagated to the
.Sy \*[toolprefix]make- Ns Sy MACHINE
wrapper script.
.
.It Fl ?
Show a help message, and exit.
.
.El
.
.Ss The \*[toolprefix]make-MACHINE wrapper script
.
If using the
.Sy build.sh
script to build
.Nx ,
a
.Sy \*[toolprefix]make-MACHINE
wrapper script will be created in
.Sy TOOLDIR Ns Pa /bin
upon the first build to assist in building subtrees on a cross-compile
host.
.Pp
The
.Sy \*[toolprefix]make-MACHINE
wrapper script can be invoked in lieu of
.Xr make 1 ,
and will instead call the up-to-date version of
.Dq \*[toolprefix]make
installed into
.Sy TOOLDIR Ns Pa /bin
with several key variables pre-set, including
.Sy MACHINE , MACHINE_ARCH ,
and
.Sy TOOLDIR .
.Sy \*[toolprefix]make-MACHINE
will also set variables specified with
.Fl V ,
and unset variables specified with
.Fl Z .
Note that by default these variables will not override
.Xr mk.conf 5 ;
see
.Sx make variables
for more details.
.Pp
This wrapper script can be symlinked into a directory listed in
.Ev PATH ,
or called with an absolute path.
.
.Sh EXAMPLES
.
.Bl -enum
.
.It
.Li "% ./build.sh [OPTIONS] tools kernel=GENERIC"
.Pp
Build a new toolchain, and use the new toolchain to
configure and build a new GENERIC kernel.
.
.It
.Li "% ./build.sh [OPTIONS] -U distribution"
.Pp
Using unprivileged mode,
build a complete distribution to a
.Sy DESTDIR
directory that
.Sy build.sh
selects (and will show).
.
.It
.Li "# ./build.sh [OPTIONS] -U install=/"
.Pp
As root, install to
.Pa /
the distribution that was built
by example 2.
Even though this is run as root,
.Fl U
is required so that the permissions stored in
.Sy DESTDIR Ns Pa /METALOG
are correctly applied to the files as they're copied to
.Pa / .
.
.It
.Li "% ./build.sh [OPTIONS] -U -u release"
.Pp
Using unprivileged mode,
build a complete release to
.Sy DESTDIR
and
.Sy RELEASEDIR
directories that
.Sy build.sh
selects (and will show).
.Sy MKUPDATE=yes
.Pq Fl u
is set to prevent the
.Dq make cleandir ,
so that if this is run after example 2, it doesn't need to redo that
portion of the release build.
.El
.
.Sh SEE ALSO
.Xr config 1 ,
.Xr ctags 1 ,
.Xr ex 1 ,
.Xr gzip 1 ,
.Xr lint 1 ,
.Xr make 1 ,
.Xr mandoc 1 ,
.Xr mkisofs 1 ,
.Xr nroff 1 ,
.Xr vi 1 ,
.Xr mk.conf 5 ,
.Xr atf 7 ,
.Xr hier 7 ,
.Xr mdoc 7 ,
.Xr release 7 ,
.Xr etcupdate 8 ,
.Xr installboot 8 ,
.Xr mount 8 ,
.Xr postinstall 8 ,
.Xr sysinst 8 ,
.Pa pkgsrc/sysutils/cdrtools
.Pp
.Em Note :
The
.Nx
manual pages are also available at
.Lk https://man.netbsd.org
.
.Sh HISTORY
.
The
.Sy build.sh
based build scheme was introduced for
.Nx 1.6
as
.Sy USE_NEW_TOOLCHAIN ,
and re-worked to
.Sy TOOLCHAIN_MISSING
after that.
.
.Sh CAVEATS
.
After significant updates to third-party components in the source
tree, the
.Dq make cleandir
operation may be insufficient to clean out old files in object
directories.
Instead, one may have to manually remove the files.
Consult the
.Pa UPDATING
file for notices concerning this.