Aren/NetBSD
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Import syspkg documentation from draga.com CVS.

Browse Source
This commit is contained in:
jwise 2002-01-07 22:46:16 +00:00
parent f629228e87
commit dec9565bbc
4 changed files with 507 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
distrib/syspkg/notes/MORE_NOTES Normal file
View File

@ -0,0 +1,43 @@
A few things have solidified since the previous documents in this directory.
This document will serve to contain late-breaking news until the other
documents can be updated.
Package names
-------------
Package names are of the following form:
<set>-<pkg>-<part>
<set> is always the containing set.
<pkg> is the software component (util, sendmail, games)
<part> is a division of a package into parts. This allows
users to normally install all parts, but specialize
if desired.
SUSHI and sysinst will be made aware of this naming scheme, so that they
can be made to offer, eg, to allow users to install `rcs' or to zoom
in and install `base-rcs-bin' and not `base-rcs-man'. This is in addition
to the bundling of packages into `package sets' corresponding to the
existing tarball sets for rapid sysinst-based installs.
Package Divisions
-----------------
The following strategy is being taken for division of the `base' package.
First off, utilities are divided into a few big packages based on the following:
directories base-sys-root, base-sys-usr
/bin base-util-root
/sbin base-sysutil-root
/usr/bin base-util-bin
/usr/lib base-sys-shlib
/usr/sbin base-sysutil-bin
supporting files are named based on the utilities they match (with the <part> changed).
Then, groups of utilities which are either likely to not be installed (UUCP, SLIP) or
distinct from other utilities (postfix, sendmail, bind) are separated from the above.
$Id: MORE_NOTES,v 1.1.1.1 2002/01/07 22:46:16 jwise Exp $

111
distrib/syspkg/notes/NOTES Normal file
View File

@ -0,0 +1,111 @@
I have been spending a lot of time lately thinking about how we
can extend our current package system to allow the use of system
packages (and sets thereof) rather than tarballs as our primary
way of installing NetBSD systems. What follows is a random jumble
of thoughts on where we can go with this. Based on people's response
to this, I will put together a more concrete proposal soon.
1. Package Granularity
People would like to be able to install/remove system software
with a high level of granularity. (Do I want Kerberos, but
not UUCP? C but not Fortran? Text formatting but not printing?)
This could be best addressed by providing our system software
as a series of packages which could be installed individually.
Dependencies would then exist among these packages, which would
insure that a working system resulted from various combinations
users might choose. For example, users could choose a C package
and/or a Fortran package, and both would depend on a binutils
package.
At the same time, users should not need to choose between a
thousand little packages when installing the system.
This probably means that there should be the concept of `sets'
of packages to be installed all at once, roughly analogous to
our current breakdown into base, comp, etc, games and so on.
A set would contain a number of packages, and a contents file
indicating which packages were part of the set. An install
program could then default to let users pick among whole sets,
but could have an advanced mode which would let users pick
individual packages instead.
Very possibly, this would open the possibility of keeping set,
information, in addition to package information around, and
offering the ability to remove all installed packages from a
given set at a later date.
Alternately, a system similar to our existing pkg-src/meta-pkgs
mechanism could be used -- users would choose meta-pkgs more
or less equivalent to the current sets, and these would trigger
dependencies on the packages which make up the set.
2. Package Registration
Currently, packages are installed in /usr/pkg and /usr/X11R6,
and all package registration is done by creating directories
in /var/db/pkg.
This is generally agreed to have a couple of problems, especially
when you want to share /usr/pkg between multiple machines.
Unless /var/db/pkg is the same on both machines, you very
quickly end up with at least one machine not knowing what
packages it has installed.
Alistair Crooks has recently put together changes to use
${LOCALBASE}/etc/pkg and ${X11BASE}/etc/pkg to store package
registration information, which is a big step toward solving
this problem (yahoo!). Presumably, system packages will use
/etc/pkg, although this leaves open the question of what to do
if you share /usr between multiple client machines.
3. Package Creation
Right now, the Makefile in .../src/etc can be used to build
distribution sets, which are tarballs to be unpacked relative
to / by the installer. Sets are defined by lists in
.../src/distrib/sets/lists, and consist of MI and MD components.
Thus, where sources are installed in /usr/src,
/usr/src/distrib/sets/lists/comp/mi contains a list of files
in the compiler set which are available on all architectures,
while /usr/src/distrib/sets/lists/comp/md.i386 contains a list
of those files which are on the i386, but not on all other
architectures. All shared libraries, for example, are in the
md set, since not all ports have shared libraries.
In order to move to using system packages, we will need to
provide a way to generate binary packages from our source tree.
It should be noted that the granularity of packages does not
really match the granularity of source subdirectories in
/usr/src/*, so barring a source tree re-organization (which
seems undesirable), we will need a way to do this which (like
the current mechanism) is outside of the normal `make clean ;
make depend; make' recursion through the source tree. A system
will also be needed to allow building a set out of a group of
packages. Both of these systems will need to be simple to use
(one make command line), and should be possible to run on a
single-package or single-set level.
4. Package Versioning and Upgrades
One of the neatest features offered by having independent
packages for different parts of the system is the ability to
upgrade a single package independently of the whole system.
Although this should not be abused (we don't want to become
like Linux where on an average system nothing quite matches
anything else), it would provide an ideal way to distribute
important patches and security fixes.
For example, the security patch for NetBSD-SA1998-004 (at(1)
being usable to view any file) was distributed as a patch to
the NetBSD-1.3.2 source tree. This requires anyone who wishes
to install the fix to have a source tree available, download
and apply the patch, and then rebuild at(1).
In a system based on install systems, an updated version of
the specific package containing at(1) could be released, which
users could download and pkg_add in a matter of moments. It
would also be easy for an admin to tell if a system had been
upgraded by looking at pkg_info output.
$Id: NOTES,v 1.1.1.1 2002/01/07 22:46:16 jwise Exp $

299
distrib/syspkg/notes/PROPOSAL Normal file
View File

@ -0,0 +1,299 @@
Proposal: NetBSD System Installation Packages
=============================================
CONTENTS
--------
0. Introduction
1. System Packages
1.1 Package Format
1.2 Package Granularity
1.2.1 Root/User/Share separation
2. Package Sets
2.1 Set format
3. Creation of Packages and Sets
4. Modifications to the NetBSD installation process
A. Working Plan
------------------------------------------------------------------------
0. Introduction
The current NetBSD installation process involves the downloading
of binary `sets', which the user can choose among at install time.
A set is a tarred, gzipped set of files, to be untarred relative
to '/'. No facility exists to choose convenient subsets of the files
in a set to be installed, or to remove a set which has been installed.
The current granularity of sets is very large, being divided into:
base -- general system binaries
comp -- compilers and related tools
etc -- system configuration files
games -- games and other amusements
man -- system manual pages
misc -- items not falling into other categories
secr -- items not exportable under US law
text -- text processing tools
xbase -- general X11R6 binaries
xcomp -- X11R6 development items
xcontrib - random binaries from the X11R6 `contrib' tree
xfont -- X11R6 fonts
xserver -- X11R6 servers for various video hardware
Users who wish to install part of a set need to either install
the full set and then determine which files they need to remove,
or abandon the normal install process, and figure out which files
to unpack by hand. Similarly, if a set is later determined to
be unnecessary, the only way to remove it is to figure out which
files on the system belonged to that set, and remove them by hand.
When it comes time to upgrade a system which has been installed this
way, the usual procedure is to unpack a new version of each installed
set over the previous version. When a file is moved, renamed, or
removed in a newer version of a set, the old version often remains on
the system for some time. In at least one recent instance (the move
of /sbin/mountd to /usr/sbin/mountd) this has resulted in much
confusion, and large amounts of traffic on the relevant mailing lists.
The remainder of this document describes a proposed method of handling
these and other problems with the current install set system by
moving to the use of fine-grained `system packages', based on the
currently existing package system for third-party software, and
allowing users to choose among either `package sets' at the same
granularity as our current install sets, or individual `packages'
at a much finer level of granularity. In either case, the new system
would also greatly simplify upgrading or removal of such packages
and sets at a later time, and would allow tracking of dependencies
between the various sets and packages distributed as part of NetBSD.
First, the format of system packages in the proposed system is
discussed, followed by the format of package sets, which will serve
as a replacement for the current install sets. The creation of
packages in an automated fashion from a NetBSD source tree is
discussed as is the effect of this system on the NetBSD installation
process. An appendix discusses my work plan to implement this new
system.
It is hoped that this document will serve as a basis for discussion
of what is involved in changing NetBSD to use system packages for
system installation and upgrades, and that after several iterations
of discussion and revision, it will serve as a plan for the actual
implementation of this system.
------------------------------------------------------------------------
1. System Packages
System packages will be the basic building blocks of a NetBSD system.
At install time, the user will choose which system packages to install,
subject to dependencies between packages. After system install,
users will be able to install additional packages or remove installed
packages. When it comes time to upgrade the system, packages can
be removed and reinstalled in a reliable fashion. All of this
functionality is already available for third-party software via the
use of the software package system in /usr/pkgsrc. This proposal
extends that functionality to the NetBSD system itself.
1.1 Package Format
System packages will be identical in format to the binary packages
used by the current third-party package system. This will allow the
same tools to be used for working with system packages as are
currently used for working with third-party packages. This will also
also allow the system to benefit from the fact that the workings of
the current package system are well understood.
1.2 Package Granularity
System packages will be at the granularity of groups of related tools
and their support files. Thus, `Kerberos', `UUCP', `Text formatting'
and `amd' might each be packages which depended on nothing but a few
base packages, while `C Development' and `Fortran development' might
be separate packages which each depended upon `Binutils' and `Base
EGCS utilities' packages. Packages sets, described below, would add
the ability to choose entire broad categories of software to install,
like todays install sets, while maintaining the ability to remove
individual packages later.
1.2.1 Root/User/Share separation
In order to support a variety of system configurations, it is crucial
that the new package system support the possibility of some part of
a system residing on a server and possibly being shared between
multiple machines on a network. A machine which has some filesystems
local and some shared must, at the very least, be able to add and
remove packages from local filesystems, and should be able to
determine what packages have been added or removed from the volumes
mounted over the network.
The most common shared configurations are to have a system share
/usr/share from the network, and have all other filesystems local,
or to share the entirety of /usr from the network, and maintain
local root and /var hierarchies, possibly as a single filesystem.
Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg.
Two steps are necessary to support this type of sharing: the system
must be able to check separate repositories for packages installed
on different filesystems, and packages must be designed so as to
allow a client to install only those parts of the system which reside
on local filesystems.
The first of these is addressed by a set of patches described by
Alistair Crooks in a post to the netbsd-current mailing list on
Friday, September 18, 1998. These patches, which have not yet been
committed cause third-party software packages installed in /usr/pkg
to be registered in /usr/pkg/etc/pkg, and packages installed in
/usr/X11R6 to be registered in /usr/X11R6/etc/pkg. This could be
extended easily to allow sharing of system package installations by
having the new system X11R6 packages also use /usr/X11R6/etc/pkg
for package registration, to have system packages installed in /usr
use /usr/etc/pkg for package registration, and to have system
packages installed in / and /var use /etc/pkg for package
registration. This would allow all of the types of filesystem
sharing described above, without introducing too much complication
into the package system.
The second step, that of insuring that a client can choose to install
only the parts of the system which reside on local volumes can be
most easily addressed by careful consideration of package contents.
A look through the contents of the current install sets suggests
that relatively few packages will in fact need to install in more
than one of /, /usr, /usr/share and /usr/X11R6. Were such packages
split into separate components, based on filesystem boundaries,
users would easily be able to install only the parts which are local
in their particular configuration.
------------------------------------------------------------------------
2. Package Sets
In moving to fine-grained system packages, it is important that
beginning users still be able to select broad categories of software
to install at once. The introduction of `package sets', analogous
in granularity, but not mechanism, to the current binary install sets
addresses this concern, while maintaining the ability of more advanced
users to choose among individual packages at install time, and
maintaining the ability to remove, upgrade, or add individual
packages at a later time.
These package sets will maintain the same layout as the current
install sets, so that a user who chooses the same sets as he would
have chosen now will see the same results. In the new system,
however, these sets will be made up of binary packages, and installing
a set will simply result in the installation of the constituent
packages.
2.1 Set format
A set will be a tar archive containing the packages which make up the
set plus a contents file. At the least, the index file will contain
the name of each included package, plus a one line description of each
package's contents. Installation utilities will offer the option of
installing the whole set, or choosing among individual packages,
based on the descriptions in the contents file. It is expected that the
contents file itself will be automatically generated from the one-line
descriptions provided in each package's pkg/COMMENT file.
When a set is installed, the contents file will be recorded in a
manner similar to the registration of package information in the
current third-party package system. This will allow users to remove
an entire set at a later date, without needing to know what individual
packages came from that set.
------------------------------------------------------------------------
3. Creation of Packages and Sets
Under the current distribution-building system, the Makefile in
/usr/src/etc creates binary install sets from an installed system,
based on the set lists in /usr/src/distrib/sets/lists. In the new
system, a new directory hierarchy, /usr/src/distrib/pkg, will
contain Makefiles and data files relevant to the creation of
system packages and package sets.
The directory /usr/src/distrib/pkg/sets will contain a directory
for each package set, and each of these directories will contain
a directory for each package in that set. The Makefile in
/usr/src/distrib/pkg/sets will recurse into these set directories
to build each set. The individual set Makefiles will recurse into
each package directory to build the individual packages, and will
then create a set file from the constituent packages and from the
contents file, which will be automatically generated from the
package directories.
The package directories will resemble the package directories for
third-party software packages in /usr/pkgsrc, except that they will
probably rely on the files making up the package already being
present in ${DESTDIR}, rather than building them directly. This
assumption is already present in the current distribution package
Makefile code, and is probably reasonable to keep.
------------------------------------------------------------------------
4. Modifications to the NetBSD installation process
Once the NetBSD system is available as system packages and package
sets, it will be possible to modify the various installation tools
to use these sets to install the system. It is expected that
installation tools will default to allow users to choose among
package sets at install time, but allow an `advanced mode' in which
packages could be selected and deselected on an individual basis.
This will require that the various package tools (at least pkg_add)
be present on install media to be used with system packages.
Modifications to sysinst and other install tools are beyond the
current scope of this proposal, but will be necessary to take
advantage of the new capabilities provided by this system.
------------------------------------------------------------------------
A. Working Plan
My current plan for implementing system packages and package sets
for NetBSD consists of four steps. All of these steps should be
taken in the CVS source tree (segregated into src/distrib/pkg, of
course), and hopefully will involve other contributors in addition
to myself:
1.) Hammer this proposal into a more detailed specification
I am submitting this proposal now in the hopes that it
will spark discussion which will lead to a refinement
of the planned system package system. Once some sort
of consensus is reached on the relevant mailing lists,
I will begin work in earnest on implementing this.
2.) Create the /usr/src/distrib/pkg hierarchy, and a template
package
The first step in actually implementing this system will
be to create either an actual or mocked-up system package
which can be used as a template for creation of the
remaining system packages.
3.) Create system packages
I expect that this step will involve most of the actual
work in implementing the new system. Packages will have
to be created for each functional group of binaries
currently shipped with NetBSD. A lot of discussion and
design will have to go into the decisions as to how
many packages should make up each set and what files
belong in which packages.
4.) Create Package Sets
Once all system packages exist, it will be necessary to
put together some code to automatically generate set
contents files and to create sets from each directory
of packages in /usr/src/distrib/pkg/sets.
Once these steps are complete, NetBSD will have system packages,
and it will be possible to begin looking at modifying the NetBSD
install process to use them. It is important to note that none
of these changes will require modifying the current installation
set building code in any way, so the use of the current system
can continue unhindered while the new system is being implemented.
------------------------------------------------------------------------
$Id: PROPOSAL,v 1.1.1.1 2002/01/07 22:46:17 jwise Exp $

54
distrib/syspkg/notes/WORKPLAN Normal file
View File

@ -0,0 +1,54 @@
Work Plan: NetBSD System Packages
=================================
This document is intended as a work plan for modification of the NetBSD
install process to use `system packages' and `package sets' to install
the base system instead of the current set tarballs.
The steps toward this goal are as follows. Step a.) has already occurred,
and the remaining steps will begin this week, with changes influenced by
discussion on these lists.
a.) update {make,check}flist, etc. to ignore all but the first
(whitespace separated) field of each line of each set file
b.) add at least additional fields to each line of each set list
containing the name of the pkg the file is to be associated
with
c.) add DESCR's, COMMENTs, etc. in a tree under src/distrib/pkg,
this tree will be hierarchical, with an organization
similar to pkgsrc.
d.) add a script to churn through the set files and syspkg/
and register the existing system, removing anything in
the obsolete lists in the process. This is working but
needs a complete rewrite.
At this point, make build can register packages as if a packaged install
had been done. pkgs will be removable, and pkg versions will be usable
in a DEPENDS=. This will _greatly_ ease the process of upgrading from
a NetBSD system installed at this point to a later packaged distribution.
Now comes the fun part:
e.) modify the distribution building makefiles to build
binary packages from the packages registered in DESTDIR
f.) modify pkg_* to understand pkgsets which are tarballs of
pkgs with a contents file of some sort. These will be
used by sysinst to 1.) minimize the number of files to
be downloaded and 2.) present the current selection
granularity by default.
g.) modify the make distrib tools to build these pkgsets from
the binary packages.
h.) modify sysinst's set installation to use these pkgsets
instead of the current (tarball) sets.
i.) modify sysinst to allow `zooming in' on a set and picking
what to install at pkg granularity, with dependencies
enforced and so on.
------------------------------------------------------------------------
$Id: WORKPLAN,v 1.1.1.1 2002/01/07 22:46:17 jwise Exp $