diff --git a/distrib/syspkg/notes/MORE_NOTES b/distrib/syspkg/notes/MORE_NOTES new file mode 100644 index 000000000000..c3884db66135 --- /dev/null +++ b/distrib/syspkg/notes/MORE_NOTES @@ -0,0 +1,43 @@ +A few things have solidified since the previous documents in this directory. +This document will serve to contain late-breaking news until the other +documents can be updated. + +Package names +------------- + +Package names are of the following form: + + -- + + is always the containing set. + is the software component (util, sendmail, games) + is a division of a package into parts. This allows + users to normally install all parts, but specialize + if desired. + +SUSHI and sysinst will be made aware of this naming scheme, so that they +can be made to offer, eg, to allow users to install `rcs' or to zoom +in and install `base-rcs-bin' and not `base-rcs-man'. This is in addition +to the bundling of packages into `package sets' corresponding to the +existing tarball sets for rapid sysinst-based installs. + +Package Divisions +----------------- + +The following strategy is being taken for division of the `base' package. + +First off, utilities are divided into a few big packages based on the following: + + directories base-sys-root, base-sys-usr + /bin base-util-root + /sbin base-sysutil-root + /usr/bin base-util-bin + /usr/lib base-sys-shlib + /usr/sbin base-sysutil-bin + +supporting files are named based on the utilities they match (with the changed). + +Then, groups of utilities which are either likely to not be installed (UUCP, SLIP) or +distinct from other utilities (postfix, sendmail, bind) are separated from the above. + +$Id: MORE_NOTES,v 1.1.1.1 2002/01/07 22:46:16 jwise Exp $ diff --git a/distrib/syspkg/notes/NOTES b/distrib/syspkg/notes/NOTES new file mode 100644 index 000000000000..b92280ab154f --- /dev/null +++ b/distrib/syspkg/notes/NOTES @@ -0,0 +1,111 @@ +I have been spending a lot of time lately thinking about how we +can extend our current package system to allow the use of system +packages (and sets thereof) rather than tarballs as our primary +way of installing NetBSD systems. What follows is a random jumble +of thoughts on where we can go with this. Based on people's response +to this, I will put together a more concrete proposal soon. + +1. Package Granularity + + People would like to be able to install/remove system software + with a high level of granularity. (Do I want Kerberos, but + not UUCP? C but not Fortran? Text formatting but not printing?) + This could be best addressed by providing our system software + as a series of packages which could be installed individually. + Dependencies would then exist among these packages, which would + insure that a working system resulted from various combinations + users might choose. For example, users could choose a C package + and/or a Fortran package, and both would depend on a binutils + package. + + At the same time, users should not need to choose between a + thousand little packages when installing the system. + + This probably means that there should be the concept of `sets' + of packages to be installed all at once, roughly analogous to + our current breakdown into base, comp, etc, games and so on. + A set would contain a number of packages, and a contents file + indicating which packages were part of the set. An install + program could then default to let users pick among whole sets, + but could have an advanced mode which would let users pick + individual packages instead. + + Very possibly, this would open the possibility of keeping set, + information, in addition to package information around, and + offering the ability to remove all installed packages from a + given set at a later date. + + Alternately, a system similar to our existing pkg-src/meta-pkgs + mechanism could be used -- users would choose meta-pkgs more + or less equivalent to the current sets, and these would trigger + dependencies on the packages which make up the set. + +2. Package Registration + + Currently, packages are installed in /usr/pkg and /usr/X11R6, + and all package registration is done by creating directories + in /var/db/pkg. + + This is generally agreed to have a couple of problems, especially + when you want to share /usr/pkg between multiple machines. + Unless /var/db/pkg is the same on both machines, you very + quickly end up with at least one machine not knowing what + packages it has installed. + + Alistair Crooks has recently put together changes to use + ${LOCALBASE}/etc/pkg and ${X11BASE}/etc/pkg to store package + registration information, which is a big step toward solving + this problem (yahoo!). Presumably, system packages will use + /etc/pkg, although this leaves open the question of what to do + if you share /usr between multiple client machines. + +3. Package Creation + + Right now, the Makefile in .../src/etc can be used to build + distribution sets, which are tarballs to be unpacked relative + to / by the installer. Sets are defined by lists in + .../src/distrib/sets/lists, and consist of MI and MD components. + Thus, where sources are installed in /usr/src, + /usr/src/distrib/sets/lists/comp/mi contains a list of files + in the compiler set which are available on all architectures, + while /usr/src/distrib/sets/lists/comp/md.i386 contains a list + of those files which are on the i386, but not on all other + architectures. All shared libraries, for example, are in the + md set, since not all ports have shared libraries. + + In order to move to using system packages, we will need to + provide a way to generate binary packages from our source tree. + It should be noted that the granularity of packages does not + really match the granularity of source subdirectories in + /usr/src/*, so barring a source tree re-organization (which + seems undesirable), we will need a way to do this which (like + the current mechanism) is outside of the normal `make clean ; + make depend; make' recursion through the source tree. A system + will also be needed to allow building a set out of a group of + packages. Both of these systems will need to be simple to use + (one make command line), and should be possible to run on a + single-package or single-set level. + +4. Package Versioning and Upgrades + + One of the neatest features offered by having independent + packages for different parts of the system is the ability to + upgrade a single package independently of the whole system. + Although this should not be abused (we don't want to become + like Linux where on an average system nothing quite matches + anything else), it would provide an ideal way to distribute + important patches and security fixes. + + For example, the security patch for NetBSD-SA1998-004 (at(1) + being usable to view any file) was distributed as a patch to + the NetBSD-1.3.2 source tree. This requires anyone who wishes + to install the fix to have a source tree available, download + and apply the patch, and then rebuild at(1). + + In a system based on install systems, an updated version of + the specific package containing at(1) could be released, which + users could download and pkg_add in a matter of moments. It + would also be easy for an admin to tell if a system had been + upgraded by looking at pkg_info output. + +$Id: NOTES,v 1.1.1.1 2002/01/07 22:46:16 jwise Exp $ diff --git a/distrib/syspkg/notes/PROPOSAL b/distrib/syspkg/notes/PROPOSAL new file mode 100644 index 000000000000..008447e72867 --- /dev/null +++ b/distrib/syspkg/notes/PROPOSAL @@ -0,0 +1,299 @@ +Proposal: NetBSD System Installation Packages +============================================= + +CONTENTS +-------- +0. Introduction +1. System Packages + 1.1 Package Format + 1.2 Package Granularity + 1.2.1 Root/User/Share separation +2. Package Sets + 2.1 Set format +3. Creation of Packages and Sets +4. Modifications to the NetBSD installation process +A. Working Plan + +------------------------------------------------------------------------ + +0. Introduction + + The current NetBSD installation process involves the downloading + of binary `sets', which the user can choose among at install time. + A set is a tarred, gzipped set of files, to be untarred relative + to '/'. No facility exists to choose convenient subsets of the files + in a set to be installed, or to remove a set which has been installed. + + The current granularity of sets is very large, being divided into: + + base -- general system binaries + comp -- compilers and related tools + etc -- system configuration files + games -- games and other amusements + man -- system manual pages + misc -- items not falling into other categories + secr -- items not exportable under US law + text -- text processing tools + xbase -- general X11R6 binaries + xcomp -- X11R6 development items + xcontrib - random binaries from the X11R6 `contrib' tree + xfont -- X11R6 fonts + xserver -- X11R6 servers for various video hardware + + Users who wish to install part of a set need to either install + the full set and then determine which files they need to remove, + or abandon the normal install process, and figure out which files + to unpack by hand. Similarly, if a set is later determined to + be unnecessary, the only way to remove it is to figure out which + files on the system belonged to that set, and remove them by hand. + + When it comes time to upgrade a system which has been installed this + way, the usual procedure is to unpack a new version of each installed + set over the previous version. When a file is moved, renamed, or + removed in a newer version of a set, the old version often remains on + the system for some time. In at least one recent instance (the move + of /sbin/mountd to /usr/sbin/mountd) this has resulted in much + confusion, and large amounts of traffic on the relevant mailing lists. + + The remainder of this document describes a proposed method of handling + these and other problems with the current install set system by + moving to the use of fine-grained `system packages', based on the + currently existing package system for third-party software, and + allowing users to choose among either `package sets' at the same + granularity as our current install sets, or individual `packages' + at a much finer level of granularity. In either case, the new system + would also greatly simplify upgrading or removal of such packages + and sets at a later time, and would allow tracking of dependencies + between the various sets and packages distributed as part of NetBSD. + + First, the format of system packages in the proposed system is + discussed, followed by the format of package sets, which will serve + as a replacement for the current install sets. The creation of + packages in an automated fashion from a NetBSD source tree is + discussed as is the effect of this system on the NetBSD installation + process. An appendix discusses my work plan to implement this new + system. + + It is hoped that this document will serve as a basis for discussion + of what is involved in changing NetBSD to use system packages for + system installation and upgrades, and that after several iterations + of discussion and revision, it will serve as a plan for the actual + implementation of this system. + +------------------------------------------------------------------------ + +1. System Packages + + System packages will be the basic building blocks of a NetBSD system. + At install time, the user will choose which system packages to install, + subject to dependencies between packages. After system install, + users will be able to install additional packages or remove installed + packages. When it comes time to upgrade the system, packages can + be removed and reinstalled in a reliable fashion. All of this + functionality is already available for third-party software via the + use of the software package system in /usr/pkgsrc. This proposal + extends that functionality to the NetBSD system itself. + +1.1 Package Format + + System packages will be identical in format to the binary packages + used by the current third-party package system. This will allow the + same tools to be used for working with system packages as are + currently used for working with third-party packages. This will also + also allow the system to benefit from the fact that the workings of + the current package system are well understood. + +1.2 Package Granularity + + System packages will be at the granularity of groups of related tools + and their support files. Thus, `Kerberos', `UUCP', `Text formatting' + and `amd' might each be packages which depended on nothing but a few + base packages, while `C Development' and `Fortran development' might + be separate packages which each depended upon `Binutils' and `Base + EGCS utilities' packages. Packages sets, described below, would add + the ability to choose entire broad categories of software to install, + like todays install sets, while maintaining the ability to remove + individual packages later. + +1.2.1 Root/User/Share separation + + In order to support a variety of system configurations, it is crucial + that the new package system support the possibility of some part of + a system residing on a server and possibly being shared between + multiple machines on a network. A machine which has some filesystems + local and some shared must, at the very least, be able to add and + remove packages from local filesystems, and should be able to + determine what packages have been added or removed from the volumes + mounted over the network. + + The most common shared configurations are to have a system share + /usr/share from the network, and have all other filesystems local, + or to share the entirety of /usr from the network, and maintain + local root and /var hierarchies, possibly as a single filesystem. + Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg. + + Two steps are necessary to support this type of sharing: the system + must be able to check separate repositories for packages installed + on different filesystems, and packages must be designed so as to + allow a client to install only those parts of the system which reside + on local filesystems. + + The first of these is addressed by a set of patches described by + Alistair Crooks in a post to the netbsd-current mailing list on + Friday, September 18, 1998. These patches, which have not yet been + committed cause third-party software packages installed in /usr/pkg + to be registered in /usr/pkg/etc/pkg, and packages installed in + /usr/X11R6 to be registered in /usr/X11R6/etc/pkg. This could be + extended easily to allow sharing of system package installations by + having the new system X11R6 packages also use /usr/X11R6/etc/pkg + for package registration, to have system packages installed in /usr + use /usr/etc/pkg for package registration, and to have system + packages installed in / and /var use /etc/pkg for package + registration. This would allow all of the types of filesystem + sharing described above, without introducing too much complication + into the package system. + + The second step, that of insuring that a client can choose to install + only the parts of the system which reside on local volumes can be + most easily addressed by careful consideration of package contents. + A look through the contents of the current install sets suggests + that relatively few packages will in fact need to install in more + than one of /, /usr, /usr/share and /usr/X11R6. Were such packages + split into separate components, based on filesystem boundaries, + users would easily be able to install only the parts which are local + in their particular configuration. + +------------------------------------------------------------------------ + +2. Package Sets + + In moving to fine-grained system packages, it is important that + beginning users still be able to select broad categories of software + to install at once. The introduction of `package sets', analogous + in granularity, but not mechanism, to the current binary install sets + addresses this concern, while maintaining the ability of more advanced + users to choose among individual packages at install time, and + maintaining the ability to remove, upgrade, or add individual + packages at a later time. + + These package sets will maintain the same layout as the current + install sets, so that a user who chooses the same sets as he would + have chosen now will see the same results. In the new system, + however, these sets will be made up of binary packages, and installing + a set will simply result in the installation of the constituent + packages. + +2.1 Set format + + A set will be a tar archive containing the packages which make up the + set plus a contents file. At the least, the index file will contain + the name of each included package, plus a one line description of each + package's contents. Installation utilities will offer the option of + installing the whole set, or choosing among individual packages, + based on the descriptions in the contents file. It is expected that the + contents file itself will be automatically generated from the one-line + descriptions provided in each package's pkg/COMMENT file. + + When a set is installed, the contents file will be recorded in a + manner similar to the registration of package information in the + current third-party package system. This will allow users to remove + an entire set at a later date, without needing to know what individual + packages came from that set. + +------------------------------------------------------------------------ + +3. Creation of Packages and Sets + + Under the current distribution-building system, the Makefile in + /usr/src/etc creates binary install sets from an installed system, + based on the set lists in /usr/src/distrib/sets/lists. In the new + system, a new directory hierarchy, /usr/src/distrib/pkg, will + contain Makefiles and data files relevant to the creation of + system packages and package sets. + + The directory /usr/src/distrib/pkg/sets will contain a directory + for each package set, and each of these directories will contain + a directory for each package in that set. The Makefile in + /usr/src/distrib/pkg/sets will recurse into these set directories + to build each set. The individual set Makefiles will recurse into + each package directory to build the individual packages, and will + then create a set file from the constituent packages and from the + contents file, which will be automatically generated from the + package directories. + + The package directories will resemble the package directories for + third-party software packages in /usr/pkgsrc, except that they will + probably rely on the files making up the package already being + present in ${DESTDIR}, rather than building them directly. This + assumption is already present in the current distribution package + Makefile code, and is probably reasonable to keep. + +------------------------------------------------------------------------ + +4. Modifications to the NetBSD installation process + + Once the NetBSD system is available as system packages and package + sets, it will be possible to modify the various installation tools + to use these sets to install the system. It is expected that + installation tools will default to allow users to choose among + package sets at install time, but allow an `advanced mode' in which + packages could be selected and deselected on an individual basis. + + This will require that the various package tools (at least pkg_add) + be present on install media to be used with system packages. + Modifications to sysinst and other install tools are beyond the + current scope of this proposal, but will be necessary to take + advantage of the new capabilities provided by this system. + +------------------------------------------------------------------------ + +A. Working Plan + + My current plan for implementing system packages and package sets + for NetBSD consists of four steps. All of these steps should be + taken in the CVS source tree (segregated into src/distrib/pkg, of + course), and hopefully will involve other contributors in addition + to myself: + + 1.) Hammer this proposal into a more detailed specification + + I am submitting this proposal now in the hopes that it + will spark discussion which will lead to a refinement + of the planned system package system. Once some sort + of consensus is reached on the relevant mailing lists, + I will begin work in earnest on implementing this. + + 2.) Create the /usr/src/distrib/pkg hierarchy, and a template + package + + The first step in actually implementing this system will + be to create either an actual or mocked-up system package + which can be used as a template for creation of the + remaining system packages. + + 3.) Create system packages + + I expect that this step will involve most of the actual + work in implementing the new system. Packages will have + to be created for each functional group of binaries + currently shipped with NetBSD. A lot of discussion and + design will have to go into the decisions as to how + many packages should make up each set and what files + belong in which packages. + + 4.) Create Package Sets + + Once all system packages exist, it will be necessary to + put together some code to automatically generate set + contents files and to create sets from each directory + of packages in /usr/src/distrib/pkg/sets. + + Once these steps are complete, NetBSD will have system packages, + and it will be possible to begin looking at modifying the NetBSD + install process to use them. It is important to note that none + of these changes will require modifying the current installation + set building code in any way, so the use of the current system + can continue unhindered while the new system is being implemented. + +------------------------------------------------------------------------ +$Id: PROPOSAL,v 1.1.1.1 2002/01/07 22:46:17 jwise Exp $ diff --git a/distrib/syspkg/notes/WORKPLAN b/distrib/syspkg/notes/WORKPLAN new file mode 100644 index 000000000000..a7430d63ea59 --- /dev/null +++ b/distrib/syspkg/notes/WORKPLAN @@ -0,0 +1,54 @@ +Work Plan: NetBSD System Packages +================================= + +This document is intended as a work plan for modification of the NetBSD +install process to use `system packages' and `package sets' to install +the base system instead of the current set tarballs. + +The steps toward this goal are as follows. Step a.) has already occurred, +and the remaining steps will begin this week, with changes influenced by +discussion on these lists. + + a.) update {make,check}flist, etc. to ignore all but the first + (whitespace separated) field of each line of each set file + + b.) add at least additional fields to each line of each set list + containing the name of the pkg the file is to be associated + with + + c.) add DESCR's, COMMENTs, etc. in a tree under src/distrib/pkg, + this tree will be hierarchical, with an organization + similar to pkgsrc. + + d.) add a script to churn through the set files and syspkg/ + and register the existing system, removing anything in + the obsolete lists in the process. This is working but + needs a complete rewrite. + +At this point, make build can register packages as if a packaged install +had been done. pkgs will be removable, and pkg versions will be usable +in a DEPENDS=. This will _greatly_ ease the process of upgrading from +a NetBSD system installed at this point to a later packaged distribution. +Now comes the fun part: + + e.) modify the distribution building makefiles to build + binary packages from the packages registered in DESTDIR + + f.) modify pkg_* to understand pkgsets which are tarballs of + pkgs with a contents file of some sort. These will be + used by sysinst to 1.) minimize the number of files to + be downloaded and 2.) present the current selection + granularity by default. + + g.) modify the make distrib tools to build these pkgsets from + the binary packages. + + h.) modify sysinst's set installation to use these pkgsets + instead of the current (tarball) sets. + + i.) modify sysinst to allow `zooming in' on a set and picking + what to install at pkg granularity, with dependencies + enforced and so on. + +------------------------------------------------------------------------ +$Id: WORKPLAN,v 1.1.1.1 2002/01/07 22:46:17 jwise Exp $