The *_FOREACH macros got added post 1.5 and were pulled up into 1.5.2. This
makes it impossible to compile mtree local on a 1.5-release box as a cross
toolchain bit for bootstrapping to -current. (and why I never saw this problem
on my 1.5.2 machine)
This whole thing is solved via a compat lib for the tools for all netbsd bits
and/or trimming down a version of mtree to just the bits needed for building
the snapshots.
distrib Makefiles have been adjusted appropriately. This change made
block devices and char devices look the same to the naive user, and
it is not appropriate.
- Add -L to walk the tree `logically', by following symbolic links in
the heirarchy.
- Add -P to walk the tree `physically'. This is the current behaviour,
and the default.
- Add "-X excludes-file" to give mtree the ability to exclude files and
directories from its traversal. excludes-file contains fnmatch(3)
patterns to exclude from the walk.
- Add "md5digest" synonym for "md5".
- Add "rmd160" keyword for RMD-160 message digest, and "rmd160digest" synonym.
- Add "sha1" keyword for SHA-1 message digest, and "sha1digest" synonym.
- Don't try to compare() other attributes if the type doesn't match;
it's nothing but trouble, and no use anyway.
- In -c, only emit "/set" records if something has changed since the
previous one.
User interface changes by me:
- Check a device's parameters before checking uid/gid/mode.
- If updating (-u), modify the following to match the specification:
- Device type (retaining existing ownership).
- Symlink target.
Fixes from (or inspired by) FreeBSD:
- Use p->ftslevel instead of own code to keep track of the level ourself.
The previous code got majorly confused if fts(3) couldn't descend
into a subdir, resulting in leaf nodes getting attached to the wrong
directory.
XXX: This new method is much much more robust, even though it's not 100%
perfect; it might result in a couple of following entries in the spec
to be incorrectly tagged as missing.
- Pass a useful pathname to rlink(), so that logical (-L) traversal
doesn't confuse symlink checking.
- Consistently use MAXPATHLEN+1 sized buffers for pathnames, so that
there's room for the NUL.
- Use mtree_err() and strerror(p->fts_errno) to report errors during
the fts(3) walk.
Fixes by me:
- Remove now-unused `const char *name' argument from compare().
- Change crc_total from an int to a u_int32_t, to match usr.bin/cksum/crc.c.
- Remove trailing whitespace.
- Remove unnecessary (void) casts on functions.
- Reorder entries in the getopt() switch.
- Replace strtoq() with strtoll(), and use strtoul() appropriately.
- Renumber F_ flags to be in alphabetical order.
of tv's previous code, which skips uname or gname parsing if -W is enabled.
- rename "lineno" to "mtree_lineno", to reduce possibility of name
clashes in code that yanks in spec.c & misc.c (unlikely, but you never know)
check on the correct node
- apply_specentry(): if this node is a duplicate of another, apply the
changes to the `master' entry instead of this one.
- fix inotype() to DTRT
- comment out some debugging info that is too verbose
- use syslog after we become a daemon to write error messages.
- failure to open a device is not fatal.
- copy timeval, because select is allowed to change it.
- if we failed to blank all the devices we are monitoring, exit
It doesn't need any special privileges or kernel devices.
Only ffs image creation is supported at this time, although makefs has been
designed to allow the addition of other file system formats by writing new
back-ends.
This program was designed & implemented by Luke Mewburn of Wasabi Systems.
It doesn't need any special privileges or kernel devices.
Only ffs image creation is supported at this time, although makefs has been
designed to allow the addition of other file system formats by writing new
back-ends.
This program was designed & implemented by Luke Mewburn of Wasabi Systems.
It doesn't need any special privileges or kernel devices.
Only ffs image creation is supported at this time, although makefs has been
designed to allow the addition of other file system formats by writing new
back-ends.
This program was designed & implemented by Luke Mewburn of Wasabi Systems.
+ remove some unused code in a comment.
+ add F_SHELL flag in userdel
+ use a separate local declaration for a temporary variable, rather
than overloading a variable that's used for something else.
various potential problems when intermixing full and relative paths
- changes to make it much easier to use spec() - the specfile parser - in
other programs (via .PATHing spec.c and misc.c):
- move excludetags, includetags and keys from mtree.c to misc.c
- implement mtree_err() using vwarnx() instead of assuming name is
"mtree"
- move inotype() and nodetype() from compare.c to misc.c
- add nodetoinode(), to convert from an mtree F_* type to a
mode_t S_IF* type
- clean up #include use; don't assume "mtree.h" pulls in some
standard includes
- change spec() to take a FILE * arg (where the specfile is read from)
- make a copy of cfparse.y called "y.tab.y" because "cfparse.h" is not
actually the yacc generated header file (duh?)
- include the tcpdump directory with -I *after* racoon's source, else
tcpdump's headers will be picked up unexpectedly
- include . *before* racoon's source so as to make the generated files
first on the list
is one that contains a `/' character that is not the first character.
All parent directories referenced in the path must exist.
* Add copyright for all the work I've done.
* Document the history of various features added since 4.3-Reno.
- parsekey(): support "all" as a keyword which returns a value with all bits
set that pertain to a valid keyword.
- dump_nodes():
- only print out a keyword if it's requested AND it's set in the node
- UNAME falls back to UID if the user name can't be determined
- GNAME falls back to GID if the group name can't be determined
- rework man page:
- use Sy instead of Cm
- use Pq, Dq, Ql, etc instead of ``...''
- create a typedef struct slist_t to hold stuff in (avoid stringlist(3)
because it's less portable)
- move the tag manipulate stuff into misc.c
- add matchtags(), which returns 0 if the node is explicitly excluded,
or not included if an include list is given; or 1 otherwise
- in -D, change tags support from files-only to all-non-directories
(xxx; still considering implementing tags support for -c and !-D)
- add a new keyword - "tags" - which is a comma separated list of tags
associated with that file.
- add "-E tags"; exclude files in -D output with matching tags
- add "-I tags"; only include files in -D output with matching tags
(default is to list all)
fixes:
- move all extern variable references from individual files into extern.h
- `quote' some more user-specified strings in error messages
- man page: list an options arguments in the description, and sort xrefs.
The purpose of that modification is to make the license on these files
"GPL-compatible," so that mdsetimage may be combined with BFD. (The
resulting license is word-for-word identicial to one of the licenses
described as being GPL compatible on
http://www.gnu.org/licenses/license-list.html.)
by various tools
- implement -R key - remove the given key(s) from the list of keys to print
(now it's possible with "-k type -R type" to remove all keys :)
- rename ftype() to nodetype(), and make it public
- use pwcache(3) functions instead of get{gr,pw}{nam,uid}(3)
- add dump_nodes(), which provides the guts for -D.
to a verification program for a binary package. The following callouts
are defined: "none", "gpg" and "pgp5".
This feature allows you to verify a binary package against a detached
signature file, and to proceed with the installation or not, depending
upon the level of trust you place in the signatory of the binary
package.
Digital signatures will be checked in a recursive manner (i.e. if
pkg_add is called with a verification type which is not "none", the
verification type will be passed to subsequent invocations of pkg_add
for the dependent packages).
At the current time, digital signatures cannot be used with the URL
form of pkg_add(1) - the detached signature file must be in the same
directory as the binary package, either locally or mounted by NFS.
If no -s argument is given, pkg_add(1) retains its current behaviour -
the package will not be verified before installation takes place.
Make sure that each va_start has one and only one matching va_end,
especially in error cases.
If the va_list is used multiple times, do multiple va_starts/va_ends.
If a function gets va_list as argument, don't let it use va_end (since
it's the callers responsibility).
Improved by comments from enami and christos -- thanks!
Heimdal/krb4/KAME changes already fed back, rest to follow.
Inspired by, but not not based on, OpenBSD.
matches a check for mode 644 (though obviously not the reverse). This
can be used by the nightly security run, making the output1 more useful
by having it contain fewer spurious permissions violations.
Note that I did not make -l work if you have a sgid/suid/sticky bit
set. I don't know how you could cause security trouble with more
stringent settings and a suid file, but I don't want to find out the
hard way.
instead of err()), as all error cases in creategid() are already commented
properly from inside creategid().
This prevents funny errors like:
miyu# groupadd test2
miyu# groupadd test2
groupadd: group `test2' already exists
groupadd: can't add group: problems with /etc/group file: Inappropriate ioctl for device
FreeBSD (three commits; the initial work, man page updates, and a fix
to ffs_reload()), with the following differences:
- Be consistent between newfs(8) and tunefs(8) as to the options which
set and control the tuning parameters for this work (avgfilesize & avgfpdir)
- Use u_int16_t instead of u_int8_t to keep track of the number of
contiguous directories (suggested by Chuck Silvers)
- Work within our FFS_EI framework
- Ensure that fs->fs_maxclusters and fs->fs_contigdirs don't point to
the same area of memory
The new algorithm has a marked performance increase, especially when
performing tasks such as untarring pkgsrc.tar.gz, etc.
The original FreeBSD commit messages are attached:
=====
mckusick 2001/04/10 01:39:00 PDT
Directory layout preference improvements from Grigoriy Orlov <gluk@ptci.ru>.
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.
------
One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.
First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:
1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
test is at wd1. Size of test file system is 8 Gb, number of cg=991,
size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
from Dec 2000 with BUFCACHEPERCENT=35
2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50
You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html
Test Results
tar -xzf ports.tar.gz rm -rf ports
mode old dirpref new dirpref speedup old dirprefnew dirpref speedup
First system
normal 667 472 1.41 477 331 1.44
async 285 144 1.98 130 14 9.29
sync 768 616 1.25 477 334 1.43
softdep 413 252 1.64 241 38 6.34
Second system
normal 329 81 4.06 263.5 93.5 2.81
async 302 25.7 11.75 112 2.26 49.56
sync 281 57.0 4.93 263 90.5 2.9
softdep 341 40.6 8.4 284 4.76 59.66
"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.
------
Algorithm description
The old dirpref algorithm is described in comments:
/*
* Find a cylinder to place a directory.
*
* The policy implemented by this algorithm is to select from
* among those cylinder groups with above the average number of
* free inodes, the one with the smallest number of directories.
*/
A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.
What I mean by a big file system ?
1. A big filesystem is a filesystem which occupy 20-30 or more percent
of total drive space, i.e. first and last cylinder are physically
located relatively far from each other.
2. It has a relatively large number of cylinder groups, for example
more cylinder groups than 50% of the buffers in the buffer cache.
The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.
My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
* Find a cylinder group to place a directory.
*
* The policy implemented by this algorithm is to allocate a
* directory inode in the same cylinder group as its parent
* directory, but also to reserve space for its files inodes
* and data. Restrict the number of directories which may be
* allocated one after another in the same cylinder group
* without intervening allocation of files.
*
* If we allocate a first level directory then force allocation
* in another cylinder group.
*/
My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.
My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.
The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:
int32_t fs_avgfilesize; /* expected average file size */
int32_t fs_avgfpdir; /* expected # of files per directory */
These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.
I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.
Obtained from: Grigoriy Orlov <gluk@ptci.ru>
=====
=====
iedowse 2001/04/23 17:37:17 PDT
Pre-dirpref versions of fsck may zero out the new superblock fields
fs_contigdirs, fs_avgfilesize and fs_avgfpdir. This could cause
panics if these fields were zeroed while a filesystem was mounted
read-only, and then remounted read-write.
Add code to ffs_reload() which copies the fs_contigdirs pointer
from the previous superblock, and reinitialises fs_avgf* if necessary.
Reviewed by: mckusick
=====
=====
nik 2001/04/10 03:36:44 PDT
Add information about the new options to newfs and tunefs which set the
expected average file size and number of files per directory. Could do
with some fleshing out.
=====
cylinder groups to work correctly, with minor modifications by me to work
with our FFS_EI code. From the FreeBSD commit message:
The ffs superblock includes a 128-byte region for use by temporary
in-core pointers to summary information. An array in this region
(fs_csp) could overflow on filesystems with a very large number of
cylinder groups (~16000 on i386 with 8k blocks). When this happens,
other fields in the superblock get corrupted, and fsck refuses to
check the filesystem.
Solve this problem by replacing the fs_csp array in 'struct fs'
with a single pointer, and add padding to keep the length of the
128-byte region fixed. Update the kernel and userland utilities
to use just this single pointer.
With this change, the kernel no longer makes use of the superblock
fields 'fs_csshift' and 'fs_csmask'. Add a comment to newfs/mkfs.c
to indicate that these fields must be calculated for compatibility
with older kernels.
Reviewed by: mckusick
code considerably and was mainly providing SGI specific logging facilities
and some hacks to improve the relyablility on SGI systems by increasing
priorities etc...
- replace the unused fs_headswitch and fs_trkseek with fs_id[2], bringing
our struct fs closer to that in freebsd & openbsd (& solaris FWIW)
- dumpfs: improve warning message when cpc == 0
- Add -u option, to update one user's records only.
- Add -s option, to update the secure database only.
- Give us roughly 1MB of cache per 50kB of password file, but keep within
a 2MB minimum and 8MB maximum.
- Tidy a little.
determine the endianness of the `struct fs *o' superblock from o->fs_magic
and set needswap as necessary, rather than trusting the caller to get
it right. invariably, almost every caller of ffs_sb_swap() was calling it
with ns set to the wrong value for ns anyway!
ansi KNF ffs_bswap.c declarations whilst here.
this fixes all sorts of problems when trying to use other-endian file systems,
notably the kernel trying to access memory *way* off, possibly corrupting or
panicing, and userland programs SEGVing and/or corrupting things (e.g,
"fsck_ffs -B" to swap a file system endianness).
whilst the previous rev of ffs_bswap.c (1.10, 2000/12/23) made this problem
worse, i suspect that the problem was always there and previous versions
just happened not to trash things at the wrong time.
FFS_EI should now be a lot more stable.
in a variable that can be added other maps than amd.home. Patch submitted
in PR 11826 by Anthony Mallet <anthony.mallet@ficus.yi.org>
* Slightly optimize calculation of DOMAIN
server1,server2,server3 on the command line. This allows the user to
optionally avoid the interactive mode of ypinit. If the -l flag is not
supplied, the old behavior is retained.
This allows users to do things like rsh out creation of YP clients, and
sushi to generate yp clients/servers/slaves.
the error detection for invalid types, as well as shrinking binary size
- implement inotype() using ftype(), rather than duplicating the switch
- change "char *" arguments to "const char *"
stdout before syslogd becomes a daemon.
- Flags for setuid/setgid/chroot syslogd after initialization is completed
- Warning instead of silent ignoring for malformed lines (with spaces instead
of tags)
Approved by Christos
the PLIST_CWD in creating the tar pipe.
This takes care of pkg_create -p PREFIX -L FAKE
where PREFIX does not exist at pkg creation time.
From OpenBSD's rev. 1.9->1.10
* Register PLIST_SRC before dependencies and conflicts so that the
PLIST_CWD != PLIST_SRC hack works with dependencies present.
From OpenBSD's rev. 1.10->1.11
* improve error message to tell full path of exec() that failed
Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:
- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).
- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.
- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.
- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).
- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.
Other changes of note include:
- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.
- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.
We were using rename() to create the target file from the temp file. Now the temp file is created in the same directory of the targer file so that rename() will always work (it needs to have the files in the same filesystem)
Using the -o file, the output file may be on a world writable directory, we hence open the temporary file using O_EXCL, and we loop trying different names until it works.
KAME rtsold.c 1.28
watch routing socket, and check for pcmcia card removal.
if pcmcia card gets removed, bring the interface state into IFS_DOWN.
KAME rtsold.c 1.29
changed the timeout period after sending MAX_RTR_SOLICITATIONS solicitation
from RTR_SOLICITATION_INTERVAL to MAX_RTR_SOLICITATION_DELAY according to
the last paragraph of RFC 2461 Section 6.3.7.
in response to a comment from Pekka Savola <pekkas@netcore.fi>
It broke a sequence of:
key1 \
value1 \
value2 \
value3 \
key2 \
value4 \
etc.
Handle both the netgroup rule and the amd.home rule using a .USEBEFORE
macro rule, since they contain the same code.
XXX: Needs new make(1)
restore the previous behaviour of packing locators.
Results in a 2.5KB increase in size for the current i386 GENERIC
kernel. My custom kernel resulted in a 12-byte increase.
a `machine' line. For each of these subarches, its
arch/${SUBARCH}/conf/files.${SUBARCH} is included, and
a symlink ${SUBARCH} -> arch/${SUBARCH}/include is made.
This will for a "richer structure" and easier code
sharing under sys/arch.
given filename.
* Use strip_txz() inside the FTP-wildcard-depends handling code to
prevent it handing a combination of glob and dewey pattern to pmatch(),
which our pattern matching code currently does not handle. Bugfix!
XXX The other places that could be changed to use strip_txz() are not
touched in this commit to keep the impact of this change small in the
light of the approaching 1.5.1 release. I'll revisit them.
to the common practice of inventing an unwieldy shell construct each
time.
Note: -f (ala route -f) is in use already, and FreeBSD provides "prior
art" for the use of -d and -a together to accomplish this.
leading to incorrect masks being used. Problem pointed out by
Martin Blapp (mb@imp.ch) who ported the code to FreeBSD, fixed
differently for the time being.
still let "pkg_add foo" work:
setenv PKG_PATH /usr/upkgsrc/packages/All
pkg_add -v kde2
Depending pkgs will be pulled in as usual.
If PKG_PATH contains a mixture of local paths and ftp://-URLs, the local
paths will be searched first.
Patch provided by Hubert.
1. Use standard shell indentation
2. Use [ expr -a expr ] instead of [ expr ] && [ expr ] (same with -o and ||)
3. Cleanup 0 length string tests
4. Factor out the routing code
5. Use a large switch statement based on $reason instead of the endless
if elif's
6. Use variables for filenames used
7. Test in all cases for the existance of domainname and hostname
8. No need for individual command redirections in structured statements.
Thanks to kleink for testing and fixing my typos!
getfile can't resolve client %s
getfile can't resolve server %s for %s
getfile lookup failed for %s
(the second condition had me chasing my tail for a while recently...)
Based on a similar patch in OpenBSD, reported there by
j@ida.interface-business.de.
Fix the second occurrance of a similar problem in that file, too, though.
- Handle message transport failure when retrieving parameters.
- Pad the parameter group buffer for Intel devices.
- Display the TID <-> device map upon request.
- Complain properly if we don't know about the command that was given.
Old kvm_mkdb had a possibility to check old kernel's version
string retained at a hole between text segment and data segment.
It might happen a misunderstanding that the running kernel's version
matches kvm.db's version.
This had been caused by checking current kernel's version string
via /dev/kmem. The version string can get via sysctl, so use it!
When invoked with 'pkg_add -u ...', and the package to be installed is
found already being installed, the old (installed) package is removed
with pkg_delete, and the new one is installed.
A very crude handling of dependencies is done (by moving the +REQUIRED_BY
file aside before pkg_delete, and back into place after the new pkg is
installed).
This needs a lot more work to TRT, but it's a first start.
because neither "ypxfr" nor the YP bits of "libc" can handle IPv6. If the
hostname is used there will be an automatic fallback to IPv4 which makes
server pushed work again in mixed IPv4 and IPv6 environments. This fixes
bin/11755 by myself.
