- space problems in casts
- don't cast 0, use NULL
- don't have long lines
- don't cast malloc
- use warn/warnx instead of fprintf
- fix bug where a failed mount_mfs would cause the program to continue.
- pass lint
- Always attempt mfs dev if missing /dev/console
- Save and run both MAKEDEV and MAKEDEV.local
- After creating the mfs dev, mknod() and freopen() /dev/console.
- If MAKEDEV is missing do not fail out early, but still create the mfs,
/dev/console.
- If we hit any errors fail out to single user.
This change _only_ affects systems booting without /dev/console.
on the command line with the -u, -g and -p options respectively.
The old malloc() replacement (and related functions) didn't work with
the libc get{pw,gr}nam() functions so it was removed and a separate
memory allocation function added that is only called to allocate the
space for the MFS file system.
Also use a table for the usage flags instead of multiple fprintf()'s
with "if (mfs)" checks sprinkled through.
restricted if -c isn't given; it just confuses a user of newfs (and
the cpg info is printed as part of the display anyway), and prints an
unnecessary warning for mount_mfs.
if there is no command line option and the disklabel gives 0 as the size.
The default used to be 8k, not the default is picked depending on the
file system size. FS < 20M gets 4k, <1G get 8k, and >1G get 16k.
The 16k default was suggested by lukem which FreeBSD is using.
- Change the default cpg from 16 to 65536, which effectively means
that the largest allowable value for cpg will be computed if it's
not specified with -c or in the disklabel for that partition.
This is much more convenient for users.
- Improve the description of the defaults for various options
to define the tape size. Requires the tape driver to either return ENOSPC
at end of media, or 0 when a write is attempted (such as the "early warning"
support in st(4) enabled with "mt eew 1"). From FreeBSD.
- SHLIBDIR Location to install shared libraries if ${USE_SHLIBDIR}
is "yes". Defaults to "/usr/lib".
- USE_SHLIBDIR If "yes", install shared libraries in ${SHLIBDIR}
instead of ${LIBDIR}. Defaults to "no".
Sets ${_LIBSODIR} to the appropriate value.
This may be set by individual Makefiles as well.
- SHLINKDIR Location of shared linker. Defaults to "/usr/libexec".
If != "/usr/libexec", change the dynamic-linker
encoded in shared programs
* Set USE_SHLIBDIR for libraries used by /bin and /sbin:
libc libcrypt libcrypto libedit libipsec libkvm libm libmi387
libtermcap libutil libz
* If ${_LIBSODIR} != ${LIBDIR}, add symlinks from ${LIBDIR}/${LIB}.so*
to ${_LIBSODIR}/${LIB}.so* for compatibility.
* Always install /sbin/init statically (for now)
The net effect of these changes depends on how the variables are set:
1.) If nothing is set or changed, there is no change from the
current behaviour:
- Static /bin, /sbin, and bits of /usr/*
- Dynamic rest
- Shared linker is /usr/libexec/ld*so
2.) If the following make variables are set:
LDSTATIC=
SHLINKDIR=/lib
SHLIBDIR=/lib
Then the behaviour becomes:
- Dynamic tools
- .so libraries used by /bin and /sbin are installed to /lib,
with symlinks from /usr/lib/lib*so to -> /lib/lib*so
where appropriate
- Shared linker is /lib/ld*so
3.) As per 2.), but add the following variable:
USE_SHLIBDIR=yes
This forces all .so's to be instaleld in /lib (with compat
symlinks), not just those tagged by their Makefiles to be.
Again, compat symlinks are installed
- Fix error if first tape was write protected. Fix NetBSD PRs 4754 and 6098.
- Make dump exit codes confirm to manual page.
- Use \a instead of \7 to make noise.
- Fix estimated number of tapes for huge dumps to cartridges.
- Use <sys/queue.h> SLIST_* instead of home-rolled lists.
- Do not exit if unable to read or create /etc/dumpdates.
- Support output (tape) device returning ENOSPC for end-of-media on a write.
Fixes by me:
- Remove unused ddates_in.
- Don't dump core if SIGINFO is received before 1 second has elapsed.
- Only process SIGINFO in current "active" child.
- Don't dump core in -w if dumpdates wasn't readable and ddatev == NULL
- Minor KNF; wrap some lines
past EOM and fail with "short read error". This check was part of
traverse.c::bread(), and was removed when that function was migrated
to rcache.c::rawread() as part of the "read cache" functionality.
This should fix the problem with dump barfing on "short read" when "-r 0",
but I'm still debugging the problem where dump gets to a point and
"does nothing" when the read cache is being used.
tape when a volume is full, and try to reopen the tape drive for 2 mn.
To be used with tape changers which load the next tape when the current one is
ejected.
While I'm there fix eject handling for remote tape.
ifconfig.pppoe* files.
Rename the source directory and files to match the primary purpose
of this utility - probably noone is using this for ISDN now anymore,
as isdnd has simpler ways to acomplish the same.
it conditional on USE_ACORN, and define that if MACHINE is arm26 or acorn32.
This is less than optimal, and maybe we should define it for all ARM systems,
but then disklabel handling across architectures is generally a mess at the
moment.
the label instead. The previous change to use opendisk() prevented
this from working with special fields such as `/dev/wd0b', and this
new method is arguably cleaner anyway. Fixes [misc/14654]
- Don't bother supporting -T for mount_mfs anymore, since it's
superfluous. The option still works (for compatibility reasons), but
it's just ignored and not documented anymore.
- Use an RPM of 10000 instead of 3600 for the fake label (why not :-)
- remove the restriction that filesystem must be a regular file
- don't try and read a disklabel
- use `p' (instead of `h') as the index of the last partition
and don't attempt to do any file name translation (e.g, search in fstab)".
In the non -F case, search for special in fstab. If found, convert fs_spec
to a raw device name. In any case, use opendisk(3) to open the device.
('Does i386 support NTFS5 now?'):
add a note both NTFS4 and NTFS5 are supported
reformat the Limitations a bit
assert the write support is not really useful, and should not be used
add a BUGS section, note missing stuff for the write support to be useful,
and the disk label thing spotted by Christos Zoulas
va_start() in between; reorder code so that we call vsnprintf and then
just fputs that buffer. crank the size of lastmsg whilst we're here
problem noted by Hideo Saito in [bin/14348].
address corresponding to RTA_GATEWAY, or nothing if it doesn't exist.
Modify the exit value of route depending on this, so that one can do
stuff like:
#!/bin/sh
gw=`route -sn get default 2>/dev/null`
if [ -z "$gw" ]; then
echo no default route
exit 1
fi
ping -w1 -c1 $gw >/dev/null 2>&1
if ! route -sn get $gw >/dev/null; then
echo default gateway not responding
exit 1
fi
echo default gateway is at $gw
and check netmask (or prefix) length as well as the destination
address when determining if a route is a "default" or not. This means
that the output from 'route show' will no longer say:
Internet6:
Destination Gateway Flags
default localhost UG
default localhost UG
but instead
Internet6:
Destination Gateway Flags
::/104 localhost UGR
::/96 localhost UGR
which makes much more sense.
- remove dependence on stuff in sys/dev/raidframe
- now rely on <dev/raidframe/*>
- bring in some needed prototypes for local functions
- nuke RF_ASSERT's.
- drag in some needed RF_ERRORMSG's
raidctl.c
- rely on <dev/raidframe/*>
- welcome to the New Way of doing RAIDframe #includes.
(No functional changes.)
Make sure that each va_start has one and only one matching va_end,
especially in error cases.
If the va_list is used multiple times, do multiple va_starts/va_ends.
If a function gets va_list as argument, don't let it use va_end (since
it's the callers responsibility).
Improved by comments from enami and christos -- thanks!
Heimdal/krb4/KAME changes already fed back, rest to follow.
Inspired by, but not not based on, OpenBSD.
aggressive; rework to be a bit less susceptable to round-off error.
now it's likely that the density might not be obtained with a small
filesystem with a large number of inodes (e.g -s 4M -i 1k), but that's
an extremely unlikely corner case that can easily be rectified with
command-line arguments.
fixed provided in private email by Takao Shinohara <shin@sm.sony.co.jp>
should resolve PRs [bin/14049] and [bin/14046]
FreeBSD (three commits; the initial work, man page updates, and a fix
to ffs_reload()), with the following differences:
- Be consistent between newfs(8) and tunefs(8) as to the options which
set and control the tuning parameters for this work (avgfilesize & avgfpdir)
- Use u_int16_t instead of u_int8_t to keep track of the number of
contiguous directories (suggested by Chuck Silvers)
- Work within our FFS_EI framework
- Ensure that fs->fs_maxclusters and fs->fs_contigdirs don't point to
the same area of memory
The new algorithm has a marked performance increase, especially when
performing tasks such as untarring pkgsrc.tar.gz, etc.
The original FreeBSD commit messages are attached:
=====
mckusick 2001/04/10 01:39:00 PDT
Directory layout preference improvements from Grigoriy Orlov <gluk@ptci.ru>.
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.
------
One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.
First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:
1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
test is at wd1. Size of test file system is 8 Gb, number of cg=991,
size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
from Dec 2000 with BUFCACHEPERCENT=35
2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50
You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html
Test Results
tar -xzf ports.tar.gz rm -rf ports
mode old dirpref new dirpref speedup old dirprefnew dirpref speedup
First system
normal 667 472 1.41 477 331 1.44
async 285 144 1.98 130 14 9.29
sync 768 616 1.25 477 334 1.43
softdep 413 252 1.64 241 38 6.34
Second system
normal 329 81 4.06 263.5 93.5 2.81
async 302 25.7 11.75 112 2.26 49.56
sync 281 57.0 4.93 263 90.5 2.9
softdep 341 40.6 8.4 284 4.76 59.66
"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.
------
Algorithm description
The old dirpref algorithm is described in comments:
/*
* Find a cylinder to place a directory.
*
* The policy implemented by this algorithm is to select from
* among those cylinder groups with above the average number of
* free inodes, the one with the smallest number of directories.
*/
A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.
What I mean by a big file system ?
1. A big filesystem is a filesystem which occupy 20-30 or more percent
of total drive space, i.e. first and last cylinder are physically
located relatively far from each other.
2. It has a relatively large number of cylinder groups, for example
more cylinder groups than 50% of the buffers in the buffer cache.
The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.
My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
* Find a cylinder group to place a directory.
*
* The policy implemented by this algorithm is to allocate a
* directory inode in the same cylinder group as its parent
* directory, but also to reserve space for its files inodes
* and data. Restrict the number of directories which may be
* allocated one after another in the same cylinder group
* without intervening allocation of files.
*
* If we allocate a first level directory then force allocation
* in another cylinder group.
*/
My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.
My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.
The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:
int32_t fs_avgfilesize; /* expected average file size */
int32_t fs_avgfpdir; /* expected # of files per directory */
These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.
I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.
Obtained from: Grigoriy Orlov <gluk@ptci.ru>
=====
=====
iedowse 2001/04/23 17:37:17 PDT
Pre-dirpref versions of fsck may zero out the new superblock fields
fs_contigdirs, fs_avgfilesize and fs_avgfpdir. This could cause
panics if these fields were zeroed while a filesystem was mounted
read-only, and then remounted read-write.
Add code to ffs_reload() which copies the fs_contigdirs pointer
from the previous superblock, and reinitialises fs_avgf* if necessary.
Reviewed by: mckusick
=====
=====
nik 2001/04/10 03:36:44 PDT
Add information about the new options to newfs and tunefs which set the
expected average file size and number of files per directory. Could do
with some fleshing out.
=====
- add CHANGEVAL() macro, which does the hard work of changing a parameter
- sort options in usage()
- use .ig [ ... ] .. to comment out sections of man pages (instead
of .\" at the start of each line
in an effort to maintain compatibility with freebsd/openbsd/whatever,
i'm attempting to get the superblock format in sync, and freebsd uses
the int32_t at this position for `fs_pendinginodes'.
if we ever decide to implement fscktime functionality, we'll:
a) make sure to liaise with the other projects to reserve the same
spare field
b) actually implement the code this time ...
(this is also preparing us for other changes, like the new dirpref code)
cylinder groups to work correctly, with minor modifications by me to work
with our FFS_EI code. From the FreeBSD commit message:
The ffs superblock includes a 128-byte region for use by temporary
in-core pointers to summary information. An array in this region
(fs_csp) could overflow on filesystems with a very large number of
cylinder groups (~16000 on i386 with 8k blocks). When this happens,
other fields in the superblock get corrupted, and fsck refuses to
check the filesystem.
Solve this problem by replacing the fs_csp array in 'struct fs'
with a single pointer, and add padding to keep the length of the
128-byte region fixed. Update the kernel and userland utilities
to use just this single pointer.
With this change, the kernel no longer makes use of the superblock
fields 'fs_csshift' and 'fs_csmask'. Add a comment to newfs/mkfs.c
to indicate that these fields must be calculated for compatibility
with older kernels.
Reviewed by: mckusick
- replace the unused fs_headswitch and fs_trkseek with fs_id[2], bringing
our struct fs closer to that in freebsd & openbsd (& solaris FWIW)
- dumpfs: improve warning message when cpc == 0
- fix round-off errors when determining the number of inodes per group,
which often resulted in the total number of inodes in the file system
being less than what the density asked for.
now you might get more inodes than requested for a given density,
rather than less.
- if the new inodes/group is <= 0, ensure that it's at least 1, preventing
a possible division by zero or other wacky problems
- use long long instead of quad_t
- reorder "special" validation to after option parsing
- use getopt(3) instead of homegrown code
- add getnum() to parse and validate a number
- clean up man page
- ansi KNF, WARNS=2
determine the endianness of the `struct fs *o' superblock from o->fs_magic
and set needswap as necessary, rather than trusting the caller to get
it right. invariably, almost every caller of ffs_sb_swap() was calling it
with ns set to the wrong value for ns anyway!
ansi KNF ffs_bswap.c declarations whilst here.
this fixes all sorts of problems when trying to use other-endian file systems,
notably the kernel trying to access memory *way* off, possibly corrupting or
panicing, and userland programs SEGVing and/or corrupting things (e.g,
"fsck_ffs -B" to swap a file system endianness).
whilst the previous rev of ffs_bswap.c (1.10, 2000/12/23) made this problem
worse, i suspect that the problem was always there and previous versions
just happened not to trash things at the wrong time.
FFS_EI should now be a lot more stable.
and exit.
Previously, combinations would produce unintended results, such as
deleting the primary IP on an interface, instead of deleting an specified
alias.
safe (since there's two separate mallocs using sbrk(2) in that case)
XXX: local malloc provided for mfs memory store allocation; need to
investigate if system (phk) malloc can be used instead.
simonb