Commit Graph

38 Commits

Author SHA1 Message Date
chs a106161b5a add spaces for KNF. confirmed to produce identical objects. 2001-11-23 21:44:25 +00:00
perseant 4e3fced95b Merge the short-lived perseant-lfsv2 branch into the trunk.
Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default.  Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
  matched to convenient physical characteristics of the partition (e.g.,
  stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
  non-512-byte-sector devices.  In theory fragments can be as large
  as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
  doesn't get old data and think it's new.  Roll-forward is enabled for
  v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
  is not yet implemented, but can be without further non-backwards-compatible
  changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
  that is, the inode is never written *just* because atime was changed.
  Because of this the inodes remain near the file data on the disk, rather
  than wandering all over as the disk is read repeatedly.  This speeds up
  repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
  longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
  I need to look more closely to make sure that the times are only updated
  during write(2) and friends, not after-the-fact during a segment write,
  and certainly not by the cleaner.
2001-07-13 20:30:18 +00:00
cgd 1a1dca038e replace \<space(s)><newline> (wrong!) with \<newline> 2000-12-20 00:24:23 +00:00
perseant 31fc62d4e9 Correct accounting of lfs_avail, locked_queue_count, and locked_queue_bytes.
(PR #11468).  In the case of fragment allocation, check to see if enough
space is available before extending a fragment already scheduled for writing.

The locked_queue_* variables indicate the number of buffer headers and bytes,
respectively, that are unavailable to getnewbuf() because they are locked up
waiting for LFS to flush them; make sure that that is actually what we're
counting, i.e., never count malloced buffers, and always use b_bufsize instead
of b_bcount.

If DEBUG is defined, the periodic calls to lfs_countlocked will now complain
if either counter is incorrect.  (In the future lfs_countlocked will not need
to be called at all if DEBUG is not defined.)
2000-11-17 19:14:41 +00:00
perseant a07c936a59 Remove debugging code that accidentally went in with yesterday's commit. 2000-11-13 00:24:30 +00:00
perseant c4c7b2adbb Do not needlessly dirty segment table blocks during lfs_segwrite,
preventing needless disk activity when the filesystem is idle.  (PR #10979.)
2000-11-12 07:58:36 +00:00
perseant a477e1b98b Cast back to int32_t in LFS_EST_BFREE and LFS_EST_RSVD macros, for
consistency with their arguments.

Change the debugging printf in lfs_reserve to match, and enclose it in
#ifdef DEBUG.

Tested on alpha, arm32, sparc.
2000-09-13 00:07:56 +00:00
perseant 9c7f8050f4 Various bug-fixes to LFS, to wit:
Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
  for writing.  Writes to the filesystem first reserve a maximum amount
  of blocks before their write is allowed to proceed; after the blocks
  are allocated the reserved total is reduced by a corresponding amount.

  If the lfs_reserve function cannot immediately reserve the requested
  number of blocks, the inode is unlocked, and the thread sleeps until
  the cleaner has made enough space available for the blocks to be
  reserved.  In this way large files can be written to the filesystem
  (or, smaller files can be written to a nearly-full but thoroughly
  clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
  is now merely a fs-creation parameter used to compute dlfs_avail and
  dlfs_bfree (and used by fsck_lfs(8) to check their accuracy).  Its
  former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
  This prevents a panic, but, if the cleaner is feeding the filesystem
  the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
  btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
  lfs_markv".  Previously it had meant "clean N segments though N calls
  to lfs_markv, before looking again to see if more need to be cleaned".
  The new behavior gives better packing of direct data on disk with as
  little metadata as possible, largely alleviating the problem that the
  cleaner can consume more disk through inefficient use of metadata than
  it frees by moving dirty data away from clean "holes" to produce
  entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
  of dirty data back to disk", rather than its former meaning of "read
  as many segments as necessary to free N segments worth of space".  The
  new meaning, combined with the new -n behavior described above,
  further aids in cleaning storage efficiency as entire segments can be
  written at once, using as few blocks as possible for segment summaries
  and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
  to error, and not attempt to clean them until they are entirely free
  of dirty blocks.  This prevents the case in which a cleanerd running
  with -n 1 and without -b (formerly the default) would spin trying
  repeatedly to clean a corrupt segment, while the remaining space
  filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
  including the changes mentioned here (in particular, the -b and -n
  flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
  lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
  cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
  the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
  lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]
2000-09-09 04:49:54 +00:00
perseant 988a012d50 Change dlfs_dmeta and dlfs_avail to signed quantities, to prevent
underflow errors, visible in userland as impossibly high values
returned from df(1).
2000-09-09 04:18:28 +00:00
perseant 90b9d9b502 Clean up accounting of lfs_uinodes (dirty but unwritten inodes).
Make lfs_uinodes a signed quantity for debugging purposes, and set it to
zero as fs mount time.

Enclose setting/clearing of the dirty flags (IN_MODIFIED, IN_ACCESSED,
IN_CLEANING) in macros, and use those macros everywhere.  Make
LFS_ITIMES use these macros; updated the ITIMES macro in inode.h to know
about this.  Make ufs_getattr use ITIMES instead of FFS_ITIMES.
2000-07-05 22:25:43 +00:00
perseant bc303e52cb Fix errors observed while trying to fill the filesystem with yesterday's
fixes:

- Write copies of bfree and avail in the CLEANERINFO block, so the
  cleaner doesn't have to guess which superblock has the current
  information (if indeed any do).

- Tighten up accounting of lfs_avail (more needs to be done).

- When cleansing indirect blocks of UNWRITTEN, make sure not to mark
  them clean, since they'll need to be rewritten later.
2000-07-04 22:30:37 +00:00
perseant ef2da50400 Allow the number of free segments reserved for the cleaner to be
parametrized in the filesystem, defaulting to MIN_FREE_SEGS = 2 but set
to something more reasonable at newfs_lfs time.

Note the number of blocks that have been scheduled for writing but which
are not yet on disk in an inode extension, i_lfs_effnblks.  Move
i_ffs_effnlink out of the ffs extension and onto the main inode, since
it's used all over the shared code and the lfs extension would clobber
it.

At inode write time, indirect blocks and inode-held blocks of inodes
that have i_lfs_effnblks != i_ffs_blocks are cleansed of UNWRITTEN disk
addresses, so that these never make it to disk.
2000-07-03 01:45:46 +00:00
perseant 39b86955ed Fixes associated with filling an LFS:
Change the space computation to appear to change the size of the *disk*
rather than the *bytes used* when more segment summaries and inode
blocks are written.  Try to estimate the amount of space that these will
take up when more files are written, so the disk size doesn't change too
much.

Regularize error returns from lfs_valloc, lfs_balloc, lfs_truncate: they
now fail entirely, rather than succeeding half-way and leaving the fs in
an inconsistent state.

Rewrite lfs_truncate, mostly stealing from ffs_truncate.  The old
lfs_truncate had difficulty truncating a large file to a non-zero size
(indirect blocks were not handled appropriately).

Unmark VDIROP on fvp after ufs_remove, ufs_rmdir, so these can be
reclaimed immediately: this vnode would not be written to disk again
anyway if the removal succeeded, and if it failed, no directory
operation occurred.

ufs_makeinode and ufs_mkdir now remove IN_ADIROP on error.
2000-06-27 20:57:11 +00:00
perseant 9139045320 Protect inode free list with seglock, instead of separate lock, so that
the head of the inode free list (on the superblock) always matches the
rest of the free list (in the ifile).

Protect lfs_fragextend with seglock, to prevent the segment byte count
fudging from making its way to disk.

Don't try to inactivate dirop vnodes that are still in the middle of
their dirop (may address PR#10285).
2000-06-06 20:19:14 +00:00
perseant 61861bace5 update for IN_ACCESSED changes 2000-05-31 01:40:01 +00:00
perseant 29f616cebe Prevent dirops from getting around lfs_check and wedging the buffer cache.
All the dirop vnops now mark the inodes with a new flag, IN_ADIROP, which
is removed as soon as the dirop is done (as opposed to VDIROP which stays
until the file is written).  To address one issue raised in PR#9357.
2000-05-27 00:19:52 +00:00
perseant f0728fdce1 Change the sementics of the last parameter from a boolean ("waitfor") to
a set of flags ("flags").  Two flags are defined, UPDATE_WAIT and
UPDATE_DIROP.

Under the old semantics, VOP_UPDATE would block if waitfor were set,
under the assumption that directory operations should be done
synchronously.  At least LFS and FFS+softdep do not make this
assumption; FFS+softdep got around the problem by enclosing all relevant
calls to VOP_UPDATE in a "if(!DOINGSOFTDEP(vp))", while LFS simply
ignored waitfor, one of the reasons why NFS-serving an LFS filesystem
did not work properly.

Under the new semantics, the UPDATE_DIROP flag is a hint to the
fs-specific update routine that the call comes from a dirop routine, and
should be wait for, or not, accordingly.

Closes PR#8996.
2000-05-13 23:43:06 +00:00
perseant 737dec82d6 Change the way LFS does block accounting, from trying to infer from the
buffer cache flags, to marking the inode and/or indirect blocks with a
special disk address UNWRITTEN==-2 when a block is accounted for.  (This
address is never written to disk, but only used in-core.  This is essentially
the same method of block accounting as on the UBC branch, where the buffer
headers don't exist.)  Make sure that truncation is handled properly,
especially in the case of holey files.

Fixes PR#9994.
2000-05-05 20:59:20 +00:00
perseant dac789be22 Changes to stabilize LFS. The first two of these should also apply to the
1.4 branch.

* Use a separate per-fs lock, instead of ufs_hashlock, to protect the Inode
  free list.  This seems to prevent the "lockmgr: %d, not exclusive lock holder
  %d, unlocking" message I was mis-attributing last night to an unlocked vnode
  being passed to vrele.

* Change calling semantics of lfs_ifind, to give better error reporting:
  If fed a struct buf, it can report the block number of the offending inode
  block as well as the inode number.

* Back out rev 1.10 of lfs_subr.c, since the replacement code was slightly
  uglier while being functionally identical.

* Make lfs_vunref use the same free list convention as vrele/vput, so that
  vget does not remove vnodes from a hash list they are not on.
2000-01-19 00:03:04 +00:00
perseant fa6a733240 In lfs_bwrite, don't mark buffers dirty if lfs is mounted read-only.
(Previously buffers could be marked dirty by the cleaner, and possibly by
other means.)

Also check for softdep mount in vfs_shutdown before trying to bawrite
buffers, since other filesystems don't need it and lfs doesn't bawrite.
(This fragment reviewed by fvdl.)

Partially addresses PR#8964.
1999-12-15 07:10:32 +00:00
simonb 3238e654f9 Use an explicitly sized type (u_int32_t) for inode numbers in the super
block instead of ino_t.  Reviewed by Konrad Schroder.
1999-12-08 23:17:31 +00:00
perseant 3c63d0830f Address ufs_hashlock/ufs_ihashins protocol bug, discovered while doing a
post-mortem of a production machine.  Also, take the active dirop
count off of the fs and make it global (since it is measuring a global
resource) and tie the threshold value LFS_MAXDIROP to desiredvnodes.
1999-11-06 20:33:05 +00:00
perseant a818a47169 Minor changes to the segment live bytes calculation. In particular, fixed
a bug in fragment extension that could run the count negative.  Also, don't
overcount for inodes, and don't count segment summaries.  Thus, for empty
segments the live bytes count should now be exactly zero.
1999-06-15 22:25:41 +00:00
perseant 88207e0cba Fixed lfs_update (and related functions) so that calls from lfs_fsync
will DTRT with vnodes marked VDIROP.  In particular, the message
"flushing VDIROP" will no longer appear, and the filesystem will remain
stable in the event of a crash.

This was particularly a problem with NFS-exported LFSes, since fsync
was called on every file close.
1999-06-01 03:00:40 +00:00
perseant e6ef2d67dd clean up unused/required #ifdefs 1999-03-25 21:39:18 +00:00
perseant d1afff4165 Move dlfs_pad to the end of struct dlfs (after the pad), for upward
compatibility.
1999-03-17 18:38:34 +00:00
perseant c5fb3ffb03 Fix pad on lfs.h so it is really 512 bytes, as advertized 1999-03-17 16:49:00 +00:00
perseant 1b8f5ea3c3 New sources should leave the LFS in a more-or-less working state. Changes
include:

	- DIROP segregation is enabled, and greater care is taken
	  to make sure that a checkpoint completes.  Fsck is not
	  needed to remount the filesystem.
	- Several checks to make sure that the LFS subsystem does not
	  overuse various resources (memory, in particular).
	- The cleaner routines, lfs_markv in particular, are completely
	  rewritten.  A buffer overflow is removed.  Greater care is taken
	  to ensure that inodes come from where lfs_cleanerd say they come
	  from (so we know nothing has changed since lfs_bmapv was called).
	- Fragment allocation is fixed, so that writes beyond end-of-file
	  do the right thing.
1999-03-10 00:20:00 +00:00
pk 37109879dc PR#6032: define fixed sized on-disk superblock structure. 1998-09-11 21:27:12 +00:00
fvdl e5bc90f40c Merge with Lite2 + local changes 1998-03-01 02:20:01 +00:00
is ebe1c82eda Make the struct lfs 512 bytes long on 32bit machines whose compiler doesn't
align 32bit integers. Use explicit sized typing at some other places.

XXX This still won't fix lfs for 64bit machines, as we have some
assumptions about sizeof(pointer)=sizeof(u_int32_t) in here, and (if I
looked right) a misaligned u_int64_t. The right fix (to cite cgd) will
be to seperate on-disk-representation from in-core, but I don't have
the time (at the moment) to do this.
1996-12-05 19:01:46 +00:00
christos 7bd9e243f3 lfs prototypes 1996-02-09 22:28:45 +00:00
mycroft 1b682c968a Add RCS ids where missing. 1994-12-21 20:00:18 +00:00
mycroft a63cb01c7d Sync with CSRG. 1994-12-14 13:03:35 +00:00
mycroft a4aea8ad25 Round struct lfs to 512 bytes. 1994-11-17 16:58:41 +00:00
cgd f0c1138373 update for new syscall args description mechanism, and deal safely
with wider types.
1994-10-20 04:20:55 +00:00
cgd fccfa11af5 New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD' 1994-06-29 06:39:25 +00:00
mycroft 264b874c14 Update to 4.4-Lite fs code, with local changes. 1994-06-08 11:41:58 +00:00