Ok, ok, a few more words about it: stop holding puffs_cc as a holy
value and passing it around to almost every possible place (popquiz:
which kernel variable does this remind you of?). Instead, pass
the natural choice, puffs_usermount, and fetch puffs_cc via
puffs_cc_getcc() only in routines which actually need it. This
not only simplifies code, but (thanks to the introduction of
puffs_cc_getcc()) enables constructs which weren't previously sanely
possible, say layering as a curious example.
There's still a little to do on this front, but this was the major
fs interface blast.
handle open is requested, it is waited for only if the node was
not previously succesfully opened. The actual wait for the file
handle happens only when the file handle is actually needed (read
or write). This in turn has the effect that reading cached files
will be quick instead of waiting for the file handle from the sftp
server first. The wait previously could be very long if there were
serveral hundred k of outstanding requests in a limited-bandwidth
link.
The code is in some need of serious handholding, but it works, so
I'll leave that as "future work".
the directory which contained the file before a getattr on the file
itself, the locally cached mtime would be updated without invalidating
the kernel page cache. Thus incorrect data would be returned when
the node was read afterwards as the node size wouldn't match the
data length in the page cache.
Fix the problem by making all vattr-setting routines use the same code.
Problem noticed again by jmmv & atf (and again by running atf over
psshfs ... sometimes you're the windshield, sometimes you're the bug)
against cached mtime instead of attrread - attrread can be reset
these days by sending SIGHUP.
Problem noticed by jmmv & atf (well.. namely by using atf through psshfs).
servers. Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs. As discussed with
Greg Oster & others.
for each node. Setting this to a small number can be used to
improve interactive performance on low-bandwidth links when performing
bulk data reads. Of course I could also open separate pipes for
bulk and other, but this was quicker and less intrusive and doesn't
require authenticating twice.
anymore do this if we fail to set size.
The whole lookup procedure should be done in a smarter fashion,
but this is the quickie fix to get things working again.
to signal no references, as that is not currently supported for
node_rename(). The removed node will not immediately be reclaimed,
but we can live with that for now.
While here, factor the removal code a bit to share with remove and
rmdir.
fixes PR kern/36637 by reinoud
portalfs. Uses the same config files etc. as the "regular"
mount_portal, and actually compiles a lot of the code directly from
under src/sbin/mount_portal.
Doesn't yet support concurrent access, but I have a pretty clear
vision on how to neatly fix that.
FORTIFY_SOURCE feature of libssp, thus checking the size of arguments to
various string and memory copy and set functions (as well as a few system
calls and other miscellany) where known at function entry. RedHat has
evidently built all "core system packages" with this option for some time.
This option should be used at the top of Makefiles (or Makefile.inc where
this is used for subdirectories) but after any setting of LIB.
This is only useful for userland code, and cannot be used in libc or in
any code which includes the libc internals, because it overrides certain
libc functions with macros. Some effort has been made to make USE_FORT=yes
work correctly for a full-system build by having the bsd.sys.mk logic
disable the feature where it should not be used (libc, libssp iteself,
the kernel) but no attempt has been made to build the entire system with
USE_FORT and doing so will doubtless expose numerous bugs and misfeatures.
Adjust the system build so that all programs and libraries that are setuid,
directly handle network data (including serial comm data), perform
authentication, or appear likely to have (or have a history of having)
data-driven bugs (e.g. file(1)) are built with USE_FORT=yes by default,
with the exception of libc, which cannot use USE_FORT and thus uses
only USE_SSP by default. Tested on i386 with no ill results; USE_FORT=no
per-directory or in a system build will disable if desired.
be reclaimed from under while we are warming the getattr cache.
Shuffle some code to prevent the effects. Theoretically the race
is still possible, but I don't think it will happen in practice.
In any case, the code could benefit from some more dusting.
getattr are usually still outstanding when we already would like
the result. Instead of issueing another stat which will be serviced
only after all the other entries in the directory, record all the
outgoing readdir getattr buffers and if we encounter an outstanding
request when we need to fetch attrs, do a puffs_framev_framebuf_ccpromote()
wait for it instead of firing off the second query. This shaves
almost 10% off the time for ls -lR.
Also, get rid of the SUPERREADDIR conditional, since it has penetrated
the code quite a bit and the #ifdef SUPERREADDIRs were starting to
look like tagliatelle alla bolognese (n.b. I love how it looks,
but I wouldn't like it either if my tagliatelle alla bolognese
looked like psshfs code). Maybe it should be re-introduced in the
form of a switch?
a bit differently: when reading the directory, store all getattr
caching queries and fire off only when the directory read is
complete. That way the common sequence is not [readdir, lots of
async getattr requests, readdir EOF] but rather [readdir, readdir
EOF, lots of async getattr]. This speeds up ls -lR by about 25%
(on my LAN).
equal, larger, respectively instead of 0/1 for non/equal. This
will allow sorting the buffers for faster matching in libpuffs.
While here, change the name from respcmp to framecmp, as that better
reflects the purpose.
NOTE! there is no obvious way to make compilation fail for file
systems which may already be using this feature (although I don't
think there are any outside our tree, as the feature is two weeks
old). Nevertheless, non-updated file systems will fail very quickly.
reference count goes to 0) reclaim of deleted nodes as opposed to
waiting for the system to start reclaiming the freelist
* combine some nodeflags from different variable to one status variable
correctly, otherwise the file server will attempt a conversion.
Specifically, if the directory bit is not set when changing the
permissions for a directory, a conversion from directory to regular
file would be attempted and naturally it being unsupported the
whole chmod would fail. So supply the file type as part of the
file mode.
PUFFS_KFLAG_WTCACHE. Second, create separate fids for reading and
writing. If opening for read, open a read-only fid and for write
a write-only fid; use these for reading and writing. When the
open-count for a node drops to zero, clunk both. This avoids hitting
the fid limit when accessing large directory hierarchies.
Two problems remain:
* does not take credentials into account, although we can only mount
the remote 9P file server with one set of credentials, so not a
huge worry
* doesn't work for the open/mmap/close/access_memory_window case, but
that will require some further kernel changes
Works, but lots of little things to nibble on:
* fix permissions to work better
* limit the amount of open files required
* do constant folding with psshfs code
* support authentication
etcetc.
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway). Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system. Keep puffs_mount() as a
shortcut for the above two for simple file systems.
Bump development ABI version to 13. After all, it's Friday the 13th.
Watch out! Bad things can happen on Friday the 13th. --No carrier--
been nodetofh translated even if they are not valid on the sftp
server anymore, because some nfs client might still be clinging on
to the file handle we are reclaiming now.
Now, when I say support, I mean "support", due to the limitations
of the backend. File handles are valid only for one session, since
nodes can only be identified by pathnames and pathnames don't (all)
fit into the nfs file handle space. Additionally, we can't detect
if a pathname is completely replaced by another file (if it's done
via some other route that through our mount, of course). But then
again, that's an inherent problem with sshfs even without nfs.
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().
(that's what you get when you copypaste code, a cid with a pin
to burst your bubble, that's what you get for all your troubles, I'll
never copypaste again)
CID 4461
this can happen legally when a file is removed from backing
storage not using this sshfs instance, a readdir is executed for
the parent directory and only then the node gets reclaimed.
* now that there is a mechanism in place which does not require a
pcc to do an sftp transaction, do not yield() in operations where
the final transaction is something where we don't care about the
return value (e.g. close handle). speedup benefit for no cost.
reclaimed nodes hanging until all their children have been reclaimed
and then reclaim everything we can as far up to root as possible.
This is because the file system structures are currently interlinked
in a fashion which would make dotdot lookup based on purely a path
instead of a in-memory node parent member pointer very difficult.
Yes, this deserves a closer look some day.
directory entries already in readdir and caches the results instead
of waiting for each individial getattr from the kernel. For
high-latency links the difference in "ls -l" is quite astounding
and even on my lan "ls -lR" is faster than for nfs in a normal
directory hierarchy (i.e. not one artifically setup to have thousands
of files per directory).
TODO: implement some sort of bandwidth/latency measurement in the
code and enable or disable this option based on than information
(and a command-line flag).
should make this work with the IETF standard some day, also.
* kludge with writes and permissions a bit to be able to flush data
cached in ubc to files which are already with r/o permissions in
the backend
pooka