xfer rate stuff, but i never completed the changes that didn't need it
set).
fixes a coredump noticed on current-users@ by Chan Yiu Wah <c5666305@hkstar.com>
features:
---------
* transfer rate throttling with the new `rate' command. syntax:
rate direction [max [incr]]
where direction is `all', `get' or `put'.
if max is not supplied, the current settings are displayed.
if max is supplied, then transfers in the given direction will
be throttled to this value.
if incr is supplied, the increment for the `on-the-fly' scaling
will be set to that, otherwise `1024' is used.
currently implemented for binary get, binary put, and url fetches.
not yet supported for ascii get or put, or local file copies.
* on-the-fly scaling of the throttle based on signals:
- SIGUSR1 raises the throttle rate by the increment for that direction
- SIGUSR2 lowers the throttle rate by the increment for that direction
* -T dir,max[,incr] option to set rate from the command line
* `k', `m', `g' suffix support for bytecounts in the `hash', `rate',
`rcvbuf' and `sndbuf' commands)
bug fixes and code mods:
------------------------
* fix up ftp_login() so that ruserpass() is always called, even for
command-line url fetches.
* implement strsuftoi(), which parses a given number into a int with
suffix support. replaces getsockbufsize()
* implement parserate(), which does the argv parsing for -T and rate
* save and restore errno in signal handlers (may not be necessary, but
it doesn't hurt)
notes:
------
the rate command has had reasonable testing, but I'd like feedback
if it doesn't do the right thing, especially from people on slower
(i.e, modem) links.
I haven't tested the rate throttle against a http server which does
`transfer-encoding: chunked' because I couldn't find a server to
test against.
transfer stats are printed when verbose is enabled even if ftp is compiled
with -DNO_PROGRESS/-DSMALL. this adds back some feedback that existed
in traditional ftp that was lost when i added the progress meter. (noticed
when ftp-ing from the -DSMALL version of ftp that's on the 1.4 boot floppy)
NO_PROGRESS. -DSMALL still implies all of those. progress meter support
isn't necessary for the smallest possible ftp client, but it adds very
little space and makes users' lives much better. Therefore, it should
be enabled for installation media if at all possible.
the latter only seemed to work for TIS Gauntlet and not TIS fwtk.
thanks to simonb@netbsd.org for testing this. fixes [bin/5556].
* if EOF (e.g, ^D) is entered at a username/password/account prompt which
happens to use fgets(), exit the login rather than treating EOF as CR.
* don't use the comma operator where separate statements are valid
* always use snprintf to copy stuff into malloced buffers, just in case
typos creep in and mean that the buffer ends up being overflowed
elements, which are used for the initial authentication attempt (if
requested by the server). in the case of $http_proxy, use the values
for proxy authentication.
.netrc entry exists for the host with a valid user. noted by
Frederick Bruckman <fb@enteract.com> in [bin/7477]
* fix a minor memory leak associated with calling ruserpass() with
an empty user, password, or account.
- the path is split on `/', and each directory is CWD-ed into separately.
(from [standards/7484] by Alan Barrett <apb@iafrica.com>)
- support a trailing `;type=X' suffix, where X is a,i, or d. (d isn't
implemented, but it is recognised)
- the only non-compliant behaviour is that empty directories sections
(e.g `//') aren't run as `CWD ' - as a lot of ftpds don't like that.
Instead, treat this as a no-op.
* don't support globbing for ftp urls, since that's technically not
RFC compliant.
* fix a couple of man-page nits
* $FTPMODE is the documented way in ftp(1) to force passive, active,
gate-ftp, or autodetect.
* AFAIK, we haven't shipped any previous release with pftp in /usr/bin
* no link was made for gate-ftp (and i don't think that makes sense in
/usr/bin either)
* even though the link isn't made, the support for pftp, gate-ftp, and
the `-p' flag should remain since people may depend on having their
own link (e.g, ~/bin/pftp -> /usr/bin/ftp) or aliases which use
these things. it doesn't hurt to leave argv[0] checking in ftp's main().
* fix fetch_ftp() so that hcode parsing is not done for file:// urls
(a } in the wrong place, and code at the wrong indent level...)
* change outfile to being a global (so it gets correctly reset)
* change parse_url to not remove leading '/' for non ftp urls.
whilst this is not totally rfc1738 compliant, other code kinda
assumes this is the case, and it doesn't hurt
* parse_url()
- only look for user[:pass] for an ftp url (per rfc1738)
- strip leading /'s in an ftp url. (almost per rfc 1738)
* fetch_url()
- decode a copy of the path and use that to build local filenames
- send port in http Host: header (suggested by cgd@netbsd.org)
* fetch_ftp()
- url_decode() the user, pass and path
- fix splitting of path into dir & file (partially from [bin/7073])
- don't bother caching the last host; it can cause problems when
using ftp:// transfers, or when the user changes between xfers
* improve documentation of auto-fetched url arguments (especially regarding
escape sequences in ftp:// urls)
* some whitespace & copyright updates
this should fix [bin/7073] William O Ferry <woferry@warp.wofme.com>,
as well as the metaissues raised in that PR.
calculations need to occur to prevent the calculated xfer rate from being
too fast if ftp(1) is backgrounded for a while during the xfer.
bug found by thor lancelot simon <tls@netbsd.org>.
* implement ftpvis(), which \ quotes the following in the given string:
SP, TAB, \, CR, "
* use ftpvis() in complete_ambiguous(), to escape characters in a word
which would confuse makeargv()/slurpstring().
* set setvbuf(ttyout, NULL, _IOLBF, 0) and remove a bunch of fflush(ttyout).
* use fwrite() instead of write() for progressmeter (don't intermix stdio
with non stdio ops)
a response, and return the base64 encoded response. only the `Basic'
scheme is supported.
* implement base64_enc(), which base64 encodes the given buffer
* add support for http response codes 401 "authorization required" and 407
"proxy authorization required", using auth_url(). [requested by veego@]
do_fetch() - which retrieves one file calling fetch_url() or fetch_ftp()
as necessary.
* don't http redirect more than 5 times for a given url
* send `User-Agent: NetBSD-ftp/1.4' header in http requests. (suggested
by Christoph Badura <bad@ora.de>)
* cleanup http return code parser, and add support for:
- 300 `Multiple Choices' - but only if the server returns a
preferred url in a Location: header because i'm *not* adding
a html parser to provide the user with options.
- 305 `Use Proxy [given in Location: header]'. (XXX: not tested)
* support http redirects to non-proxied ftp://urls. (bug discovered by
Chris Demetriou <cgd@netbsd.org>)
* auto-login to an ftp site (using the FTP protocol) if an ftp://host/dir/
style url is given and ftp_proxy is set. whilst this is less orthoganol
with other ftp://host/file urls it's *much* more convenient.
1. changed to use timegm(3) on NetBSD and
2. supply alternative code for other Unix-like OSs
(NetBSD ftp shall be portable for some reasons :-).
This fix closes PR #6448.
NOTE: This should be fixed again if a portable UTC to time_t
conversion method is specified in some standards.
* in autofetch mode, don't attempt to 'cd /' on first xfer (only on
subsequent xfers), as some ftp daemons don't permit that.
[noted by dbj@netbsd.org].
XXX: code could be smarter (but a lot more complicated to boot); this
works for now.
* fix up progress meter display for files once > ~20.47MB has been
transferred. (noted by darren reed)
* use xsignal() for SIGWINCH, so window resizes don't abort xfers on
drainbamaged systems.
* don't assume that TIOCGWINSZ returns a non zero width.
* support $no_proxy, which is a comma or space separated list of
host[:port] elements for which proxying is to be disabled.
(asked for by cgd in [bin/5027])
* if $FTPANONPASS is defined, use that as the anon ftp password
(instead of "`whoami`@")
* allow http URL's without a filename as long as an output file
is specified.
other stuff:
* implement parse_url(), which breaks up a URL into its bits, and use.
* simplify url_get() and auto_fetch() to use parse_url() and to not
modify the supplied URL or a copy of it.
* implement xmalloc() and xstrdup(); error-checked malloc()/strdup()
* add more consistency to messages, quoting strings in output as `%s'
* parse Last-Modified: HTTP headers for date, and change time of
retrieved file if parsing succeeded
* support file:/// and file://localhost/ URLs
* in url_get(), re-write byte moving code to consistently use fread()
and fwrite(), and check errors at end with ferror()
* add about:* easter-egg (#ifndef SMALL :-)
remove server from using a persistent connection, which speeds
up such requests.
* support http 301 and 302 redirects
* rewrite guts of url_get() to use fparseln() et al instead of
read(s,&p,1)... enables each in the header to be parsed
as necessary
* rename login to ftp_login, to remove conflict with util.h::login
* cleanup verbose messages during http proxy requests
which in turn can allow the use of larger TCP windows. This is a work in
progress; there is not yet support for specifying global defaults or
user prefrences on a host/network basis.
- ensure hostname from gethostname() is nul-terminated in all cases
- minor KNF
- use MAXHOSTNAMELEN over various other values/defines
- be safe will buffers that hold hostnames
* default to passive with active fallback. $FTPMODE modifies this behaviour.
-A forces active connection.
* support '-o outfile' for auto-fetched files. outfile can be a file,
`-' (for stdout), or '|command' (to output each file through command).
* support '-r waittime', which retries the connection after waittime seconds
if it fails.
* fix 'page file' when restart is non-zero.
* try all ip-addresses of a host in a http fetch (as the normal ftp fetch
does).
XXX: a ``broken pipe'' error sometimes occurs with -o '|command';
i haven't tracked this down yet.
* in command completion, append a space to a definite match
* in local path completion, append a space or / to a definite match,
depending if the path is a non-directory or directory respectively
(nothing added to remote path completion yet, until a simple sane method
of determining if the path is a directory or not is available).
SIGINFO and at the end of the transfer):
* display time taken as [[x days?] hh:]mm:ss
* display ETA as [hh:]mm:ss
* print ' (stalled)' if the transfer is stalled
doesn't use garbage for the username. from "Soren S. Jorvang" <soren@t.dk>
in [bin/4559]
* use in_port_t for ports, and USHRT_MAX instead of 0xffff
(from millert@openbsd.org)
* use `NULL' instead of `(.... *)0' where appropriate.
where a remote completion or `mget' would confuse the client a `restart'
had been issued beforehand. now, `restart' is remembered until an operation
that can actually use it is invoked.
* in sendrequest(), don't reset restart_point upon entry. fixes `restart'
for `put' operations.
* if `restart' is invoked with no arguments, print current setting instead
of displaying a usage
* consistently use printf("%qd", (long long)restart_point) when displaying
restart_point
* use strto[lq]() instead of atol() when parsing `mark' and `restart' values
* remove unnecessary strlen()s when result of previous snprintf() will do
* replace a few malloc()/strcpy()s with strdup()s
* use SECSPERHOUR instead of '3600'
* Use an int, not "union wait".
* Move the "parsed_url" label so that the next statement is not an "else" --
some ansi C compilers don't like it the old way (SGI's for example).
* Deal with the possibility of getlogin(2) failing.
* Don't error out if the remote server doesn't support the "MDTM" command.
* don't interpret '-' or '|' when a local filename is determined from
the remote name (i.e, in mget, and in get with only one argument).
This is implemented using an extra argument to recvrequest().
Fixes a major security hole.
* clean up memory leak when using globulize()
* clean up a couple of comments
* fix wording in TNF copyright
features added:
* support for TIS fwtk gate-ftp servers:
* read defaults from $FTPSERVER && $FTPSERVERPORT
* start in gate-ftp mode if invoked as 'gate-ftp'
* toggle or set with 'gate [host [port]]'
* cleanup for WARNS=1 (including some ugly '(void)&var' bits wrapped in
#ifdef __GNUC__ to shut up gcc warnings WRT setjmp/longjmp)
* use strtol() instead of atoi(), and more extensively check result of
conversion
* use u_int16_t instead of short or int for TCP port addresses
auto-login ftp URL is used when $ftp_proxy is defined. It now prints:
Auto-login using ftp URLs isn't supported when using $ftp_proxy
Should solve rest of [bin/3643].
Whilst this is inconsistant with the behaviour when $ftp_proxy isn't
defined, the following constraints apply:
* it's not possible to support ftp URL auto-login when $ftp_proxy is
defined, since it uses http not ftp, and you can't `login' to http
servers; fudging this would require a major rewrite of ftp anyway)
* silently ignoring $ftp_proxy and not using it if an ftp auto-login
URL is given is bad user interface design)
* mrg & others will harrass me if I remove support for autologin ftp URLs
when $ftp_proxy isn't defined, even though it made the behaviour
consistant whether $ftp_proxy was set or not.
* differentiate between being connected, and being logged in
* cleanup some text messages
* support username & password ftp URLs (ftp://user:pass@host/) in non-proxy
situations; assume proxy supports it for proxy situations.
* cd to / before performing any autofetch transfers
* use strncasecmp in URL parsing. fix from <Todd.Miller@courtesan.com>
* add "more" & "less" as synonyms for "page"
* move editline setup code into controlediting(), and call appropriately.
only setup setup terminal if going into interactive mode. inspired
by Todd Miller <todd.miller@courtesan.com>
* only echo "Passive mode" in verbose mode; scripts that use ftp
may get unwanted output otherwise
* disable progress bar and modification time preservation when
retreiving to a non-regular files. fixes progress bar getting in
way of "get file /dev/tty"
* setup el_init() et al if editing is set, not if fromatty.
TODO: migrate this to a function, and call if editing is turned on later
in the session. also implement edit_cleanup if editing is turned off
* call el_set() after setting SIGWINCH handler. This fixes the problem
when suspending in a non-cbreak shell (e.g, csh) would trash your tty mode.
* reset interactive mode correctly in auto_fetch() mget mode
* support remglobbing of auto_fetch arguments
* new flag - '-e'; disable editing
* "page file" == "get file |${PAGER-less}"
Bugfixes/cleanup:
* consistently use a trailing '.' on messages
* code cleanup, including buffer overrun fixes, use puts
and putchar in places, etc (inspired by OpenBSD mods)
* disable progress bar when local-file is a pipe or '-'
* skip \r in http headers
* fix remote ftpd slash bug more elegantly (so it works with ////)
* abort_remote(): check if cout==NULL before using it. should fix [bin/3273]
* fixed up cosmetic problems when complete_remote() generated errors from the
remote server (such as "no files found", "login with user and pass", ...)
done by adding extra argument to remglob(), which is a pointer to an error
buffer to put messages in rather than printing to stdout.
barring any more little things people want added ...]
New features:
* progressmeter is now asynchronous, so "stalled" transfers can be
detected. "- stalled -" is displayed instead of the ETA in this case.
When the xfer resumes, the time that the xfer was stalled for is
factored out of the ETA. It is debatable whether this is better than
not factoring it out, but I like it this way (I.e, if it stalls for 8
seconds and the ETA was 30 seconds, when it resumes the ETA will still
be 30 seconds).
* verbosity can be disabled on the command line (-V), so that in auto-fetch
mode the only lines displayed will be a description of the file, and
the progress bar (if possible)
* if the screen is resized (and detected via the SIGWINCH signal), the
progress bar will rescale automatically.
Bugs fixed:
* progress bar will not use the last character on the line, as this can
cause problems on some terminals
* screen dimensions (via ioctl(TIOCWINSZ)) should use stdout not stdin
* progressmeter() used some vars before initialising them
* ^D will quit now. [fixes bin/3162]
* use hstrerror() to generate error message for host name lookup failure.
* use getcwd instead of getwd (it should have been OK, but why tempt fate?)
* auto-fetch transfers will always return a positive exit value upon failure
or interruption, relative to the file's position in argv[].
* remote completion of / will work, without putting a leading "///".
This is actually a bug in ftpd(1), where "NLST /" prefixes all names
with "//", but fixing every ftpd(1) is not an option...
tron