elapsed time. (suggested by simonb)
- actually display transfer stats after a URL fetch. (bug introduced a
*long* time ago)
- update copyright & version
Fixes [bin/15415] from Takahiro Kambe <taca@sky.yamashina.kyoto.jp>
- Fix bad mode passed by mls() to recvrequest().
Fixes [bin/16642] from <steve.mcclellan@radisys.com>
From Hajimu UMEMOTO, via Mike Heffner of FreeBSD.
(FreeBSD has imported NetBSD's ftp as their ftp client;
Mike is sending back some of their local changes).
- add -N netrc and $NETRC, as methods to select an alternative .netrc file
- cache local user name and home directory for further use
- in mget(), use docase() instead of a local version to do the case
conversion.
after the chunksize (before the \r\n), Apache 1.3.11 puts *multiple*
trailing spaces after the chunksize. I 'm fairly certain that this is
contrary to RFC 2068 section 3.6, but whatever...
Found by David Brownlee <abs@mono.org>
it to a numeric port number
* use parseport() in parse_url() and hookup()
* don't try and lookup the port number using getaddrinfo(), as it's too hard
to separate a failed host name lookup from a failed service name lookup.
this was causing lossage on systems that don't have `http' in services(5)
(such as solaris), but only crept in when we started using getaddrinfo()
unconditionally.
equivalents. name change suggested by Klaus Klein <kjk@netbsd.org>
- change defined(BSD4_4) || HAVE_SIN_LEN tests into HAVE_SOCKADDR_SA_LEN,
and set the latter if BSD4_4 exists
paths. (lukemftp will provide replacements for these on older systems)
* rename __USE_SELECT to USE_SELECT
* rename BSD4_4 to HAVE_SIN_LEN
* replace union sockunion {} with struct sockinet {}, and modify the code
accordingly. this is possibly more portable, as it doesn't rely upon the
structure alignment within the union for our own stuff.
(XXX: haven't tested the ipv6 stuff)
#define NOQUAD ! NOQUAD
------- ------ - ------
QUADF "%ld" "%lld"
QUADFP(x) "%" x "ld" "%" x "lld"
QUADT long long long
STRTOL(x,y,z) strtol(x,y,z) strtoll(x,y,z)
(from RFC 2389).
add support for MLST & MLSD (machine parseble listings) with 'mlst', 'mlsd'
and 'pmlsd' (mlsd |$PAGER) commands. (from draft-ietf-ftpext-mlst-11)
rename remotesyst() to getremoteinfo(), and modify to parse the result from
FEAT (if supported), and take into account the support for the various
extensions such as MDTM, SIZE, REST (STREAM), MLSD, and FEAT/OPTS.
put each feature into one of the following categories:
- known to work (explicit FEAT)
- unknown but assume works until explicit failure, when it's
then tagged as `known not to work'.
- known not to work (FEAT succeeded but didn't return anything,
or was unknown and then explicit failure)
assign results into features[] matrix.
add support to getreply() so that an optional callback will be called
for each line received from the server except for the first and last.
this is used in FEAT (and MLST) parsing.
modify various commands to check if REST (STREAM), MDTM and SIZE are
explicitly or implicitly supported before using.
fix `syst' when verbose is off.
minor knf (indent goto labels by one space, etc).
simply various command usage handlers by assuming that argv != NULL except
for quit() and disconnect().
call remotesyst() only when login has been successful
some servers don't let you run SYST until you've successfully logged in.
* in fetch_ftp(), always call setpeer() with autologin disabled, and use
the following ftp_login() to DTRT.
this prevents ftp from trying to login a second time if the first autologin
fails when connecting to a remote site anonymously using autofetch.
* reset unix_proxy and unix_server in cleanuppeer()
* missed a function conversion in the KNF sweep...
* initconn(): use in_addr_t instead of u_int32_t when manipulating IPv6
addresses (and assume anything with ipv6 has in_addr_t; if not, i'll
add an autoconf test for it)
* ai_unmapped(): not all systems have sin_len; so only set #ifdef BSD4_4
* fix some lint
- ftp(1): treats IPv4 mapped destination as IPv4 peer, not native IPv6 peer.
this does not support network with SIIT translator.
- rshd(8)/rlogind(8): rejects accesses from IPv4 mapped peer, to avoid
possible abuse of IPv4 mapped addr (rshd/rlogind use source address-based
auth so it is important to check the condition).
Now that after many years on not caring we find certain popular
ftp servers are starting to obey RFC959 to the letter of the law
and will only return a list of filenames (not directories or
other filetypes) in the output of `NLST', then `LIST' is more useful
in this case. (Note that the aforementioned pedanticness means that
filename completion isn't as useful as it could be...)
Fixes [bin/8937] by David A. Gatwood <dgatwood@deepspace.mklinux.org>
* sanity check a length (otherwise certain bogus responses can crash ftp)
* allow a transfer encoding type of `binary'; certain firewall vendors
return this bogus type...
send Host: directive with RFC2732 bracket notation for IPv6 numeric,
otherwise "host:port" is ambiguous to servers (clarification will be submitted
as update to RFC2732).
typedef void (*sigfunc) __P((int));
and replace use of sig_t and void (*)(int).
certain other OSes define sig_t differently to that (they add extra arguments),
and it causes problems due to function mismatches, etc...
the output of `MDTM'.
obviously the programmer of aforementioned lame ftpd's did something like
"19%02d", tm->tm_year
instead of
"%04d", tm->tm_year + TM_YEAR_BASE
fixes [bin/9289] by jbernard@mines.edu
the word is already complete then return CC_REFRESH so that the higher layer
may append a suffix if necessary. Fix from Launey Thomas <ljt@alum.mit.edu>.
Problem noticed/debugging assisted by giles lean <giles@nemeton.com.au>.
XXX: rate limiting with chunked xfers might not limit correctly (i.e,
the limit may be too high or too low); fixing this is non trivial,
and will probably occur if i ever rototill fetch_url()
* fix bug where the second press of <TAB> on an empty word (i.e, list
all options) may have resulted in an strncmp() against NULL. (detected
by _DIAGASSERT())
for listings of the current working directory; some ftp servers don't
like `NLST .'.
[noted by Giles Lean <giles@nemeton.com.au>]
* recvrequest(): treat remote=="" as remote==NULL when calling command().
(to support the above change)
* support `[user@]' in `[user@]host' and `[user@]host[:][path]'.
[based on idea (and initial code) from David Maxwell <david@fundy.ca>]
* `idle' may be invoked without any args
* reformat some comments
* reformat usage string in program and man page
* call updateremotepwd() after successful login, not after successful connect
* always call setsockopt(, IPPROTO_IP, IP_TOS, ) (et al); using #if
defined(IPPROTO_IP) doesn't work on certain foreign systems where
enums instead of #defines are used...
[noted by Matthias Pfaller <leo@dachau.marco.de>]
EL_RPROMPT support i added to editline(3).
- allow $FTPPROMPT and $FTPRPROMPT to override defaults for the relevant
prompts
- move `%' formatting code from prompt() to expandbuf().
- implement `%.' and `%c', similar to the same % codes in tcsh(1)
(functionality I added to tcsh nearly 6 years ago), except that `%.'
always does `...trailing' and `%c' always does `/<x>trailing'.
- unknown `%foo' codes get printed as `%foo'
to contain the remote working directory.
- add `set prompt', a user configurable prompt. (defaults to `ftp> ').
the following escape characters a la tcsh(1) are supported: %/, %m,
%M, and %n.
- add global var `username'; used by prompt code
- fix a couple of minor memory leaks
- bump version
the provided port is a valid number use that rather than trying to do
getservbyname() against it.
fixes a problem on foreign systems noted by Chuck Silvers <chuq@chuq.com>
- add `usage'; displays the usage of a command.
implemented by calling the c_handler() with argc = 0, argv = "funcname".
- add `passive auto'; does the same as $FTPMODE=auto.
- add `set [option value]'; display all options, or set an option to a value.
- add `unset option'; unset an option.
- add getoptionvalue() to retrieve an option's value, and replace a few
global variables with calls to this.
- implement cleanuppeer(), which resets various bits of state back to
`disconnected'. call in disconnect() and lostpeer().
- support completing on `options'.
- improve recovery after a SIGINT may have closed the connection.
XXX: there's still a couple to fix
other stuff:
- various consistency fixes in the man page.
- ensure that the command usage strings in the code and man page match reality.
- mput/mget: check that the connection still exists before each xfer.
- minor cosmetic changes in confirm().
- set code correctly in sizecmd() and modtime()
- don't need \n in err() strings.
- change lostpeer to take an argument (rather than casting (sig_t)lostpeer
in signal handlers)
- knf and whitespace police.
* simplify main loop (don't need `top' variable any more)
* use a struct sockaddr_in6.sin6_addr for the result from inet_pton(),
rather than u_char buf[16]
* add a few more comments
* confirm(): move from util.c to cmds.c. display mnemonic string in its prompt.
add support for `q' (terminate current xfer), `?' (show help list)
* in various signal handlers, output a linefeed only if fromatty.
* if fgets(stdin) returned NULL (i.e, EOF), clearerr(stdin) because you don't
want future fgets to fail. this is not done for the fgets() in the main
command loop, since ftp will quit at that point.
* unless ftp is invoked with -a, don't retain the anonftp setting between
hosts (`ftp somehost:' sets anonftp, but you don't want that to `stick'
if you close that connection and open a new one).
stage. if this occurs, just call lostpeer() to close the connection. whilst this
might be considered brutal, it's also extremely handy if you're impatient or there's
lossage at the remote end.
don't save the signal mask on some foreign systems.
* ensure signal handlers don't use stdio and do reset errno if they
don't exit with siglongjmp()
* use a common SIGINT handler for {send,recv}request()
(work around editline's override)
* minor cleanup of signal handler (along the lines of similar work in
recvrequest()). the handlers should now be reset everytime the cleanup
handler was callled.
prints out info if bytes > 0). only set the handler if SIGINFO is defined
* hijack SIGQUIT to be the same as SIGINFO (foreign ports have this, and it's
annoying to have SIGQUIT dump core on netbsd when it prints info on other
systems)
* in {recv,send}request(), factor a lot of duplicated code out into a
`cleanup' section at the end
* rework shell() a bit
* implement xsignal_restart(), which only sets the SA_RESTART flag if
specifically requested
* xsignal() is now a wrapper to xsignal_restart(). INFO, USR1, USR2 and WINCH
are restartable, ALRM, INT, PIPE and QUIT are not.
* improve getreply()'s timeout code to take advantage of the above.
other changes:
* improve wording of how globbing works for `classic' URLs (host:path).
suggested by John Refling <johnr@imageworks.com> in relation to PRs
[bin/8519] and [bin/8520]
* always compile in the `edit' command even if NO_EDITCOMPLETE defined.
it's just a no-op in the latter case, which is more consistent to
the users.
* always compile in about: support (i.e, remove NO_ABOUT).
i'm entitled to some vanity in this program...
* clean up some whitespace
the disabled state can be overridden by toggling epsv4.
(I got sick of the errors about EPSV not being supported on almost
every server I connect to. This way we retain support for epsv4, but
it's not so whiny after the first failure...)
to "extern" if it's not set. define GLOBAL to (empty) in main.c.
this effectively moves all the globals into main.c whilst retaining
namespace access to them in other source files.
(global vars in header files confuse foreign linkers)
should be in host order. found/fixed by Matthias Pfaller <leo@dachau.marco.de>
* parse_url(): improve checking of portnum, and add an extra argument to pass
back the parsed portnum to the caller (reduces a bit of code duplication)
* Move the KAME/WIDE copyrights after the BSD/TNFi ones. Since there was
significantly less code added under the former, it's only fair on the latter.
ETA might be over and will appear to drop rapidly, rather than being under
and appear not to change. the original code makes sense when you're testing
by suspending & resuming the client. however, the unfudged number is probably
better in reality, especially for slow spurty networks.
requested by ITOH Yasufumi <itohy@netbsd.org> in [bin/7977]
lpage page local files
pdir as dir, but through your $PAGER
pls as ls, but through your $PAGER
* implement docase() (a la dotrans() et al) and use appropriately, rototilling
some duplicated code
* globulize(): modify to return a pointer to the strdup()ed result in all cases,
and hack the code that calls it to take this into account
* replace strcpy() and strncpy() with strlcpy()
* put(), getit(): use some aptly named local vars instead of argv[...]
* delint
prototype for idle() in <unistd.h> (which i thought was against namespace
and sensibility guidelines, but...)
* consistently use xsignal() instead of signal(). we get known behaviour
in all cases (SA_RESTART), which is good for some borken foreign systems.
* remove signal.h from most files; it's unnecessary now
* fetch_url(): use `long chunksize' instead of ssize_t; it's more portable, and
we're setting chunksize with strtol() anyway
* xsignal(): only use SA_RESTART if it exists. SunOS 4.x doesn't have it
but has the inverse (SA_INTERRUPT). the original function i was inspired
from had this support (lib/signal.c, W. Richard Stevens' `UNP 2nd ed Vol 1').
* remove <termios.h> from util.c; it should be unnecessary now
prototype for idle() in <unistd.h> (which i thought was against namespace
and sensibility guidelines, but...)
* consistently use xsignal() instead of signal(). we get known behaviour
in all cases (SA_RESTART), which is good for some borken foreign systems.
* remove signal.h from most files; it's unnecessary now
* fetch_url(): use `long chunksize' instead of ssize_t; it's more portable, and
we're setting chunksize with strtol() anyway
* xsignal(): only use SA_RESTART if it exists. SunOS 4.x doesn't have it
but has the inverse (SA_INTERRUPT). the original function i was inspired
from had this support (lib/signal.c, W. Richard Stevens' `UNP 2nd ed Vol 1').
* remove <termios.h> from util.c; it should be unnecessary now
it's more portable and more obvious
* remove the mkgmtime() && HAVE_TIMEGM stuff:
a) why should netbsd have to define HAVE_TIMEGM to compile cleanly?
b) foreign compiles of ftp should just be linked with working
timegm function
a more portable version of this ftp client will be released as a 3rdparty
product; no use polluting our code with half-baked attempts...
* abort_remote(): replace borken MIN(4,BUFSIZ) with just BUFSIZ; it
should have been MAX(4,BUFSIZ), but it's probably safe to assume that
BUFSIZ is at least 3... (fix from simonb)
* auth_url(): use the correct variable when calculating a buffer size.
add lots of comments about how to size up the buffers, and add extra
checks to hopefully ensure that there won't be an overflow (unless
someone modifies the length of the sprintf()s).
* as part of the above, slightly rework the way the `*' bar is calculated.
also fixes a display bug when > 160 stars were needed to be printed.
the maximum progress bar width at this time is 256.
* remove some code that checks the port that was #if 0-ed out as part of the
ipv6 migration; it's not going to be used again.
* document the above three commands
* rototill the way the sndbuf and rcvbuf work. remove resetsockbufsize()
* use the appropriate socket buffer size as the size of the buffer that
the read()/write() loops use. speeds up things in some cases.
connection destination, hoping this to help ftpd's behavior with
scoped IPv6 addresses.
I'm not sure if it is the right way, but it is the best way available to us.
LPRT or EPRT command gives no information about which interface (or scope)
to be used for new data connection.
ftp(1): On data connection establishment, warn if scoped address is used.
If peer (ftp daemon) does not handle scoped address, data connection
may not work right.
This seems to be sort of protocol spec hole, not implementation issue.
- just display the hostname:port of the proxy url, rather than the full url.
this prevents someone `shoulder surfing' a proxy username/password
in $http_proxy. [suggested by perry]
- compact verbose notes for http fetchs; now displays
(via host:port, with authorization, with proxy authorization)
with each component being optional.
(and a couple introduced with the ipv6 mods...)
- don't override host with the canonical name; this prevented fetches from
http/1.1 virtual hosts from working if the virtual host was a CNAME.
[noted by bernd]
- call freeaddrinfo() if res was built with getaddrinfo()
the an empty directory (e.g, between `some' and `path' in
`ftp://host/some//path'), then execute `CWD ' (without a path).
This command will probably fail on rfc 959 compliant servers, so
issue a warning in this case and bail. [noted by cgd].
(i wonder if the people who wrote rfc 1738 actually realised that this
requirement appears to contravene the spec for `cwd' in rfc 959 ?)
* replace isurl() with isipv6addr(), and use appropriately. fixes
auto-login with `classic ftp URLs' (e.g, `ftp somehost:')
* cleanup and rework some of the ipv6 stuff in parse_url()
* prevent potential coredump in fetch_ftp() when parsing `;type=X'
* KNF a few lines
* fix a couple of comments
* cleanup the man page a bit
this is mainly for (hypothetical) ftp server which disconnect clients
that use EPSV/EPRT. I've never seen any ftp server like this, but
epsv4 command may be of use when such an ftp server is found.
xfer rate stuff, but i never completed the changes that didn't need it
set).
fixes a coredump noticed on current-users@ by Chan Yiu Wah <c5666305@hkstar.com>
features:
---------
* transfer rate throttling with the new `rate' command. syntax:
rate direction [max [incr]]
where direction is `all', `get' or `put'.
if max is not supplied, the current settings are displayed.
if max is supplied, then transfers in the given direction will
be throttled to this value.
if incr is supplied, the increment for the `on-the-fly' scaling
will be set to that, otherwise `1024' is used.
currently implemented for binary get, binary put, and url fetches.
not yet supported for ascii get or put, or local file copies.
* on-the-fly scaling of the throttle based on signals:
- SIGUSR1 raises the throttle rate by the increment for that direction
- SIGUSR2 lowers the throttle rate by the increment for that direction
* -T dir,max[,incr] option to set rate from the command line
* `k', `m', `g' suffix support for bytecounts in the `hash', `rate',
`rcvbuf' and `sndbuf' commands)
bug fixes and code mods:
------------------------
* fix up ftp_login() so that ruserpass() is always called, even for
command-line url fetches.
* implement strsuftoi(), which parses a given number into a int with
suffix support. replaces getsockbufsize()
* implement parserate(), which does the argv parsing for -T and rate
* save and restore errno in signal handlers (may not be necessary, but
it doesn't hurt)
notes:
------
the rate command has had reasonable testing, but I'd like feedback
if it doesn't do the right thing, especially from people on slower
(i.e, modem) links.
I haven't tested the rate throttle against a http server which does
`transfer-encoding: chunked' because I couldn't find a server to
test against.
transfer stats are printed when verbose is enabled even if ftp is compiled
with -DNO_PROGRESS/-DSMALL. this adds back some feedback that existed
in traditional ftp that was lost when i added the progress meter. (noticed
when ftp-ing from the -DSMALL version of ftp that's on the 1.4 boot floppy)
NO_PROGRESS. -DSMALL still implies all of those. progress meter support
isn't necessary for the smallest possible ftp client, but it adds very
little space and makes users' lives much better. Therefore, it should
be enabled for installation media if at all possible.
the latter only seemed to work for TIS Gauntlet and not TIS fwtk.
thanks to simonb@netbsd.org for testing this. fixes [bin/5556].
* if EOF (e.g, ^D) is entered at a username/password/account prompt which
happens to use fgets(), exit the login rather than treating EOF as CR.
* don't use the comma operator where separate statements are valid
* always use snprintf to copy stuff into malloced buffers, just in case
typos creep in and mean that the buffer ends up being overflowed
elements, which are used for the initial authentication attempt (if
requested by the server). in the case of $http_proxy, use the values
for proxy authentication.
.netrc entry exists for the host with a valid user. noted by
Frederick Bruckman <fb@enteract.com> in [bin/7477]
* fix a minor memory leak associated with calling ruserpass() with
an empty user, password, or account.
- the path is split on `/', and each directory is CWD-ed into separately.
(from [standards/7484] by Alan Barrett <apb@iafrica.com>)
- support a trailing `;type=X' suffix, where X is a,i, or d. (d isn't
implemented, but it is recognised)
- the only non-compliant behaviour is that empty directories sections
(e.g `//') aren't run as `CWD ' - as a lot of ftpds don't like that.
Instead, treat this as a no-op.
* don't support globbing for ftp urls, since that's technically not
RFC compliant.
* fix a couple of man-page nits
* $FTPMODE is the documented way in ftp(1) to force passive, active,
gate-ftp, or autodetect.
* AFAIK, we haven't shipped any previous release with pftp in /usr/bin
* no link was made for gate-ftp (and i don't think that makes sense in
/usr/bin either)
* even though the link isn't made, the support for pftp, gate-ftp, and
the `-p' flag should remain since people may depend on having their
own link (e.g, ~/bin/pftp -> /usr/bin/ftp) or aliases which use
these things. it doesn't hurt to leave argv[0] checking in ftp's main().
* fix fetch_ftp() so that hcode parsing is not done for file:// urls
(a } in the wrong place, and code at the wrong indent level...)
* change outfile to being a global (so it gets correctly reset)
* change parse_url to not remove leading '/' for non ftp urls.
whilst this is not totally rfc1738 compliant, other code kinda
assumes this is the case, and it doesn't hurt
* parse_url()
- only look for user[:pass] for an ftp url (per rfc1738)
- strip leading /'s in an ftp url. (almost per rfc 1738)
* fetch_url()
- decode a copy of the path and use that to build local filenames
- send port in http Host: header (suggested by cgd@netbsd.org)
* fetch_ftp()
- url_decode() the user, pass and path
- fix splitting of path into dir & file (partially from [bin/7073])
- don't bother caching the last host; it can cause problems when
using ftp:// transfers, or when the user changes between xfers
* improve documentation of auto-fetched url arguments (especially regarding
escape sequences in ftp:// urls)
* some whitespace & copyright updates
this should fix [bin/7073] William O Ferry <woferry@warp.wofme.com>,
as well as the metaissues raised in that PR.
calculations need to occur to prevent the calculated xfer rate from being
too fast if ftp(1) is backgrounded for a while during the xfer.
bug found by thor lancelot simon <tls@netbsd.org>.
* implement ftpvis(), which \ quotes the following in the given string:
SP, TAB, \, CR, "
* use ftpvis() in complete_ambiguous(), to escape characters in a word
which would confuse makeargv()/slurpstring().
* set setvbuf(ttyout, NULL, _IOLBF, 0) and remove a bunch of fflush(ttyout).
* use fwrite() instead of write() for progressmeter (don't intermix stdio
with non stdio ops)
a response, and return the base64 encoded response. only the `Basic'
scheme is supported.
* implement base64_enc(), which base64 encodes the given buffer
* add support for http response codes 401 "authorization required" and 407
"proxy authorization required", using auth_url(). [requested by veego@]
do_fetch() - which retrieves one file calling fetch_url() or fetch_ftp()
as necessary.
* don't http redirect more than 5 times for a given url
* send `User-Agent: NetBSD-ftp/1.4' header in http requests. (suggested
by Christoph Badura <bad@ora.de>)
* cleanup http return code parser, and add support for:
- 300 `Multiple Choices' - but only if the server returns a
preferred url in a Location: header because i'm *not* adding
a html parser to provide the user with options.
- 305 `Use Proxy [given in Location: header]'. (XXX: not tested)
* support http redirects to non-proxied ftp://urls. (bug discovered by
Chris Demetriou <cgd@netbsd.org>)
* auto-login to an ftp site (using the FTP protocol) if an ftp://host/dir/
style url is given and ftp_proxy is set. whilst this is less orthoganol
with other ftp://host/file urls it's *much* more convenient.
1. changed to use timegm(3) on NetBSD and
2. supply alternative code for other Unix-like OSs
(NetBSD ftp shall be portable for some reasons :-).
This fix closes PR #6448.
NOTE: This should be fixed again if a portable UTC to time_t
conversion method is specified in some standards.
grant