Commit Graph

50 Commits

Author SHA1 Message Date
lukem 5a1118c5f9 * add url_decode() - `in-place' decode %xx escapes in a given url component
* parse_url()
-   only look for user[:pass] for an ftp url (per rfc1738)
-   strip leading /'s in an ftp url. (almost per rfc 1738)
* fetch_url()
-   decode a copy of the path and use that to build local filenames
-   send port in http Host: header (suggested by cgd@netbsd.org)
* fetch_ftp()
-   url_decode() the user, pass and path
-   fix splitting of path into dir & file (partially from [bin/7073])
-   don't bother caching the last host; it can cause problems when
    using ftp:// transfers, or when the user changes between xfers
* improve documentation of auto-fetched url arguments (especially regarding
  escape sequences in ftp:// urls)
* some whitespace & copyright updates

this should fix [bin/7073] William O Ferry <woferry@warp.wofme.com>,
as well as the metaissues raised in that PR.
1999-03-08 04:36:12 +00:00
lukem e2352fc342 only print "Requesting..." messages if (verbose) 1999-01-25 23:17:37 +00:00
lukem 0bd80459cb * -v enables verbose & progress, -V disables both
* set setvbuf(ttyout, NULL, _IOLBF, 0) and remove a bunch of fflush(ttyout).
* use fwrite() instead of write() for progressmeter (don't intermix stdio
  with non stdio ops)
1999-01-24 02:39:29 +00:00
lukem 0e87f1b53d support -f, which forces a cache flush for http xfers using either
`Pragma: no-cache' (http/1.0) or `Cache-Control: no-cache' (http/1.1)
1999-01-23 15:46:24 +00:00
lukem 6ae118efdb add missing braces (this is C not python...). fixes coredump on solaris 1999-01-05 00:31:20 +00:00
lukem 234fb36cc6 support http/1.1's `transfer-encoding: chunked' mode. [noted by hubertf & bad] 1999-01-01 13:26:31 +00:00
lukem 1f3ec47c0f * implement auth_url(); given a challenge, decode it, ask the user for
a response, and return the base64 encoded response. only the `Basic'
  scheme is supported.
* implement base64_enc(), which base64 encodes the given buffer
* add support for http response codes 401 "authorization required" and 407
  "proxy authorization required", using auth_url(). [requested by veego@]
1999-01-01 10:00:46 +00:00
lukem 974dee5975 use uname(3) to determine info to send in User-agent: header. (suggested by cgd) 1998-12-31 02:10:34 +00:00
lukem cc66a8544e * major code reorg; gut auto_fetch() and url_get() into a smaller auto_fetch(),
do_fetch() - which retrieves one file calling fetch_url() or fetch_ftp()
  as necessary.
* don't http redirect more than 5 times for a given url
* send `User-Agent: NetBSD-ftp/1.4' header in http requests. (suggested
  by Christoph Badura <bad@ora.de>)
* cleanup http return code parser, and add support for:
    - 300 `Multiple Choices' - but only if the server returns a
      preferred url in a Location: header because i'm *not* adding
      a html parser to provide the user with options.
    - 305 `Use Proxy [given in Location: header]'. (XXX: not tested)
* support http redirects to non-proxied ftp://urls. (bug discovered by
  Chris Demetriou <cgd@netbsd.org>)
* auto-login to an ftp site (using the FTP protocol) if an ftp://host/dir/
  style url is given and ftp_proxy is set. whilst this is less orthoganol
  with other ftp://host/file urls it's *much* more convenient.
1998-12-29 14:59:04 +00:00
lukem ee32f5a96b be saner about the method to determine which $proxy to use 1998-12-27 05:49:53 +00:00
explorer faade9c8e2 Send an Accept: */* header, since some sites require content negotiation.
Grr.
1998-11-22 06:52:32 +00:00
itohy 6d075e2860 Temporary fix of UTC -> time_t conversion:
1. changed to use timegm(3) on NetBSD and
2. supply alternative code for other Unix-like OSs
(NetBSD ftp shall be portable for some reasons :-).

This fix closes PR #6448.

NOTE: This should be fixed again if a portable UTC to time_t
conversion method is specified in some standards.
1998-11-18 07:24:25 +00:00
lukem fa954d84d1 * fix previous in a more portable manner (timegm() is not portable/standard).
use mktime(), and adjust to GMT as per code in util.c::remotemodtime()
1998-11-12 22:27:17 +00:00
itohy 638ec4fd83 HTTP uses UTC time, and changed to use timegm(3) instead of mktime(3)
for conversion of HTTP time.

XXX Unfortunately, timegm(3) is not portable (ex. SunOS 4 have, but 5 not).
1998-11-12 18:19:02 +00:00
lukem da0a823fb0 * hash & progress are mutually exclusive. [noted by mrg@netbsd.org]
* in autofetch mode, don't attempt to 'cd /' on first xfer (only on
  subsequent xfers), as some ftp daemons don't permit that.
  [noted by dbj@netbsd.org].
  XXX: code could be smarter (but a lot more complicated to boot); this
  works for now.
1998-10-08 14:45:26 +00:00
lukem 70b3ec2f32 fix handling of port on non-proxied auto-fetch ftp:// urls 1998-09-01 04:42:49 +00:00
lukem f0622bf9a4 some compilers bitch about int used when enum expected... 1998-08-08 11:23:46 +00:00
lukem 07a9ebee13 * fix skipping of leading / in dir, which may have resulted in
parsing junk memory (picked up whilst porting to solaris)
* remove superfluous var assignments
1998-08-08 04:40:50 +00:00
lukem 6a9c077701 pull in <sys/time.h> for utimes(), reset port to 0 in auto_fetch loop 1998-08-08 04:04:17 +00:00
lukem e4a646bf1c use utimes() instead of futimes() - the former is more portable 1998-08-08 03:33:20 +00:00
lukem edcec00949 technically, you need <sys/stat.h> for struct stat .... 1998-08-08 03:06:00 +00:00
lukem 69761d6f64 cleaner version of fvdl's previous fix, using xstrdup as well 1998-08-04 03:35:24 +00:00
fvdl 14f785118c Avoid using freed memory when using URLs and a proxy. 1998-08-03 19:10:29 +00:00
lukem 4a8ec5494d features:
* support $no_proxy, which is a comma or space separated list of
  host[:port] elements for which proxying is to be disabled.
  (asked for by cgd in [bin/5027])
* if $FTPANONPASS is defined, use that as the anon ftp password
  (instead of "`whoami`@")
* allow http URL's without a filename as long as an output file
  is specified.

other stuff:
* implement parse_url(), which breaks up a URL into its bits, and use.
* simplify url_get() and auto_fetch() to use parse_url() and to not
  modify the supplied URL or a copy of it.
* implement xmalloc() and xstrdup(); error-checked malloc()/strdup()
* add more consistency to messages, quoting strings in output as `%s'
1998-08-03 01:49:25 +00:00
lukem 5ae8291cc6 minor grammar fix 1998-07-26 12:59:40 +00:00
lukem b521b370b5 * spell `retrieved' correctly!
* parse Last-Modified: HTTP headers for date, and change time of
  retrieved file if parsing succeeded
* support file:/// and file://localhost/ URLs
* in url_get(), re-write byte moving code to consistently use fread()
  and fwrite(), and check errors at end with ferror()
* add about:* easter-egg (#ifndef SMALL :-)
1998-07-26 12:58:16 +00:00
lukem 8d879385dd * send 'Connection: close' in HTTP/1.1 headers, preventing the
remove server from using a persistent connection, which speeds
  up such requests.
* support http 301 and 302 redirects
* rewrite guts of url_get() to use fparseln() et al instead of
  read(s,&p,1)...  enables each in the header to be parsed
  as necessary
* rename login to ftp_login, to remove conflict with util.h::login
* cleanup verbose messages during http proxy requests
1998-07-22 16:06:27 +00:00
thorpej 7a12770f94 Add "sndbuf" and "rcvbuf" commands for setting the socket buffer sizes,
which in turn can allow the use of larger TCP windows.  This is a work in
progress; there is not yet support for specifying global defaults or
user prefrences on a host/network basis.
1998-07-10 04:39:03 +00:00
lukem 9a6e9b2ce7 some fixes & enhancements from openbsd's ftp, with extra fixes by me:
* default to passive with active fallback. $FTPMODE modifies this behaviour.
  -A forces active connection.
* support '-o outfile' for auto-fetched files. outfile can be a file,
  `-' (for stdout), or '|command' (to output each file through command).
* support '-r waittime', which retries the connection after waittime seconds
  if it fails.
* fix 'page file' when restart is non-zero.
* try all ip-addresses of a host in a http fetch (as the normal ftp fetch
  does).

XXX: a ``broken pipe'' error sometimes occurs with -o '|command';
i haven't tracked this down yet.
1998-06-04 08:28:35 +00:00
tv 063efa8267 Add the ability to work with HTTP-1.1-style virtual hosts, using the
HTTP/1.0 backwards compatible "Host:" field method.
1998-06-03 15:50:34 +00:00
christos 2187d01f57 - cast arg to is*() to unsigned char. 1998-05-20 00:54:26 +00:00
lukem 6ab4bdf006 define MAX_IN_PORT_T and use instead of USHRT_MAX. suggested by cgd@netbsd.org 1998-01-18 22:09:39 +00:00
lukem 7ee412ebee * ensure buffer for username is initialised, so ^D on username prompt
doesn't use garbage for the username.  from "Soren S. Jorvang" <soren@t.dk>
  in [bin/4559]
* use in_port_t for ports, and USHRT_MAX instead of 0xffff
  (from millert@openbsd.org)
* use `NULL' instead of `(.... *)0' where appropriate.
1998-01-18 14:23:33 +00:00
lukem d067f3bbd2 * in recvrequest(), ignore restart_point unless "RETR"ieving. fixes problems
where a remote completion or `mget' would confuse the client a `restart'
  had been issued beforehand. now, `restart' is remembered until an operation
  that can actually use it is invoked.
* in sendrequest(), don't reset restart_point upon entry. fixes `restart'
  for `put' operations.
* if `restart' is invoked with no arguments, print current setting instead
  of displaying a usage
* consistently use printf("%qd", (long long)restart_point) when displaying
  restart_point
* use strto[lq]() instead of atol() when parsing `mark' and `restart' values
* remove unnecessary strlen()s when result of previous snprintf() will do
* replace a few malloc()/strcpy()s with strdup()s
* use SECSPERHOUR instead of '3600'
1997-11-01 14:36:49 +00:00
lukem 8732be2e09 More cleanups from Todd Miller <Todd.Miller@courtesan.com>:
* Use an int, not "union wait".
* Move the "parsed_url" label so that the next statement is not an "else" --
  some ansi C compilers don't like it the old way (SGI's for example).
* Deal with the possibility of getlogin(2) failing.
* Don't error out if the remote server doesn't support the "MDTM" command.
1997-09-21 01:06:30 +00:00
lukem 87453d32bd Fixes from Todd Miller <Todd.Miller@courtesan.com>:
* use size_t instead of int in places
* use symbolic constants when using access()
1997-09-13 09:05:52 +00:00
lukem 2fec2a28f5 bugs fixed:
* don't interpret '-' or '|' when a local filename is determined from
  the remote name (i.e, in mget, and in get with only one argument).
  This is implemented using an extra argument to recvrequest().
  Fixes a major security hole.
* clean up memory leak when using globulize()
* clean up a couple of comments
* fix wording in TNF copyright

features added:
* support for TIS fwtk gate-ftp servers:
	* read defaults from $FTPSERVER && $FTPSERVERPORT
	* start in gate-ftp mode if invoked as 'gate-ftp'
	* toggle or set with 'gate [host [port]]'
1997-08-18 10:20:13 +00:00
lukem 076ac8bdc3 Send \r with \n in http requests. Allow "content-length" to be missing.
From [bin/3891] by Krister Walfridsson <cato@ulysses.df.lth.se>
1997-07-20 12:49:26 +00:00
lukem b9d5554d5d * use RCSID() && COPYRIGHT() macros
* cleanup for WARNS=1 (including some ugly '(void)&var' bits wrapped in
  #ifdef __GNUC__ to shut up gcc warnings WRT setjmp/longjmp)
* use strtol() instead of atoi(), and more extensively check result of
  conversion
* use u_int16_t instead of short or int for TCP port addresses
1997-07-20 09:45:35 +00:00
lukem 4a8077f211 Provide a different error message than `invalid url' when an
auto-login ftp URL is used when $ftp_proxy is defined. It now prints:
	Auto-login using ftp URLs isn't supported when using $ftp_proxy
Should solve rest of [bin/3643].

Whilst this is inconsistant with the behaviour when $ftp_proxy isn't
defined, the following constraints apply:
* it's not possible to support ftp URL auto-login when $ftp_proxy is
  defined, since it uses http not ftp, and you can't `login' to http
  servers; fudging this would require a major rewrite of ftp anyway)
* silently ignoring $ftp_proxy and not using it if an ftp auto-login
  URL is given is bad user interface design)
* mrg & others will harrass me if I remove support for autologin ftp URLs
  when $ftp_proxy isn't defined, even though it made the behaviour
  consistant whether $ftp_proxy was set or not.
1997-06-29 06:34:50 +00:00
lukem c04e79d154 url_get(): use origline in some messages, simplifying them, and free up copy
of origline before returning. also, previous commit (fix for [bin/3643]) was
inspired by suggested fix in said PR by Alan Barrett <apb@iafrica.com>.
1997-05-23 18:54:18 +00:00
lukem 7cb8cd9611 Actually report why certain URLs are invalid, rather than just exit
with a non-zero exit code. fixes most of [bin/3643].
1997-05-23 18:42:36 +00:00
lukem 4ecb9a5a20 * cleanup parser of ftp://[user:pass@]host[:port]/[dir/][file]. should
be more robust now. fixes [bin/3520]. (missed this in previous commit msg)
* rename bad_url: to bad_ftp_url:
1997-04-21 18:45:47 +00:00
lukem d0e2ef4d67 i 1997-04-21 18:41:03 +00:00
lukem a9a78ba448 More enhancements/bugfixes (when will it end?)
* differentiate between being connected, and being logged in
* cleanup some text messages
* support username & password ftp URLs (ftp://user:pass@host/) in non-proxy
  situations; assume proxy supports it for proxy situations.
* cd to / before performing any autofetch transfers
* use strncasecmp in URL parsing. fix from <Todd.Miller@courtesan.com>
1997-04-14 09:09:15 +00:00
lukem d473547d56 * support $ftp_proxy for ftp:// transfers [bin/3245]
* add "more" & "less" as synonyms for "page"
* move editline setup code into controlediting(), and call appropriately.
  only setup setup terminal if going into interactive mode. inspired
  by Todd Miller <todd.miller@courtesan.com>
1997-04-05 03:27:32 +00:00
lukem ae07641250 Fixes from <Todd.Miller@cs.colorado.edu>, with some cleanup and reworks by me:
* only echo "Passive mode" in verbose mode; scripts that use ftp
  may get unwanted output otherwise
* disable progress bar and modification time preservation when
  retreiving to a non-regular files. fixes progress bar getting in
  way of "get file /dev/tty"
* setup el_init() et al if editing is set, not if fromatty.
  TODO: migrate this to a function, and call if editing is turned on later
	in the session. also implement edit_cleanup if editing is turned off
* call el_set() after setting SIGWINCH handler. This fixes the problem
  when suspending in a non-cbreak shell (e.g, csh) would trash your tty mode.
* reset interactive mode correctly in auto_fetch() mget mode
1997-03-16 14:24:14 +00:00
lukem 3b5054739d Features:
* support remglobbing of auto_fetch arguments
* new flag - '-e'; disable editing
* "page file" == "get file |${PAGER-less}"

Bugfixes/cleanup:
* consistently use a trailing '.' on messages
* code cleanup, including buffer overrun fixes, use puts
  and putchar in places, etc (inspired by OpenBSD mods)
* disable progress bar when local-file is a pipe or '-'
* skip \r in http headers
* fix remote ftpd slash bug more elegantly (so it works with ////)
* abort_remote(): check if cout==NULL before using it. should fix [bin/3273]
* fixed up cosmetic problems when complete_remote() generated errors from the
  remote server (such as "no files found", "login with user and pass", ...)
  done by adding extra argument to remglob(), which is a pointer to an error
  buffer to put messages in rather than printing to stdout.
1997-03-13 06:23:11 +00:00
lukem 3a50014d60 [Yet Another Huge Ftp Commit - hopefully the last for a while,
barring any more little things people want added ...]

New features:
* progressmeter is now asynchronous, so "stalled" transfers can be
  detected. "- stalled -" is displayed instead of the ETA in this case.
  When the xfer resumes, the time that the xfer was stalled for is
  factored out of the ETA. It is debatable whether this is better than
  not factoring it out, but I like it this way (I.e, if it stalls for 8
  seconds and the ETA was 30 seconds, when it resumes the ETA will still
  be 30 seconds).
* verbosity can be disabled on the command line (-V), so that in auto-fetch
  mode the only lines displayed will be a description of the file, and
  the progress bar (if possible)
* if the screen is resized (and detected via the SIGWINCH signal), the
  progress bar will rescale automatically.

Bugs fixed:
* progress bar will not use the last character on the line, as this can
  cause problems on some terminals
* screen dimensions (via ioctl(TIOCWINSZ)) should use stdout not stdin
* progressmeter() used some vars before initialising them
* ^D will quit now. [fixes bin/3162]
* use hstrerror() to generate error message for host name lookup failure.
* use getcwd instead of getwd (it should have been OK, but why tempt fate?)
* auto-fetch transfers will always return a positive exit value upon failure
  or interruption, relative to the file's position in argv[].
* remote completion of / will work, without putting a leading "///".
  This is actually a bug in ftpd(1), where "NLST /" prefixes all names
  with "//", but fixing every ftpd(1) is not an option...
1997-02-01 10:44:54 +00:00
lukem 7a7fa80740 New features:
* Command line editing via editline(3) library.
* Context sensitive command and file completion, including remote files.

Enhancements to auto-fetch feature:
* Support for http:// URLs using the http protocol, including proxy HTTP
  support via $htty_proxy if it's defined.
* The connection is kept open between successive files on the same host.
  (obviously, this does not count for http requests.)
* Return value of ftp is 0 on no error, or the offset in argv[] of the file
  which failed (i.e., argv[x] failed, ftp returns x).
* If the path in an ftp URL or classic format line has a trailing '/',
  cd to the path and enter interactive mode. Fixes [bin/3011], albiet
  requiring the user to help ftp in determining the operation.

Other changes:
* '-P port' works for normal ftp, and is the default for all classic style
  auto-fetch transfers and for ftp URLs that don't specify the port.
  (previously it would just work for the first xfer.)
* Some code moved into separated files along logical divisions.
* Editing and completion can be compiled out with -DSMALLFTP.
1997-01-19 14:19:02 +00:00