Commit Graph

72 Commits

Author SHA1 Message Date
Vincent Sanders
5a2f69388c hoist the Referer header generation logic up to the low level cache
This removes the need for the fetchers to have any interaction with
 the Referer header. It has not been completely removed from the
 fetch interface as fetch.c:fetch_set_cookie() still uses it for
 unverifiable cookie decision logic. (There is an anchient todo here)
2021-03-25 23:36:41 +00:00
Daniel Silverstone
97870122dd
fetch: When aborting, ensure we note this to not multi-fail
When aborting a fetch, it may be cleaned up, in which case we
would otherwise send a spurious failure message too early. Instead
we record that the abort has occurred to suppress such an error.

Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
2020-02-24 10:24:23 +00:00
Daniel Silverstone
083ba385d7
fetch: Automatically handle fetches which fail to finish
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
2020-02-24 09:47:52 +00:00
Vincent Sanders
0c34d06494 Keep the complete certificate chain from a fetch
Instead of extracting information from the X509 certificate chain in
the fetcher the entire chain is propagated in Distinguished Encoding
Rules (DER) format.

This allows all the information contained in a certificate chain to be
retained which can subsequently be presented to the user
2020-02-23 16:23:50 +00:00
Vincent Sanders
214478fc15 Improve some content documentation comments 2020-02-23 16:17:51 +00:00
Daniel Silverstone
1cf1ec55bc Support SSL verification through new about: handler
In doing this, also propagate why the certificates were bad
so that the page can display a reason.  We will need FatMessages
for all these.

Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
2019-08-06 13:15:23 +01:00
Daniel Silverstone
8469f4cc8e Reimplement handling of BAD_AUTH inside browser_window
We now handle authentication requests via an `about:` page
which presents a nice form built into the browser window.
In order to do this, we add internal navigation as a concept
to the browser window and we strip the 401login support from all
frontends except monkey.

The 401login callback is now intended for password safe type support
rather than an immediately interactive prompt.

Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
2019-08-06 11:26:47 +01:00
Daniel Silverstone
bfb1bb1192 Migrate SSL certificate storage to the browser window
* Fetchers now provide the certificates before headers
* This is propagated all the way to the browser window
* When a query occurs, we retrieve it from there and fire
  the query with those stored certificates.
* The serial number is a bignum, store it as hex.

Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
2019-08-05 18:12:47 +01:00
Vincent Sanders
5c96acd6f1 fix url encoding to be compatible with nsurl API changes.
As part of this fix the form submission error handling and
 reporting has been improved.
2018-09-26 17:21:48 +01:00
Vincent Sanders
1f52b2d514 remove unused fetch acessor 2017-01-31 22:56:45 +00:00
Daniel Silverstone
c523bb47a0 Actually use utils/inet.h instead of sys/select.h 2016-06-27 21:58:09 +01:00
Daniel Silverstone
be1ff548cb fetch.h needs sys/select.h for the fd_set type et al. 2016-06-27 21:50:54 +01:00
Daniel Silverstone
47ccd9855d Move fetcher_fdset to fetch.h (and rename to fetch_fdset). Maybe not ideal but better 2016-06-27 21:09:39 +01:00
Daniel Silverstone
4d1ef3bac4 Add support for retrying timed-out cURL fetches.
This is an attempt to amelioriate the situation found in #2384 where
we see the cURL connect() failing to complete.  Based on the pcap
from the bug log, we believe that RISC OS is likely failing to signal
the completion of the connection to cURL.  As such, cURL times out.

This change permits retries of timed out connections in the hope that
a fresh socket FD might subsequently function correctly.  The defaults
chosen mean that the previous behaviour of 30 seconds before timeout
is reported will remain the same, but in that time we will make 3 separate
attempts to connect the socket.
2015-11-10 21:51:54 +00:00
Vincent Sanders
73e45ff024 Fix error reporting from fetch_start
Any fetch start error was being reported as "out of memory" which was
clearly insufficient. Foe example bad urls (reported was file:// with
a missing /) were causing a warn_user with out of memory. This change
now at least causes a "bad url" message.
2015-06-24 10:31:13 +01:00
Vincent Sanders
4b2101ba6a clean up the fetcher factory and improve its API 2014-06-19 18:27:24 +01:00
Vincent Sanders
0475693394 create table for fetcher operations and move all operations into it 2014-01-23 00:00:18 +00:00
Vincent Sanders
4987a3a8a7 remove forward refs from content/fetch.c and cleanup doc comments 2014-01-19 21:18:47 +00:00
Daniel Silverstone
581d877576 In theory, store raw filenames and pass them through for file upload. Untested due to no file-upload in GTK frontend just yet 2014-01-04 19:34:18 +00:00
John-Mark Bell
3dff750ae2 Downgrade TLS version support if it turns out the server can't cope with TLSv1.1. 2013-01-04 23:13:23 +00:00
John Mark Bell
07309888d6 return;
Fix bug #3442642: allow scheme-specific fetchers to have a say in whether an URL can be fetched.

svn path=/trunk/netsurf/; revision=13182
2011-11-27 14:14:36 +00:00
John Mark Bell
dcbafe6b87 Clean up fetch callback API. Inject some long-needed type safety.
svn path=/trunk/netsurf/; revision=13137
2011-11-08 21:51:42 +00:00
Michael Drake
6cfd37e60f Convert fetchers to nsurl.
svn path=/trunk/netsurf/; revision=12910
2011-09-29 15:31:54 +00:00
Michael Drake
e3211bf4fc fetch_can_fetch can take const nsurl.
svn path=/trunk/netsurf/; revision=12902
2011-09-27 14:42:45 +00:00
Michael Drake
e1f7a37f15 Port fetch layer to nsurl. Remove unused fetch_get_referer function.
svn path=/trunk/netsurf/; revision=12899
2011-09-27 11:07:32 +00:00
Michael Drake
9992ee9119 Document fetch_add_fetcher.
svn path=/trunk/netsurf/; revision=12892
2011-09-26 21:23:47 +00:00
Michael Drake
c94271edf5 Fetchers register with an lwc_string, rather than a string.
svn path=/trunk/netsurf/; revision=12891
2011-09-26 21:07:19 +00:00
John Mark Bell
b0f1b7b170 Improve const-correctness of fetch_start
svn path=/trunk/netsurf/; revision=10342
2010-04-10 17:13:53 +00:00
Daniel Silverstone
6c9e4bea8e Purge FETCH_TYPE which is unused throughout NetSurf. Clearing a todo in llcachec.
svn path=/trunk/netsurf/; revision=10300
2010-04-08 12:16:52 +00:00
Daniel Silverstone
2b1146bcb3 Remove parent from fetches, clearing a todo from llcache.c
svn path=/trunk/netsurf/; revision=10299
2010-04-08 11:47:05 +00:00
Daniel Silverstone
270ef59a98 Merge jmb/new-cache; r=dsilvers,rs=vince
svn path=/trunk/netsurf/; revision=10180
2010-03-28 12:56:39 +00:00
Vincent Sanders
6990c2cd04 remove uneccissary includes
svn path=/trunk/netsurf/; revision=9931
2010-01-29 11:32:53 +00:00
John Mark Bell
355799ce0b Merge branches/MarkieB/gtkmain to trunk.
svn path=/trunk/netsurf/; revision=9729
2009-12-17 23:55:02 +00:00
John Mark Bell
5feb7018c5 Replace parent_url with a pointer to the parent content.
svn path=/trunk/netsurf/; revision=8428
2009-07-10 00:26:37 +00:00
John Mark Bell
b98f105b24 Purge WITH_AUTH
svn path=/trunk/netsurf/; revision=6600
2009-02-22 15:13:10 +00:00
John Mark Bell
847b776285 Purge NETSURF_USE_SSL and WITH_SSL
svn path=/trunk/netsurf/; revision=6599
2009-02-22 15:07:50 +00:00
John Mark Bell
79b4cee9d3 Make redirects behave as a NOP wrt the verifiability of fetches.
Apparently, the interweb depends on this.

svn path=/trunk/netsurf/; revision=6542
2009-02-16 20:24:54 +00:00
James Bursa
8282253a54 Move struct cache_data from fetch to content as it is no longer needed by fetch. Make it a member instead of pointer in struct content.
svn path=/trunk/netsurf/; revision=4246
2008-06-03 01:10:46 +00:00
James Bursa
ae992eab4d Add new fetch callback FETCH_HEADER for headers and move as much header parsing as possible from fetch_curl.c to fetchcache.c. This simplifies fetch_curl.c and will make it possible to store response headers in future.
svn path=/trunk/netsurf/; revision=4226
2008-05-30 04:11:16 +00:00
John Mark Bell
cc3481a2ff Make core fetching code handle verifiability of transactions
Make core fetching code be responsible for inserting cookies into the urldb
Provide accessor to a fetch's parent url (this is defined as being the URL of the verifiable fetch which caused this one to occur)
Make fetchcache's 3xx handling use the parent url when spawning new fetches

svn path=/trunk/netsurf/; revision=3809
2008-01-30 19:56:41 +00:00
Vincent Sanders
2f864eeecc Update all source code file headers to reflect GPL version 2 only and contain appropriate licence text
svn path=/trunk/netsurf/; revision=3486
2007-08-08 16:16:03 +00:00
James Bursa
594eb74c4c Fix deadlock when fetching stylesheets when max_fetchers_per_host is reached by splitting fetch_can_be_freed() into fetch_remove_from_queues() and fetch_free().
svn path=/trunk/netsurf/; revision=3380
2007-07-04 18:44:13 +00:00
John Mark Bell
87ca206e13 Warning fixes.
svn path=/trunk/netsurf/; revision=3332
2007-06-10 18:08:22 +00:00
Daniel Silverstone
1dd7e97eb3 Merge scheme switcher branch in.
svn path=/trunk/netsurf/; revision=3330
2007-06-10 17:46:44 +00:00
Daniel Silverstone
6807b4208a Remove the netsurf/ from the include paths and rationalise use of <> vs "" in includes
NetSurf includes are now done with ""s and other system includes with <>s as C intended.
The scandeps tool has been updated to only look for ""ed includes, and to verify that the
files exist in the tree before adding them to the dependency lines. The depend rule has
therefore been augmented to make sure the autogenerated files are built before it is run.

This is untested under self-hosted RISC OS builds. All else tested and works.


svn path=/trunk/netsurf/; revision=3307
2007-05-30 22:39:54 +00:00
John Mark Bell
25213dca24 Fix handling of cookies in unverifiable transactions caused by a redirect from a fetch into a browser window which was varifiable.
svn path=/trunk/netsurf/; revision=3165
2007-02-02 23:08:13 +00:00
James Bursa
a0b6661eb6 Make GTK build compile on FreeBSD.
svn path=/trunk/netsurf/; revision=3154
2007-01-29 22:27:15 +00:00
John Mark Bell
df4dbaf4cf Handle cookies in unverifiable transactions
svn path=/trunk/netsurf/; revision=3151
2007-01-27 20:58:20 +00:00
James Bursa
e76140557d Add fetch_http_code() to retrieve the HTTP status code of a fetch.
svn path=/trunk/netsurf/; revision=3139
2007-01-13 00:19:02 +00:00
Michael Drake
ba23e4b693 Update project URL.
svn path=/trunk/netsurf/; revision=3073
2006-11-27 15:35:18 +00:00