This simplifies the poll loops a little more and makes me less worried
that some other corner case will bite us in the future.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Because cURL can do connection caching behind the scenes, we
need to have a cache for the SSL certificate chains which we
send onward on first header back from cURL.
This uses the new hashmap implementation to mean that we cache
chains on a hostname:port basis.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Instead of extracting information from the X509 certificate chain in
the fetcher the entire chain is propagated in Distinguished Encoding
Rules (DER) format.
This allows all the information contained in a certificate chain to be
retained which can subsequently be presented to the user
* Changed ETag storage to be time_t, rather than int.
* Changed `If-None-Match` value parsing to use proper
time_t parsing, rather than `atoi`.
We emit FETCH_NOTMODIFIED if the resource hasn't changed.
Any errors from the fetch which are not already handled are
reported with an internal query page instead of a modal
dialog.
This is much less invasive for the user and much more in
keeping with how this is handled by other browsers.
The handler is similar to the timeout handler but the
functionality is kept separate as it is intended timeout
handling be extended in future.
cURL will prevent channel reuse if NTLM auth is enabled because
NTLM authenticates a channel not a request. As such we were
unable to reuse curl handles since we handed off connection
reuse to curl instead of our own handle cache. This mitigates
the effect, though curl authors are looking at fixing it upstream
too.
Fixes: #2707
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
The curl fetcher can operate without openssl library being
available, additionaly curl itself may be compiled with a
different TLS library.
In either case this will simply cause the "unknown" error to be
reported for all TLS failiures and page information to lack any
certificate information.
Since OpenSSL 1.0.2 there has been hostname verification support
which cURL doesn't turn on for some reason. Turn it on so that
we get better hostname verification handling.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
In doing this, also propagate why the certificates were bad
so that the page can display a reason. We will need FatMessages
for all these.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
* Fetchers now provide the certificates before headers
* This is propagated all the way to the browser window
* When a query occurs, we retrieve it from there and fire
the query with those stored certificates.
* The serial number is a bignum, store it as hex.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Because we need to be slightly more careful now, lodge ourselves
`inside_curl` whenever we might be processing via the cURL messages
or similar.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
In order to cope better with modern cURL which prevents making
cURL calls when inside a callback from within cURL, defer fetch
start when we are processing in `fetch_curl_data()`.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
stop curl fetcher logging being special case to standard error and
use the fetch catagory at DEBUG level instead.
The special suppress_curl_debug option is currently still obeyed
On ubuntu 19.4, curl is built with HTTP2 support, and we
segfault.
==18174== Invalid read of size 1
==18174== at 0x4ACCE7D: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4B054B1: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AD398A: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AD7A0B: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AE93EE: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AEA8A8: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x1F2EF7: fetch_curl_poll (curl.c:1209)
==18174== by 0x1EEC5C: fetcher_poll (fetch.c:271)
==18174== by 0x2A1ED4: schedule_run (schedule.c:160)
==18174== by 0x15F941: framebuffer_run (gui.c:596)
==18174== by 0x15F941: main (gui.c:2206)
==18174== Address 0x9de95a8 is 3,224 bytes inside a block of size 6,304 free'd
==18174== at 0x483997B: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==18174== by 0x4AD497B: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AE158C: curl_easy_cleanup (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x1F30DE: fetch_curl_cache_handle (curl.c:761)
==18174== by 0x1F30DE: fetch_curl_stop (curl.c:840)
==18174== by 0x1F30DE: fetch_curl_done (curl.c:1122)
==18174== by 0x1F30DE: fetch_curl_poll (curl.c:1223)
==18174== by 0x1EEC5C: fetcher_poll (fetch.c:271)
==18174== by 0x2A1ED4: schedule_run (schedule.c:160)
==18174== by 0x15F941: framebuffer_run (gui.c:596)
==18174== by 0x15F941: main (gui.c:2206)
==18174== Block was alloc'd at
==18174== at 0x483AB35: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==18174== by 0x4AE165F: curl_easy_duphandle (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x1F15EB: fetch_curl_get_handle (curl.c:738)
==18174== by 0x1F15EB: fetch_curl_start (curl.c:750)
==18174== by 0x1EEB22: fetch_dispatch_job (fetch.c:156)
==18174== by 0x1EEB22: fetch_choose_and_dispatch (fetch.c:187)
==18174== by 0x1EEB22: fetch_dispatch_jobs (fetch.c:247)
==18174== by 0x1EF1BB: fetch_start (fetch.c:573)
==18174== by 0x26C779: llcache_object_refetch (llcache.c:916)
==18174== by 0x26D5E4: llcache_object_fetch (llcache.c:979)
==18174== by 0x26D5E4: llcache_object_retrieve_from_cache (llcache.c:1767)
==18174== by 0x26D5E4: llcache_object_retrieve (llcache.c:1865)
==18174== by 0x26E42C: llcache_fetch_redirect (llcache.c:2110)
==18174== by 0x26E42C: llcache_fetch_callback (llcache.c:2810)
==18174== by 0x1F1295: fetch_curl_process_headers (curl.c:922)
==18174== by 0x1F13A0: fetch_curl_data (curl.c:1324)
==18174== by 0x4ACD4C3: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
==18174== by 0x4AE00DA: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0)
The avoids situations were we threw away the length, only for
the caller to have to strlen the returned string.
Note, there seems to be a case of the amiga front end writing
beyond end of allocation. Added a TODO for now.
When processing a x509 certificate chain from openssl it is necessary
to allow teh entire chain to be processed rather than halting
processing at the first certificate with an error. This allows errors
with a certificate authority to be examined.
The wallclock() API uses gettimeofday which can be affected by the the
systems clock being changed etc. The curl fetcher usage of this API is
to generate a timing delta and does not cope with the gettimeofday
issues.
This changes the fetcher to use the nsutils library monotonic time
function which does not suffer from the issues with gettimeofday.
The config header was causing many source files to unecessarily
include the dirent headers causing extra dependancies. This has been
fixed by providing a utility dirent header that provides a common API
across all platforms while removing the unecessary dirent header usage.
The utility configuration header dragged in a number of bsd sockets
and related API as a side effect of setting up the configuration. By
splitting the header and API setup into a separate header only the
small number of places that need the functionality explitly include
it.
The printf formatting for size_t is set in c99 as %zu but in windows
it is %Iu this is solved by adding and inttypes style PRI macro for size_t
This also uses this macro everywhere size_t is formatted.
This is an attempt to amelioriate the situation found in #2384 where
we see the cURL connect() failing to complete. Based on the pcap
from the bug log, we believe that RISC OS is likely failing to signal
the completion of the connection to cURL. As such, cURL times out.
This change permits retries of timed out connections in the hope that
a fresh socket FD might subsequently function correctly. The defaults
chosen mean that the previous behaviour of 30 seconds before timeout
is reported will remain the same, but in that time we will make 3 separate
attempts to connect the socket.
This changes the LOG macro to be varadic removing the need for all
callsites to have double bracketing and allows for future improvement
on how we use the logging macros.
The callsites were changed with coccinelle and the changes checked by
hand. Compile tested for several frontends but not all.
A formatting annotation has also been added which allows the compiler
to check the parameters and types passed to the logging.
This splits up a great deal of the win32 window code out from other
gui code. It also remove large quantities of unused and junk
variables and functions.
The data scheme fetcher was over allocating the space for decoded
base64 encoded urls and not using the base64 API that allocated the
correct size storage.
By using an error code return we can gracefully handle fetcher
registration faliures instead of just immediately aborting.
The curl handler was also cleaned up and documentation improved
as a side effect.
