netsurf/docs/source-object-backing-store.md

Source Object (low level) cache backing store
=============================================

Introduction
------------

The source object cache provides a system to extend the life of source
objects (HTML files, images etc.) after they are no longer immediately
being used.

Only fetch types where we have well defined rules on caching are
considered, in practice this limits us to HTTP(S). The section in
RFC2616 [1] on caching specifies these rules.

To further extend the objects lifetime they can be pushed into a
backing store where the objects are available for reuse less quickly
than from RAM but faster than retrieving from the network again.

The backing store implementation provides a key:value infrastructure
with a simple store, retrieve and invalidate interface.

Generic filesystem backing store
--------------------------------

Although the backing store interface is fully pluggable a generic
implementation based on storing objects on the filesystem in a
hierarchy of directories.

The option to alter the backing store format exists and is controlled
by a version field. It is implementation defined what happens if a
version mis-match occurs.

As the backing store only holds cache data one should not expect a
great deal of effort to be expended converting formats (i.e. the cache
may simply be discarded).

Layout version 1.1
------------------

An object has an identifier value generated from the URL (NetSurf
backing stores uses the URL as the unique key). The value used is
obtained using nsurl_hash() which is currently a 32 bit FNV so is
directly usable.

This identifier is adequate to ensure the collision rate for the
hashed URL values (a collision for every 2^16 URLs added) is
sufficiently low the overhead of returning the wrong object (which
backing stores are permitted to do) is not significant.

An entry list is maintained which contains all the metadata about a
given identifier. This list is limited in length to constrain the
resources necessary to maintain it. It is made persistent to avoid the
overhead of reconstructing it at initialisation and to keep the data
used to improve the eviction decisions.

Each object is stored and retrieved directly into the filesystem using
a filename generated from a RFC4648 base32 encoding of an address
value. The objects address is derived from the identifier by cropping
it to a shorter length.

A mapping between the object address and its entry is maintained which
uses storage directly proportional to the size of the address length.

The cropping length is stored in the control file with the default
values set at compile time. This allows existing backing stores to
continue operating with existing data independently of new default
setting. This setting gives some ability to tune the default cache
index size to values suitable for a specific host operating system.

E.g. Linux based systems can easily cope with several megabytes of
mmaped index but RISC OS might want to limit this to a few megabytes
of heap at most.

The files are stored on disc using their base32 address value.
By creating a directory for each character of the encoded filename
(except the last which is of course the leafname) we create a
directory structure where no directory has more than 32 entries.

E.g. A 19bit address of 0x1 would be base32 encoded into AAAB
resulting in the data being stored in a file path of
"/store/prefix/d/B/A/A/BAAAAA".

An address of 0x00040001 encodes to BAAB and a file path of
"/store/prefix/m/B/A/A/BAABAAA"

Version 1.0
-----------

The version 1 layout was identical to the 1.1 except base64url
encoding was used, this proved problematic as some systems filesystems
were case insensitive so upper and lower case letters collided.

There is no upgrade provision from the previous version simply delete
the cache directory.

Control files
~~~~~~~~~~~~~

control
+++++++
A control file is used to hold a list of values describing how the
other files in the backing store should be used.

entries
+++++++

this file contains a table of entries describing the files held on the
filesystem.

Each control file table entry is 28 bytes and consists of

 - signed 64 bit value for last use time

 - 32bit full url hash allowing for index reconstruction and
   additional collision detection. Also the possibility of increasing
   the ADDRESS_LENGTH although this would require renaming all the
   existing files in the cache and is not currently implemented.

 - unsigned 32bit length for data

 - unsigned 32bit length for metadata

 - unsigned 16bit value for number of times used.

 - unsigned 16bit value for flags

 - unsigned 16bit value for data block index (unused)

 - unsigned 16bit value for metatdata block index (unused)

Address to entry index
~~~~~~~~~~~~~~~~~~~~~~

An entry index is held in RAM that allows looking up the address to
map to an entry in the control file.

The index is the only data structure whose size is directly dependant
on the length of the hash specifically:

(2 ^ (ADDRESS_BITS - 3)) * ENTRY_BITS) in bytes

where ADDRESS_BITS is how long the address is in bits and ENTRY_BITS
is how many entries the control file (and hence the while
cache) may hold.

RISCOS values
+++++++++++++

By limiting the ENTRY_BITS size to 14 (16,384 entries) the entries
list is limited to 448kilobytes.

The typical values for RISC OS would set ADDRESS_BITS to 18. This
spreads the entries over 262144 hash values which uses 512 kilobytes
for the index. Limiting the hash space like this reduces the
effectiveness of the cache.

A small ADDRESS_LENGTH causes a collision (two URLs with the same
address) to happen roughly for every 2 ^ (ADDRESS_BITS / 2) = 2 ^ 9 =
512 objects stored. This roughly translates to a cache miss due to
collision every ten pages navigated to.

Larger systems
++++++++++++++

In general ENTRY_BITS set to 16 as this limits the store to 65536
objects which given the average size of an object at 8 kilobytes
yields half a gigabyte of disc used which is judged to be sufficient.

For larger systems e.g. those using GTK frontend we would most likely
select ADDRESS_BITS as 22 resulting in a collision every 2048 objects
but the index using some 8 Megabytes

Typical values
--------------

Example 1
~~~~~~~~~

For a store with 1034 objects generated from a random navigation of
pages linked from the about:welcome page.

Metadata total size is 593608 bytes an average of 574 bytes. The
majority of the storage is used to hold the URLs and headers.

Data total size is 9180475 bytes a mean of 8879 bytes 1648726 in the
largest 10 entries which if excluded gives 7355 bytes average size

Example 2
~~~~~~~~~

355 pages navigated in 80 minutes from about:welcome page and a
handful of additional sites (google image search and reddit)

2018 objects in cache at quit. 400 objects from news.bbc.co.uk alone

Metadata total 987,439 bytes mean of 489 bytes

data total 33,127,831 bytes mean of 16,416 bytes

with one single 5,000,811 byte gif

data totals without gif is 28,127,020 mean 13,945

[1] http://tools.ietf.org/html/rfc2616#section-13