8.7 KiB
Source Object (low level) cache backing store
[TOC]
Introduction
The source object (referred to as low level in the code) content cache provides a unified API for the rest of the browser to retrieve objects (HTML files, images etc.) from a URL.
The cache initialy always fufuls these requsts by using the fetcher system to retrive data according to the URL scheme (network for HTTP, disc for file etc.) and storing the result in memory.
The cache also provides a system to extend the life of source objects in memory when they are no longer immediately being used. Only fetch types where there are well defined rules on caching are considered, in practice this limits the cache to URLS with HTTP(S) schemes. The section in RFC2616 [1] on caching specifies these rules.
To further extend the objects lifetime they can be pushed into a backing store where the objects are available for reuse less quickly than from memory but faster than retrieving from the network again.
The backing store implementation provides a key:value infrastructure with a simple store, retrieve and invalidate interface.
The key is the object URL which by definition is unique for a source object. The value is the source object data and the associated metadata
Generic filesystem backing store
Although the backing store interface is fully pluggable a generic implementation based on storing objects on the filesystem in a hierarchy of directories.
The option to alter the backing store format exists and is controlled by a version field. It is implementation defined what happens if a version mis-match occurs.
As the backing store only holds cache data one should not expect a great deal of effort to be expended converting formats (i.e. the cache may simply be discarded).
Layout version 2.02
The version 2 layout stores cache entries in a hash map thus only uses memory proportional to the number of entries present removing the need for large fixed size indexes.
The object identifier is generated from nsurl_hash() and data entries are stored in either a fixed size disc blocks or in separate files on disc.
The file path if stored on disc must conform to the limitations of all the filesystems the cache can be placed upon.
From http://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits the relevant subset is:
- path elements no longer than 8 characters
- acceptable characters are A-Z, 0-9
- short total path lengths (255 or less)
- no more than 77 entries per directory (6bits worth)
The short total path lengths mean the encoding must represent as much data as possible in the least number of characters.
To achieve all these goals we use RFC4648 base32 encoding which packs five bits into each character of the filename. By splitting the 32bit identifier using six bits per directory level only five levels of directory are required with a maximum of 64 entries per directory. This requires a total path length of 22 bytes (including directory separators) BA/BB/BC/BD/BE/ABCDEFG
Files that are under 8KiB in size are stored in "small block files" these are pre allocated 8 Megabyte files on disc in which remove the need to have many, many small files stored on disc at the expensie of a some amount of wasted space for files that are smaller than the 8K block size.
Layout version 1.1
An object has an identifier value generated from the URL (the unique key). The value used is obtained using nsurl_hash() which is currently a 32 bit FNV so is directly usable.
This identifier is adequate to ensure the collision rate for the hashed URL values (a collision for every 2^16 URLs added) is sufficiently low the overhead of returning the wrong object (which backing stores are permitted to do) is not significant.
An entry list is maintained which contains all the metadata about a given identifier. This list is limited in length to constrain the resources necessary to maintain it. It is made persistent to avoid the overhead of reconstructing it at initialisation and to keep the data used to improve the eviction decisions.
Each object is stored and retrieved directly into the filesystem using a filename generated from a RFC4648 base32 encoding of an address value. The objects address is derived from the identifier by cropping it to a shorter length.
A mapping between the object address and its entry is maintained which uses storage directly proportional to the size of the address length.
The cropping length is stored in the control file with the default values set at compile time. This allows existing backing stores to continue operating with existing data independently of new default setting. This setting gives some ability to tune the default cache index size to values suitable for a specific host operating system.
E.g. Linux based systems can easily cope with several megabytes of mmaped index but RISC OS might want to limit this to a few megabytes of heap at most.
The files are stored on disc using their base32 address value. By creating a directory for each character of the encoded filename (except the last which is of course the leafname) we create a directory structure where no directory has more than 32 entries.
E.g. A 19bit address of 0x1 would be base32 encoded into AAAB resulting in the data being stored in a file path of "/store/prefix/d/B/A/A/BAAAAA".
An address of 0x00040001 encodes to BAAB and a file path of "/store/prefix/m/B/A/A/BAABAAA"
Layout Version 1.0
The version 1.0 layout was identical to the 1.1 except base64url encoding was used, this proved problematic as some systems filesystems were case insensitive so upper and lower case letters collided.
There is no upgrade provision from the previous version simply delete the cache directory.
Control files
control
A control file is used to hold a list of values describing how the other files in the backing store should be used.
entries
this file contains a table of entries describing the files held on the filesystem.
Each control file table entry is 28 bytes and consists of
- signed 64 bit value for last use time
- 32bit full url hash allowing for index reconstruction and additional collision detection. Also the possibility of increasing the ADDRESS_LENGTH although this would require renaming all the existing files in the cache and is not currently implemented.
- unsigned 32bit length for data
- unsigned 32bit length for metadata
- unsigned 16bit value for number of times used.
- unsigned 16bit value for flags
- unsigned 16bit value for data block index (unused)
- unsigned 16bit value for metatdata block index (unused)
Address to entry index
An entry index is held in RAM that allows looking up the address to map to an entry in the control file.
The index is the only data structure whose size is directly dependant on the length of the hash specifically:
(2 ^ (ADDRESS_BITS - 3)) * ENTRY_BITS) in bytes
where ADDRESS_BITS is how long the address is in bits and ENTRY_BITS is how many entries the control file (and hence the while cache) may hold.
RISCOS values
By limiting the ENTRY_BITS size to 14 (16,384 entries) the entries list is limited to 448kilobytes.
The typical values for RISC OS would set ADDRESS_BITS to 18. This spreads the entries over 262144 hash values which uses 512 kilobytes for the index. Limiting the hash space like this reduces the effectiveness of the cache.
A small ADDRESS_LENGTH causes a collision (two URLs with the same address) to happen roughly for every 2 ^ (ADDRESS_BITS / 2) = 2 ^ 9 = 512 objects stored. This roughly translates to a cache miss due to collision every ten pages navigated to.
Larger systems
In general ENTRY_BITS set to 16 as this limits the store to 65536 objects which given the average size of an object at 8 kilobytes yields half a gigabyte of disc used which is judged to be sufficient.
For larger systems e.g. those using GTK frontend we would most likely select ADDRESS_BITS as 22 resulting in a collision every 2048 objects but the index using some 8 Megabytes
Typical values
Example 1
For a store with 1034 objects generated from a random navigation of pages linked from the about:welcome page.
Metadata total size is 593608 bytes an average of 574 bytes. The majority of the storage is used to hold the URLs and headers.
Data total size is 9180475 bytes a mean of 8879 bytes 1648726 in the largest 10 entries which if excluded gives 7355 bytes average size
Example 2
355 pages navigated in 80 minutes from about:welcome page and a handful of additional sites (google image search and reddit)
2018 objects in cache at quit. 400 objects from news.bbc.co.uk alone
Metadata total 987,439 bytes mean of 489 bytes
data total 33,127,831 bytes mean of 16,416 bytes
with one single 5,000,811 byte gif
data totals without gif is 28,127,020 mean 13,945