50 lines
2.2 KiB
Markdown
50 lines
2.2 KiB
Markdown
|
|
||
|
|
20-11-2020
|
||
|
|
|
||
|
|
# Memory Allocation In vdbesort.c
|
||
|
|
|
||
|
|
Memory allocation is slightly different depending on:
|
||
|
|
|
||
|
|
* whether or not SQLITE_CONFIG_SMALL_MALLOC is set, and
|
||
|
|
* whether or not worker threads are enabled.
|
||
|
|
|
||
|
|
## SQLITE_CONFIG_SMALL_MALLOC=0
|
||
|
|
|
||
|
|
Assuming SQLITE_CONFIG_SMALL_MALLOC is not set, keys passed to the sorter are
|
||
|
|
added to an in-memory buffer. This buffer is grown using sqlite3Realloc() as
|
||
|
|
required it reaches the size configured for the main pager cache using "PRAGMA
|
||
|
|
cache_size". i.e. if the user has executed "PRAGMA main.cache_size = -2048",
|
||
|
|
then this buffer is allowed to grow up to 2MB in size.
|
||
|
|
|
||
|
|
Once the buffer has grown to its threshold, keys are sorted and written to
|
||
|
|
a temp file. If worker threads are not enabled, this is the only significant
|
||
|
|
allocation the sorter module makes. After keys are sorted and flushed out to
|
||
|
|
the temp file, the buffer is reused to accumulate the next batch of keys.
|
||
|
|
|
||
|
|
If worker threads are available, then the buffer is passed to a worker thread
|
||
|
|
to sort and flush once it is full, and a new buffer allocated to allow the
|
||
|
|
main thread to continue to accumulate keys. Buffers are reused once they
|
||
|
|
have been flushed, so in this case at most (nWorker+1) buffers are allocated
|
||
|
|
and used, where nWorker is the number of configured worker threads.
|
||
|
|
|
||
|
|
There are no other significant users of heap memory in the sorter module.
|
||
|
|
Once sorted buffers of keys have been flushed to disk, they are read back
|
||
|
|
either by mapping the file (via sqlite3_file.xFetch()) or else read back
|
||
|
|
in one page at a time.
|
||
|
|
|
||
|
|
All buffers are allocated by the main thread. A sorter object is associated
|
||
|
|
with a single database connection, to which it holds a pointer.
|
||
|
|
|
||
|
|
## SQLITE_CONFIG_SMALL_MALLOC=1
|
||
|
|
|
||
|
|
This case is similar to the above, except that instead of accumulating
|
||
|
|
multiple keys in a single large buffer, sqlite3VdbeSorterWrite() stores
|
||
|
|
keys in a regular heap-memory linked list (one allocation per element).
|
||
|
|
List elements are freed as they are flushed to disk, either by the main
|
||
|
|
thread or by a worker thread.
|
||
|
|
|
||
|
|
Each time a key is added the sorter (and an allocation made),
|
||
|
|
sqlite3HeapNearlyFull() is called. If it returns true, the current
|
||
|
|
list of keys is flushed to a temporary file, even if it has not yet
|
||
|
|
reached the size threshold.
|