mirrors/sqlite
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update various documentation comments in sqlite3ota.c and sqlite3ota.h.

Browse Source 
FossilOrigin-Name: 60e0a46b82dd9c704e8aa977d1ccdd73d388422f
This commit is contained in:
dan 2015-02-19 18:06:40 +00:00
parent 71405b27c9
commit 138bf3bc7f
5 changed files with 144 additions and 185 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

134
ext/ota/README.txt
View File

@ -1,134 +0,0 @@
This file contains notes regarding the implementation of the OTA extension.
User documentation is in sqlite3ota.h.
SQLite Hacks
------------
1) PRAGMA pager_ota_mode=1:
This pragma sets a flag on the pager associated with the main database only.
In a zipvfs system, this pragma is intercepted by zipvfs and the flag is set
on the lower level pager only.
The flag can only be set when there is no open transaction and the pager does
not already have an open WAL file. Attempting to do so is an error.
Once the flag has been set, it is not possible to open a regular WAL file.
If, when the next read-transaction is opened, a *-wal file is found or the
database header flags indicate that it is a wal-mode database,
SQLITE_CANTOPEN is returned.
Otherwise, if no WAL file or flags are found, the pager opens the *-oal file
and uses it as a write-ahead-log with the *-shm data stored in heap-memory.
The 8-bytes of "salt" at the start of an *-oal file is a copy of the 8 bytes
starting at offset 24 of the database file header (the change counter and the
number of pages in the file). If the *-oal file already exists when it is
opened, SQLite checks that the salt still matches the database header fields.
If not, it concludes that the database file has been written by a
rollback-mode client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT
error is returned. No read-transaction can be opened in this case.
A pager with the pager_ota_mode flag set never runs a checkpoint.
Other clients see a rollback-mode database on which the pager_ota_mode client
is holding a SHARED lock. There are no locks to arbitrate between multiple
pager_ota_mode connections. If two or more such connections attempt to write
simultaneously, the results are undefined.
2) PRAGMA pager_ota_mode=2:
The pager_ota_mode pragma may also be set to 2 if the main database is open
in WAL mode. This prevents SQLite from checkpointing the wal file as part
of sqlite3_close().
The effects of setting pager_ota_mode=2 if the db is not in WAL mode are
undefined.
3) sqlite3_ckpt_open/step/close()
API for performing (and resuming) incremental checkpoints.
The OTA extension
-----------------
The OTA extension requires that the OTA update be packaged as an SQLite
database. The tables it expects to find are described in sqlite3ota.h.
Essentially, for each table xyz in the target database that the user wishes
to write to, a corresponding data_xyz table is created in the OTA database
and populated with one row for each row to update, insert or delete from
the target table.
The OTA extension opens the target and OTA update databases using a single
database handle (the target database is "main", and the OTA update database is
attached as "ota"). It executes both the "pager_ota_mode" and "ota_mode"
pragmas described above. For each data_xyz table in then:
* CREATEs an ota_xyz table in the OTA update database.
* Loops through the data_xyz table, running the INSERT, UPDATE or DELETE
command on the corresponding target database table. Only the main b-tree
is updated by these statements. Modified pages are appended to the *-oal
file.
Temporary triggers installed on the target database catch the old.*
values associated with any UPDATEd or DELETEd rows and store them in
the ota_xyz table (in the OTA update database).
* For each index on the data_xyz table in the target database:
Loop through a union of the data_xyz and ota_xyz tables in the order
specified by the data_xyz index. In other words, if the index is on
columns (a, b), read rows from the OTA update database using:
SELECT * FROM data_xyz UNION ALL ota_xyz ORDER BY a, b;
For each row visited, use an sqlite3_index_writer() VM to update the index
in the target database.
* DROPs the ota_xyz table.
At any point in the above, the process may be suspended by the user. In this
case the "ota_state" table is created in the OTA database, containing a single
row indicating the current table/index being processed and the number of updates
already performed on it, and the transaction on the target database is committed
to the *-oal file. The next OTA client will use the contents of the ota_state
table to continue the update from where this one left off.
Alternatively, if the OTA update is completely applied, the transaction is
committed to the *-oal file and the database connection closed. sqlite3ota.c
then uses a rename() call to move the *-oal file to the corresponding *-wal
path. At that point it is finished - it does not take responsibility for
checkpointing the *-wal file.
Problems
--------
The rename() call might not be portable. And in theory it is unsafe if some
other client starts writing the db file.
When state is saved, the commit to the *-oal file and the commit to the OTA
update database are not atomic. So if the power fails at the wrong moment they
might get out of sync. As the main database will be committed before the OTA
update database this will likely either just pass unnoticed, or result in
SQLITE_CONSTRAINT errors (due to UNIQUE constraint violations).
If some client does modify the target database mid OTA update, or some other
error occurs, the OTA extension will keep throwing errors. It's not really
clear how to get out of this state. The system could just by delete the OTA
update database and *-oal file and have the device download the update again
and start over.
At present, for an UPDATE, both the new.* and old.* records are collected in
the ota_xyz table. And for both UPDATEs and DELETEs all fields are collected.
This means we're probably writing a lot more data to disk when saving the
state of an ongoing update to the OTA update database than is strictly
necessary.

138
ext/ota/sqlite3ota.c
View File

@ -9,6 +9,75 @@
** May you share freely, never taking more than you give.
**
*************************************************************************
**
**
** OVERVIEW
**
** The OTA extension requires that the OTA update be packaged as an
** SQLite database. The tables it expects to find are described in
** sqlite3ota.h. Essentially, for each table xyz in the target database
** that the user wishes to write to, a corresponding data_xyz table is
** created in the OTA database and populated with one row for each row to
** update, insert or delete from the target table.
**
** The update proceeds in three stages:
**
** 1) The database is updated. The modified database pages are written
** to a *-oal file. A *-oal file is just like a *-wal file, except
** that it is named "<database>-oal" instead of "<database>-wal".
** Because regular SQLite clients do not look for file named
** "<database>-oal", they go on using the original database in
** rollback mode while the *-oal file is being generated.
**
** During this stage OTA does not update the database by writing
** directly to the target tables. Instead it creates "imposter"
** tables using the SQLITE_TESTCTRL_IMPOSTER interface that it uses
** to update each b-tree individually. All updates required by each
** b-tree are completed before moving on to the next, and all
** updates are done in sorted key order.
**
** 2) The "<database>-oal" file is moved to the equivalent "<database>-wal"
** location using a call to rename(2). Before doing this the OTA
** module takes an EXCLUSIVE lock on the database file, ensuring
** that there are no other active readers.
**
** Once the EXCLUSIVE lock is released, any other database readers
** detect the new *-wal file and read the database in wal mode. At
** this point they see the new version of the database - including
** the updates made as part of the OTA update.
**
** 3) The new *-wal file is checkpointed. This proceeds in the same way
** as a regular database checkpoint, except that a single frame is
** checkpointed each time sqlite3ota_step() is called. If the OTA
** handle is closed before the entire *-wal file is checkpointed,
** the checkpoint progress is saved in the OTA database and the
** checkpoint can be resumed by another OTA client at some point in
** the future.
**
** POTENTIAL PROBLEMS
**
** The rename() call might not be portable. And OTA is not currently
** syncing the directory after renaming the file.
**
** When state is saved, any commit to the *-oal file and the commit to
** the OTA update database are not atomic. So if the power fails at the
** wrong moment they might get out of sync. As the main database will be
** committed before the OTA update database this will likely either just
** pass unnoticed, or result in SQLITE_CONSTRAINT errors (due to UNIQUE
** constraint violations).
**
** If some client does modify the target database mid OTA update, or some
** other error occurs, the OTA extension will keep throwing errors. It's
** not really clear how to get out of this state. The system could just
** by delete the OTA update database and *-oal file and have the device
** download the update again and start over.
**
** At present, for an UPDATE, both the new.* and old.* records are
** collected in the ota_xyz table. And for both UPDATEs and DELETEs all
** fields are collected. This means we're probably writing a lot more
** data to disk when saving the state of an ongoing update to the OTA
** update database than is strictly necessary.
**
*/
#include <assert.h>
@ -2403,34 +2472,59 @@ sqlite3_int64 sqlite3ota_progress(sqlite3ota *pOta){
** Beginning of OTA VFS shim methods. The VFS shim modifies the behaviour
** of a standard VFS in the following ways:
**
** 1. Whenever the first page of a main database file is read or
** written, the value of the change-counter cookie is stored in
** ota_file.iCookie. Similarly, the value of the "write-version"
** database header field is stored in ota_file.iWriteVer. This ensures
** that the values are always trustworthy within an open transaction.
** 1. Whenever the first page of a main database file is read or
** written, the value of the change-counter cookie is stored in
** ota_file.iCookie. Similarly, the value of the "write-version"
** database header field is stored in ota_file.iWriteVer. This ensures
** that the values are always trustworthy within an open transaction.
**
** 2. When the ota handle is in OTA_STAGE_OAL or OTA_STAGE_CKPT state, all
** EXCLUSIVE lock attempts on the target database fail. This prevents
** sqlite3_close() from running an automatic checkpoint. Until the
** ota handle reaches OTA_STAGE_DONE - at that point the automatic
** checkpoint may be required to delete the *-wal file.
** 2. Whenever an SQLITE_OPEN_WAL file is opened, the (ota_file.pWalFd)
** member variable of the associated database file descriptor is set
** to point to the new file. A mutex protected linked list of all main
** db fds opened using a particular OTA VFS is maintained at
** ota_vfs.pMain to facilitate this.
**
** 3. In OTA_STAGE_OAL, the *-shm file is stored in memory. All xShmLock()
** calls are noops. This is just an optimization.
** 3. Using a new file-control "SQLITE_FCNTL_OTA", a main db ota_file
** object can be marked as the target database of an OTA update. This
** turns on the following extra special behaviour:
**
** 4. In OTA_STAGE_OAL mode, when SQLite calls xAccess() to check if a
** *-wal file associated with the target database exists, the following
** special handling applies:
** 3a. If xAccess() is called to check if there exists a *-wal file
** associated with an OTA target database currently in OTA_STAGE_OAL
** stage (preparing the *-oal file), the following special handling
** applies:
**
** a) if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA
** target database may not be in wal mode already.
** * if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA
** target database may not be in wal mode already.
**
** b) if the *-wal file does not exist, set the output parameter to
** non-zero (to tell SQLite that it does exist) anyway.
** * if the *-wal file does not exist, set the output parameter to
** non-zero (to tell SQLite that it does exist) anyway.
**
** 5. In OTA_STAGE_OAL mode, if SQLite tries to open a *-wal file
** associated with a target database, open the corresponding *-oal file
** instead.
** Then, when xOpen() is called to open the *-wal file associated with
** the OTA target in OTA_STAGE_OAL stage, instead of opening the *-wal
** file, the ota vfs opens the corresponding *-oal file instead.
**
** 3b. The *-shm pages returned by xShmMap() for a target db file in
** OTA_STAGE_OAL mode are actually stored in heap memory. This is to
** avoid creating a *-shm file on disk. Additionally, xShmLock() calls
** are no-ops on target database files in OTA_STAGE_OAL mode. This is
** because assert() statements in some VFS implementations fail if
** xShmLock() is called before xShmMap().
**
** 3c. If an EXCLUSIVE lock is attempted on a target database file in any
** mode except OTA_STAGE_DONE (all work completed and checkpointed), it
** fails with an SQLITE_BUSY error. This is to stop OTA connections
** from automatically checkpointing a *-wal (or *-oal) file from within
** sqlite3_close().
**
** 3d. In OTA_STAGE_CAPTURE mode, all xRead() calls on the wal file, and
** all xWrite() calls on the target database file perform no IO.
** Instead the frame and page numbers that would be read and written
** are recorded. Additionally, successful attempts to obtain exclusive
** xShmLock() WRITER, CHECKPOINTER and READ0 locks on the target
** database file are recorded. xShmLock() calls to unlock the same
** locks are no-ops (so that once obtained, these locks are never
** relinquished). Finally, calls to xSync() on the target database
** file fail with SQLITE_INTERNAL errors.
*/
/*

40
ext/ota/sqlite3ota.h
View File

@ -24,9 +24,9 @@
** containing the entry being inserted or deleted must be modified. If the
** working set of leaves is larger than the available cache memory, then a
** single leaf that is modified more than once as part of the transaction
** may be loaded from or written to the persistent media more than once.
** may be loaded from or written to the persistent media multiple times.
** Additionally, because the index updates are likely to be applied in
** random order, access to pages within the databse is also likely to be in
** random order, access to pages within the database is also likely to be in
** random order, which is itself quite inefficient.
**
** One way to improve the situation is to sort the operations on each index
@ -40,10 +40,10 @@
** Additionally, this extension allows the work involved in writing the
** large transaction to be broken down into sub-transactions performed
** sequentially by separate processes. This is useful if the system cannot
** guarantee that a single update process may run for long enough to apply
** the entire update, for example because the update is running on a mobile
** device that is frequently rebooted. Even after the writer process has
** committed one or more sub-transactions, other database clients continue
** guarantee that a single update process will run for long enough to apply
** the entire update, for example because the update is being applied on a
** mobile device that is frequently rebooted. Even after the writer process
** has committed one or more sub-transactions, other database clients continue
** to read from the original database snapshot. In other words, partially
** applied transactions are not visible to other clients.
**
@ -62,8 +62,8 @@
** * INSERT statements may not use any default values.
**
** * UPDATE and DELETE statements must identify their target rows by
** PRIMARY KEY values. If the table being written has no PRIMARY KEY
** declaration, affected rows must be identified by rowid.
** PRIMARY KEY values. If the table being written has no PRIMARY KEY,
** affected rows must be identified by rowid.
**
** * UPDATE statements may not modify PRIMARY KEY columns.
**
@ -148,11 +148,11 @@
** the new values of all columns being update. The text value in the
** "ota_control" column must contain the same number of characters as
** there are columns in the target database table, and must consist entirely
** of "x" and "." characters. For each column that is being updated,
** the corresponding character is set to "x". For those that remain as
** they are, the corresponding character of the ota_control value should
** be set to ".". For example, given the tables above, the update
** statement:
** of 'x' and '.' characters (or in some special cases 'd' - see below). For
** each column that is being updated, the corresponding character is set to
** 'x'. For those that remain as they are, the corresponding character of the
** ota_control value should be set to '.'. For example, given the tables
** above, the update statement:
**
** UPDATE t1 SET c = 'usa' WHERE a = 4;
**
@ -207,10 +207,10 @@
**
** 4) Calls sqlite3ota_close() to close the OTA update handle. If
** sqlite3ota_step() has been called enough times to completely
** apply the update to the target database, then it is committed
** and made visible to other database clients at this point.
** Otherwise, the state of the OTA update application is saved
** in the OTA database for later resumption.
** apply the update to the target database, then the OTA database
** is marked as fully applied. Otherwise, the state of the OTA
** update application is saved in the OTA database for later
** resumption.
**
** See comments below for more detail on APIs.
**
@ -285,9 +285,9 @@ int sqlite3ota_step(sqlite3ota *pOta);
/*
** Close an OTA handle.
**
** If the OTA update has been completely applied, commit it to the target
** database. Otherwise, assuming no error has occurred, save the current
** state of the OTA update appliation to the OTA database.
** If the OTA update has been completely applied, mark the OTA database
** as fully applied. Otherwise, assuming no error has occurred, save the
** current state of the OTA update appliation to the OTA database.
**
** If an error has already occurred as part of an sqlite3ota_step()
** or sqlite3ota_open() call, or if one occurs within this function, an

15
manifest
View File

@ -1,5 +1,5 @@
C Merge\slatest\strunk\schanges\swith\sthis\sbranch.
D 2015-02-19T14:41:24.386
C Update\svarious\sdocumentation\scomments\sin\ssqlite3ota.c\sand\ssqlite3ota.h.
D 2015-02-19T18:06:40.917
F Makefile.arm-wince-mingw32ce-gcc d6df77f1f48d690bd73162294bbba7f59507c72f
F Makefile.in 6b9e7677829aa94b9f30949656e27312aefb9a46
F Makefile.linux-gcc 91d710bdc4998cb015f39edf3cb314ec4f4d7e23
@ -123,7 +123,6 @@ F ext/misc/totype.c 4a167594e791abeed95e0a8db028822b5e8fe512
F ext/misc/vfslog.c fe40fab5c077a40477f7e5eba994309ecac6cc95
F ext/misc/vtshim.c babb0dc2bf116029e3e7c9a618b8a1377045303e
F ext/misc/wholenumber.c 784b12543d60702ebdd47da936e278aa03076212
F ext/ota/README.txt 2ce4ffbb0aaa6731b041c27a7359f9a5f1c69152
F ext/ota/ota.c c11a85af71dccc45976622fe7a51169a481caa91
F ext/ota/ota1.test ba408c5e777c320ef72f328e20cd2ae2a8888cda
F ext/ota/ota10.test 85e0f6e7964db5007590c1b299e75211ed4240d4
@ -138,8 +137,8 @@ F ext/ota/ota9.test d3eee95dd836824d07a22e5efcdb7bf6e869358b
F ext/ota/otaA.test ef4bfa8cfd4ed814ae86f7457b64aa2f18c90171
F ext/ota/otafault.test 8c43586c2b96ca16bbce00b5d7e7d67316126db8
F ext/ota/otafault2.test fa202a98ca221faec318f3e5c5f39485b1256561
F ext/ota/sqlite3ota.c 6c329a3c1f1ca625f51161321724b25c24646f2d
F ext/ota/sqlite3ota.h 1cc7201086fe65a36957740381485a24738c4077
F ext/ota/sqlite3ota.c a8e19ef2a297627fdc1e18f8679110afeaa5cd4b
F ext/ota/sqlite3ota.h 69106b04616f4e7e565aa4dc2092a2f095212cc2
F ext/ota/test_ota.c 9ec6ea945282f65f67f0e0468dad79a489818f44
F ext/rtree/README 6315c0d73ebf0ec40dedb5aa0e942bc8b54e3761
F ext/rtree/rtree.c 14e6239434d4e3f65d3e90320713f26aa24e167f
@ -1258,7 +1257,7 @@ F tool/vdbe_profile.tcl 67746953071a9f8f2f668b73fe899074e2c6d8c1
F tool/warnings-clang.sh f6aa929dc20ef1f856af04a730772f59283631d4
F tool/warnings.sh 0abfd78ceb09b7f7c27c688c8e3fe93268a13b32
F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f
P a3c1bc5d5e3f4b197f48cbbc240608e94bfc2b45 81f242e338d6122e27aad86986bfd140012c6582
R d299d5655569d5895ceed70c48c0588b
P 6f5888a5e430feb5d9a50009a2eb103d9945bd22
R 536b36bcdeeeb4de103e032d97a52335
U dan
Z b1c159c49abb337fc6eddedfd6fb5f7b
Z 4148e3e31feb6bcdc60726ac41a858b3

2
manifest.uuid
View File

@ -1 +1 @@
6f5888a5e430feb5d9a50009a2eb103d9945bd22
60e0a46b82dd9c704e8aa977d1ccdd73d388422f