mirrors/sqlite
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
27,644 Commits 1,487 Branches 166 Tags 416 MiB
0810150532
Commit Graph

811 Commits

Author SHA1 Message Date
dan
9f6c0d0027 Fix a crash that may occur when handling corrupt records in contentless_delete=1 mode. 
FossilOrigin-Name: 74d7610a8e32ac62a608141ff16bfe517c8d2cdfd43f81ef3a5df2b2003e541a
 2023-07-12 19:34:32 +00:00
drh
9faa3bbdbb Get access to SQL functions and virtual tables used for debugging and analysis 
of FTS5 using SQLITE_FTS5_DEBUG and without the need for SQLITE_TEST.

FossilOrigin-Name: 383de8e2259adb3f8881a618b8b5e06c1f3b2bd7ac274bd047410ad7d2a18f0f
 2023-07-12 18:38:47 +00:00
dan
b184c91076 Have contentless_delete=1 tables support REPLACE statements. 
FossilOrigin-Name: 2f553a660e00564e51bc4209c92bd3628fb1266f4a52832792fbf91e4234a0ba
 2023-07-11 20:57:19 +00:00
dan
b92669db73 Have contentless_delete=1 tables support regular DELETE statements, instead of just the special INSERT syntax. 
FossilOrigin-Name: fffb8616905501669a94231d5d9f53446bf09553353f2cdab7c43ca54bbb7fa6
 2023-07-11 20:19:46 +00:00
dan
d23f210377 Use a hash-table instead of a flat list to store tombstone rowids. 
FossilOrigin-Name: 948267b066d0dbe667881b3d26a007fa24576da6e57c112676fadeb846c13f0b
 2023-07-11 18:55:19 +00:00
dan
6788c7b7c0 Begin adding support for deleting rows from contentless fts5 tables. 
FossilOrigin-Name: e513bea84dfaf2280f7429c9a528b3a1354a46c36e58ab178ca45478975634e0
 2023-07-10 20:44:09 +00:00
dan
d679f9e5d1 Fix a buffer over-read that could occur as part of an fts5 secure-delete operation on a database containing corrupt fts5 records. 
FossilOrigin-Name: a35c50eca907d0cf373b7962c359aebfce8ac7c34ce81edeb99122d4fb5804f7
 2023-06-12 11:06:57 +00:00
dan
870fe69c72 Add extra tests for fts5, large rowids and "ORDER BY rowid DESC". No changes to code. 
FossilOrigin-Name: b5b5eaeed3ef55415e27aa04561ab446f3be881b56b7cbf9b8ba3307f0b2bf69
 2023-05-27 19:34:46 +00:00
dan
b838a229be Fix a buffer overrun that could occur in fts5 when processing corrupt records. 
FossilOrigin-Name: 4891dbd938f4bfd345eaef01f2addea9512eaa98f860844c73abb907b6a8e0e8
 2023-05-22 11:02:15 +00:00
dan
4042d2b424 Limit the number of nested NOT nodes in an fts5 expression to 256. 
FossilOrigin-Name: 0e5c1ee40a146ef8b2b3c5f53d0a45e092bc8d8e933f3819805c995819d31bae
 2023-05-15 17:14:16 +00:00
dan
cad6aae45e Allow virtual tables to prepare writes to shadow tables in defensive mode from with xRelease() callbacks - even if the xRelease() is invoked from within sqlite3_finalize(). 
FossilOrigin-Name: a17106494a8a67bc92cfe27ee8ef9bf160517d6930d796980ae12287b98c1cfe
 2023-05-05 11:47:31 +00:00
drh
62b28db5c7 Add another assert() to FTS5 to demonstrate that a pointer is never NULL. 
FossilOrigin-Name: b8a84a1bf53fa07ef01c57c6db6329ba439774a262b3adcbe94e7bd77525e296
 2023-05-03 14:10:05 +00:00
dan
3f23eb6813 Add an assert() to fts5_config to ensure that a potential OOM is being handled correctly. 
FossilOrigin-Name: fe9c207657400f9d9f4e822eb658157bc147ed538e2701322f6f973933f023ed
 2023-05-03 13:57:57 +00:00
drh
3442306989 Protect a macro argument with parentheses in FTS5. 
FossilOrigin-Name: bc07fe51fe0c6bb50ca8ae1baefcc35c8f5395b2d0de641bf0b0cedc92d754d4
 2023-05-03 13:48:33 +00:00
drh
f9b58eb4f8 Fix harmless compiler warnings. 
FossilOrigin-Name: 342af5b4fa0bd7c699e5497161db13d0cf795c7a5875ae30d666122e518f213b
 2023-05-01 20:42:15 +00:00
drh
1f02cd6cd6 Fix harmless compiler warnings in FTS5. 
FossilOrigin-Name: f6210017b3143ccbaea23655852f947f779bb438ad4fbf582f5da42f7cd1d9f5
 2023-05-01 19:59:10 +00:00
drh
59fe13e39a Fix harmless compiler warnings. 
FossilOrigin-Name: 0a926994691ccb02ea4f0ba3c2e8f728871ae67d24c705fbc0f7af81f38d5f47
 2023-05-01 18:52:35 +00:00
dan
e01e3b5b94 Fix a crash that could occur in fts5 'secure-delete' mode when operating on corrupt records. 
FossilOrigin-Name: 2e85b0e3dcae0915aa6472a3654c8ac72a6b2083c11747f3f657c79bbdaf530b
 2023-04-29 18:31:44 +00:00
dan
4d24eff8dc Avoid a crash in fts5 when combining an empty expression with another. 
FossilOrigin-Name: 33681ff3d29c9f096dbd8c5d885993f780e93974a7653adc00fa41a158356d7c
 2023-04-22 20:47:08 +00:00
dan
8cb4f0b1b2 Add tests for the progress-handler callback with the fts5 'secure-delete' feature. 
FossilOrigin-Name: 48505ad950bc0902d58210be066d4672e6085eb27c525ba2bc663fde7e932ff7
 2023-04-19 20:29:26 +00:00
dan
a5eab9579a Fix an assert() in fts5 that could fail following an OOM or IO error. 
FossilOrigin-Name: 49595083ba58e13d404b6a11c28f53768a257c59f3f36be3591438127df7e875
 2023-04-18 11:33:35 +00:00
dan
68c0da6f66 Fix some problems with using fts5 options 'secure-delete' and detail=none together. 
FossilOrigin-Name: 4d3f27ba90aa59837e49ceaae9f36cad426af3a33707d85d1bb848fc1bda6fed
 2023-04-17 18:32:22 +00:00
dan
48824f23cb Fix problems handling corrupt records in new fts5 code. 
FossilOrigin-Name: 2f63d4a8bac6dc72197df32efd84c148adbc4794943088dff08da57b15dd7900
 2023-04-17 11:41:42 +00:00
dan
5c45a9fbaf Fix an assert() in fts5 that may be true when dealing with corrupt records. 
FossilOrigin-Name: e8de2774463c07b76cd6b9a44ab32a583fe973f42a21b21af46fe82ab177c1b8
 2023-04-17 10:39:51 +00:00
dan
ae6f6ef3b0 Change a memcpy() to memmove() in new fts5 code. 
FossilOrigin-Name: 8de4542553ae20947787b4daf5e88bbe69f815679960e700e19f9c1148234b08
 2023-04-15 21:00:55 +00:00
dan
0e429cf934 Fix a use-after-free that could follow an OOM or IO error in fts5 when using 'secure-delete' mode. 
FossilOrigin-Name: 904be83d9f3dd4b21a3aa47dde337b5c045b925ffdde4bccd1f8e8fdbef68c18
 2023-04-15 19:58:58 +00:00
dan
e45317a50a Avoid dropping the error code following an OOM in fts5 when using 'secure-delete' mode. 
FossilOrigin-Name: 9ee24a499c7b4a048f269f1c03c7c0b22b2898a3c23ea3e2511c2ba0342afa30
 2023-04-15 19:13:51 +00:00
dan
112500028e Further tests for fts5 secure-delete mode. 
FossilOrigin-Name: c5a47063b7971868ebbb3f0718434dd6bd22d01c43697f48941df9aae3885abc
 2023-04-14 18:00:03 +00:00
dan
3fc123b7bc Add extra OOM test for the new code on this branch. 
FossilOrigin-Name: 846ae7e099ce67dab6b5df0ad5648a01d7bf7f2acf8312d817ed8505d4ea5aec
 2023-04-14 17:00:29 +00:00
dan
d36106c954 Add tests for the new feature on this branch. 
FossilOrigin-Name: 0268d1a59316d169927cbd4c562725e46a9023b7d7123ade911c12203e222adf
 2023-04-14 16:11:05 +00:00
dan
02877e9558 Remove temporary debugging code accidentally left on this branch. 
FossilOrigin-Name: 0a0f64870feaf95d2673efc8884ad1236d9bbf9bdb94364f8d2602221bd36ff3
 2023-04-12 18:06:43 +00:00
dan
015020cd1a Add the 'secure-delete' option to fts5. Used to configure fts5 to aggressively remove old full-text-index entries belonging to deleted or updated rows. 
FossilOrigin-Name: 4240fd09b717dbc69dffe3b88ec9149777ca4c3efa12f282af65be3af6fa5bb0
 2023-04-12 17:40:44 +00:00
drh
ebec9c475e A prepared statement that aborts due to SQLITE_SCHEMA should not invalidate 
cursors in other prepared statements that are already running.
See [forum:/forumpost/cae4367d9b|forum post cae4367d9b] for the original
trouble report.

FossilOrigin-Name: 857d0f5e16ba69ac9e5ee581befca2f7ed933edfe6e36396b10dae7979b44a57
 2023-04-06 13:35:42 +00:00
dan
5f708cc832 Fix a problem with the fts5 snippet() function that shows up when snippets just 1 token in length are requested. 
FossilOrigin-Name: 96d5116d17cbf3de6be38952881c6ddf6a226ddfd7649197bdb2df39c42b5d8e
 2023-03-10 13:36:19 +00:00
dan
a3e6192941 Fix a problem with the fts5 trigram tokenizer and LIKE or GLOB patterns for which contain runs of 2 or fewer non-wildcard characters that are 3 or more bytes when encoded as utf-8. 
FossilOrigin-Name: 00714b39b39c51519edbc0194f98c7275fecf96763a06fd95db6e1d81bb9f1f1
 2023-02-10 17:17:04 +00:00
dan
0d12a79310 Better fix the problem where optimizing an fts5 table too often causes it to become unreadable (first attempt was [35bed981]). 
FossilOrigin-Name: 459d986d38fc0ccbfd66801e0f22900cfed831268cf59ac8d1cd1e556f0d1441
 2023-02-08 17:28:08 +00:00
dan
d40a183506 Fix a problem where optimizing an fts5 table too often might cause it to become unreadable. 
FossilOrigin-Name: 35bed981416269018a0d52171d9997c43d3b097f5db1910d10d882eabe2e137f
 2023-02-03 19:28:50 +00:00
drh
3601db8f86 The json_group_array(), json_group_object(), and fts5_source_id() functions 
should all be deterministic and innocuous.

FossilOrigin-Name: edcb83fa0d3c832d044b267d608fed31a2ac841b81c7c9406b2136ac43d61a1a
 2023-01-20 13:34:14 +00:00
dan
1dff401c42 Fix a broken assert() in fts5. 
FossilOrigin-Name: ff5a9b054f4d5ad3d33787aee93b4181822c6dfc513626236d9867bb431f64da
 2022-12-27 13:52:25 +00:00
drh
3547e4997f Fix lots of harmless, nuisance compiler warnings, mostly unused parameter 
warnings in extensions.

FossilOrigin-Name: c14bbe1606c1450b709970f922b94a641dfc8f9bd09126501d7dc4db99ea4772
 2022-12-23 14:49:24 +00:00
dan
73e460c932 Fix a faulty assert() in fts5. 
FossilOrigin-Name: 569018170b928cadfa0745db631981084e595d90a8c3d1e236c42adae161018f
 2022-12-19 20:14:22 +00:00
dan
1532f9d155 Fix an assert() in fts5. Simplify memdb xLock/xUnlock some. 
FossilOrigin-Name: 7fe158aa8071acadd959b2b4a4b66e8a7f7eecec207ba681abf516b0457c2921
 2022-12-19 14:06:36 +00:00
dan
838865c0bb Better handle an error in the fts5 integrity-check code. dbsqlfuzz e87c62f9b67ea21aebdc36ab71cab7cc3eda8dc3. 
FossilOrigin-Name: ae43e97087a3207a5ca3ffae75fbe7a33c01f4a38ce0d1d7eed8591ae3083617
 2022-11-09 11:17:57 +00:00
dan
6aafd74853 Avoid some cases of signed integer overflow in fts5 by casting to unsigned values. 
FossilOrigin-Name: 46a78c8c0ed518c4521e6e0bdebeb065bab07076abc444775002e7f4361d2242
 2022-08-08 19:29:53 +00:00
dan
cf2ad7ae36 Remove a few unsuitable scripts from the "veryquick" test suite. Also have every second testrunner.tcl process favour running test scripts that contain text like "testrunner: slow" before any others. 
FossilOrigin-Name: 22d280a5cd395abbedcfffbac3d3b3a614c327be25763ca380c1338a2a7bd33a
 2022-07-18 19:32:30 +00:00
dan
f703b42dc4 Update the expected error messages in some OOM test cases to account for [44d77a7f807]. 
FossilOrigin-Name: b3d6b3c3fc68dca7e20418eefa35ce3b583322b884b88a11c6773419f027a7a4
 2022-07-11 21:40:34 +00:00
drh
de7a820fd0 Minor typo fix in a comment, for the benefit of broken IDE syntax highlighters. 
FossilOrigin-Name: 680ccecbc839373194ed2dcc1053e25512d9ea3f12f8c8963cb8fbd1a034d2f8
 2022-04-29 12:09:43 +00:00
drh
c381056e6d Fix harmless compiler warnings. 
FossilOrigin-Name: 61095b5bcb5c30d29f03efa5b4809e3a0b780ed57e35227c292aab9dd1cfe3f6
 2022-04-07 10:11:35 +00:00
dan
7ef4d75bd2 Fix a case in fts5 where a corrupt database could cause a crash. 
FossilOrigin-Name: 5e95df261cce275adb5b2226e9584f4c389bb0d53c3b4feafd5a14be8d02ce63
 2022-03-11 12:02:18 +00:00
drh
7d44b22d45 Fix test cases so that they all still work even with -DSQLITE_DQS=0. 
FossilOrigin-Name: 4883776669ee2f2310ea82b0d6df4d008eebaa7cb252102539cf21a635402ebb
 2022-01-16 19:11:13 +00:00
drh
a6ca90c38c Attempt to fix a harmless compiler warning in FTS5. 
FossilOrigin-Name: 8e619c21e2326be1538b60908e7cd211558ec840835c6eb69e768eb190e1fd0b
 2022-01-01 19:55:29 +00:00
dan
c436b3056d Fix an OOB read that could occur in fts5 when processing corrupt records. 
FossilOrigin-Name: bb9b1a15f7e80483162049dfd981d059dc69d03348b521f7ac164a8cd3ae3cc4
 2021-12-06 18:57:02 +00:00
drh
913306a53a Improved defenses against integer overflow when computing the size of a 
memory allocations.  No bugs were fixed here.  But perhaps future bugs will
be prevented.

FossilOrigin-Name: eb878c01f317f09e8ef6b1bd2ec8d6d5cd6ce0bdfe9da7fa7d92d2047cc9d9e4
 2021-11-26 17:10:18 +00:00
dan
76c3ae6848 Fix a broken assert() in fts5 that could fail if the snippet() function were used with a query phrase containing 0 tokens on a detail=none or detail=col table. 
FossilOrigin-Name: 0a5b94b99cf45df759a59cb5fc62da111ca33f8c2614769a81930d72b51f093a
 2021-11-02 07:32:13 +00:00
drh
134f544ab2 Fix a harmless scan-build warning in FTS5. 
FossilOrigin-Name: 0bf42bb5611dc3f672cb898b8be245fd25f7a3862c1e0734effd18d75e812f22
 2021-10-16 19:50:03 +00:00
dan
f7ff755651 Fix an assert() in fts5 that could fail with a corrupt database. 
FossilOrigin-Name: e99979855de937ed5ee0994b180054501400bf8776fb70acd31786d2ba1ad49a
 2021-10-16 13:59:08 +00:00
dan
33cf194218 Fix a signed integer overflow in fts5 leading to a segfault that could occur when processing corrupt records. 
FossilOrigin-Name: 69a3ff0cc159cdf84a5367eaf708882ddeda4fa65c96a5b546ae4a0114f02cb7
 2021-10-14 21:13:02 +00:00
drh
13ca72d011 Fix harmless compiler warnings, mostly caused by prior efforts to get static 
analyzers to run without warnings.

FossilOrigin-Name: cddd3b382a118bb16be3a602ae711fe2c2a7d1c673dd30dd3977b67ee555e666
 2021-10-05 18:59:05 +00:00
drh
d7ace22d8c Fix more harmless static-analyzer warnings. As of this check-in, no warnings 
are generated by scan-build for clang-6 or clang-10.

FossilOrigin-Name: a7835bead85b1b18a8affd9835240b0baf9c7af887196bbdcc3f5d58055042fc
 2021-10-05 18:33:38 +00:00
drh
7a3b4451a1 Fixes for harmless static-analyzer warnings. This also makes the code easier 
for humans to understand.

FossilOrigin-Name: 36177a62feeb4fa93ab6e3c6f4dbe1ddcf63bb02f93284abab979da0261b218e
 2021-10-05 17:41:12 +00:00
drh
640574f61a Small change to FTS5 to help static analyzers understand that it is not 
dereferencing a NULL pointer.

FossilOrigin-Name: 0d853df1b19a0270b2cb0ebdf5df8067572f19ab9a82d047cb047f3f9e850348
 2021-10-05 13:00:50 +00:00
drh
922c54206f Some #defines somehow failed to get set correctly in the previous check-in. 
Fixed here.

FossilOrigin-Name: 15bbdf9ac840a220f384411d3025ef22f949d310194b60bca8e6d6a759e6042e
 2021-10-04 18:57:42 +00:00
drh
11a9ad5669 Fix harmless static analyzer warnings in sessions, rtree, fts3 and fts5. 
Add the -DSQLITE_OMIT_AUXILIARY_SAFETY_CHECKS compile-time option to cause
ALWAYS() and NEVER() macros to be omitted from the build.

FossilOrigin-Name: 1c67f957fc77e37ce8f0d447c41ca975e8e79a35d332739c24a633649b5b0387
 2021-10-04 18:21:14 +00:00
dan
adfb203a7d Fix a buffer overread in fts5 that could occur when processing an "ORDER BY rowid DESC" query against corrupt database records. 
FossilOrigin-Name: fe098a93340b8dac15752f1be4f681e2a591f7ae1a94b177f400ccd7ef638480
 2021-10-01 15:05:19 +00:00
dan
415540ddaa Fix an out-of-bounds read in fts5 that could occur when processing corrupt records. 
FossilOrigin-Name: 9bbc01fb239b4aa752a56c584baa29655a3c845425d5c17d24a3939984d54fe2
 2021-09-29 13:36:58 +00:00
drh
dd31c033c6 Fix harmless compiler warnings. 
FossilOrigin-Name: d678ecca02698753d1b33e072566112e94ea36d0d3a8f4a24d2b09d131968d88
 2021-09-22 14:43:35 +00:00
dan
eb84c6ef35 Fix a memory leak in fts5 that could occur when writing to an fts5 table that is being scanned by an fts5vocab cursor. 
FossilOrigin-Name: 1183552de10b64a776874a23be8962c0693e5e0cd1a191ea8fa5d6a02b691e4a
 2021-09-07 16:05:48 +00:00
dan
fb8ca7de0c Fix a use-after-free error that could occur if an fts5 table is written while scanning it using an fts5vocab cursor. 
FossilOrigin-Name: e751c2ec786b5c1a1c9640fdc3fde036879a2c32db2bd67fe7c72604780f67b8
 2021-09-06 16:15:23 +00:00
drh
20de9f6cb1 Minor tweaks to various TCL build scripts so that they work even for TCL 8.4, 
which is obsolete, but is also all that is available for some legacy platforms.

FossilOrigin-Name: 2269ce64f707c4a198fcb9a72933648ea61a80b4251024b8058ff28ad81b4917
 2021-07-29 16:49:28 +00:00
dan
73983658c9 In defensive mode, allow statements that write to shadow tables to be prepared from with virtual-table xSync() calls. 
FossilOrigin-Name: c8601d83fbecf84ca7991d339449b380deb5a7620dc84756a91c2880f030b423
 2021-07-19 14:00:29 +00:00
dan
cc516af4cc Instead of disallowing writes to fts5 tables if there are fts5vocab cursors open on them (commit [c49a6ed7]), abort any fts5vocab queries if the on-disk structure of the fts5 table changes. 
FossilOrigin-Name: 9dbdc9001e3258e71ca995fbcdebf66ab95890ded87fa7125c6cb4bd43010aaf
 2021-07-07 11:51:03 +00:00
dan
b9324fea07 Do not allow writes to an fts5 table if there are any open fts5vocab cursors. 
FossilOrigin-Name: c49a6ed78a917d4972e048e2a9bbe4d400691f97ce7e022f0e4436ceaed7fb73
 2021-07-05 19:01:09 +00:00
dan
90d3b24a16 Fix a memory leak in fts5 that could occur when processing a query containing multiple classes of error. 
FossilOrigin-Name: 7c279670f544a5bb9733a88862d9e490062b07415cd790e0d6d1e2cb584d2007
 2021-06-30 14:04:25 +00:00
dan
c629e21786 Fix an assert() in fts5 that might fail when dealing with corrupt records. 
FossilOrigin-Name: 9e760e256089c0ebbf816a8de8b186f62e1ffcd6b8b0e76dab8387798cd4d0e1
 2021-06-30 12:11:19 +00:00
dan
8d2a9c3b3a Add defensive code to better handle corrupt records in fts5. 
FossilOrigin-Name: a99c29b5efaf17e34506cda5ce017ec0112b3cb24bf8a2f1831f0c14a8aa8d30
 2021-06-09 13:39:11 +00:00
drh
18f9be897d Remove two assert()s in FTS5 that checks for database corruption right before 
actual code does the same check.
dbsqlfuzz 9680db8f5338cb9b6060eb3a02d984555d4472e3.

FossilOrigin-Name: f3f82967b2e9fd4168d86300138ac4bbcd6ffd7a3ff5ecf20acdc29e5356ac3b
 2021-06-08 23:46:59 +00:00
dan
bc85a51595 Fix a buffer overread that could occur in fts5 when handling corrupt records. 
FossilOrigin-Name: 078962a2164a784b135bacee51ef10973dc2e30de04353d48698d0e72edd63d8
 2021-06-07 17:36:57 +00:00
drh
ff9186137d The fix in the previous check-in was only correct if the OOM occurs on 
the initial allocation.  This changes should make it correct for a resize
as well.

FossilOrigin-Name: 57087ab2f297e4d96da5b9f3b89a26d8dd2bc4d22c2be8045b5d956d86282f39
 2021-05-22 11:00:03 +00:00
drh
d42dbd9dfe If an FTS5 Cursor fails to enlarge the space for the aInst array, set the 
size of the aInst array to zero.
dbsqlfuzz 294254b8105cca409f27a711f1eb2e9e63cbcac5.

FossilOrigin-Name: 4ae5e5b5ecbce580d3f7f970076e828fc6fb5fbcdc3bbe8381ab38f52870a54f
 2021-05-22 01:30:30 +00:00
drh
e31cba5675 Remove an assert() in FTS5 that can (rarely) be false in the event of an 
OOM while processing a corrupt database file.

FossilOrigin-Name: 5f12f89154843e83b1d69ea1652e881d583aa329df732af72dd59029cd7eb577
 2021-05-06 13:45:37 +00:00
dan
a4cd0bbcb3 Fix a segfault that could occur when querying a corrupt data structures with an fts5token table. 
FossilOrigin-Name: cbedfa05501f3eb9750fb21e2ab8516f8f14c479c847f454e52572af278aab71
 2021-04-27 11:28:57 +00:00
dan
f3d5a68461 Fix an assert() in fts5_index.c that may fail if the database records are corrupt. 
FossilOrigin-Name: dfadb96edfef1d49e48a659264b9eb8d7f98d95d159d95c92c30dd4ba6c6e739
 2021-04-23 11:37:01 +00:00
drh
214b50314a Fix harmless compiler warnings of unused function parameters in FTS5. 
FossilOrigin-Name: 14273915979f8fbf80111e795b5511a6ee24638896b9d434d6ccf4bf7f7a7f62
 2021-04-22 19:34:31 +00:00
dan
6a4f7f0225 Remove fts5 test functions fts5_expr(), fts5_expr_tcl(), fts5_isalnum() and fts5_fold() from release builds. 
FossilOrigin-Name: c68a6c6c898d2acd1f7032ff98f8f3f99eb0b37ffbc6ee88d0dd7badf55a0e0a
 2021-04-19 16:45:09 +00:00
dan
304cbc17c2 Fix an undefined signed integer overflow in fts5. 
FossilOrigin-Name: e6f0adb00da84561e686a8db83858c7fd6b008756dd1aef807ea68f878ca3db7
 2021-04-13 17:45:36 +00:00
drh
f1f12661c3 Avoid taking the address of a NULL pointer following an OOM in FTS5. Doing 
so is harmless in actual practice, but it technically UB so we want to
avoid it.

FossilOrigin-Name: 1cfcd9dceb56b5987e6900a36a0ec092f0e1b13a7e754b8c3d8efb943e5bcc66
 2021-04-12 18:32:33 +00:00
dan
d73ade7d87 Fix issue with failing --enable-debug --enable-fts5 build. 
FossilOrigin-Name: e7491acc0c93b7b0b5a27130d956bb23f876b90ced11cdd416a86727af076ee1
 2021-04-06 17:51:18 +00:00
dan
f22c590bab Remove some test routines from release builds of fts5. 
FossilOrigin-Name: 3088bf15ca53fac3bad94f241e648836631b6d801d42837c238ff27d85074ca4
 2021-04-06 14:46:50 +00:00
dan
acbae3ba10 Add "#ifdef SQLITE_DEBUG" blocks around test variables sqlite3_fts5_may_be_corrupt and sqlite3_fts3_may_be_corrupt. 
FossilOrigin-Name: 004559544e661f9afa60306c4ac6842c29a8630b3663d62aa0cdc637f176932e
 2021-03-29 20:04:31 +00:00
drh
8210233c7b Revise tests cases to align with the new EXPLAIN QUERY PLAN output. 
FossilOrigin-Name: 50fbd532602d2c316813046ed6be8be2991c281eb5f295c4c28520a0de73862c
 2021-03-20 15:11:29 +00:00
dan
55b2e8b062 Fix further sanitizer complaints in the fts5 code. 
FossilOrigin-Name: 4cb00b9d4020a51f690720c6f51009c5be8c22ef8c70807ead1d953d19784f26
 2021-03-10 19:32:47 +00:00
dan
3be2b860d0 Fix a few sanitizer complaints in fts5. 
FossilOrigin-Name: 4fe4fee00e9294fa384bd073ed8e9a2a4ad10dcf653b6ad4851e44a64443ede9
 2021-03-10 14:57:10 +00:00
dan
8718f0b613 Fix a buffer overrun that could occur in fts5 when running a prefix query against a corrupt db. 
FossilOrigin-Name: 4da40620ac8557aba6e1e06e720418b55e9950f406288b03749ef1b117041cd9
 2021-03-03 19:36:40 +00:00
dan
5763f3d582 Fix a broken assert() in fts5 that could be triggered by corrupt database records. 
FossilOrigin-Name: b79f59f9ad897d5bd4b9d17e6219bc765b02450bfe14dc020485f221ba6b02cb
 2020-12-19 15:39:10 +00:00
dan
84b168ffa7 Change an fts5 assert() that can be triggered by a corrupt database to an if() condition. 
FossilOrigin-Name: ea0a7f103a6f6a9e57d7377140ff9f372bf2b156f86f148291fb05a7030f2b36
 2020-12-15 13:55:38 +00:00
dan
1418b9daf6 Add tests for a 'delete' command on a contentless table where values that weren't actually inserted are NULL. 
FossilOrigin-Name: 818c647cec7063b33b6c5de3e23599a1d61439fa6e9bf6c974b2522a5a9e1b44
 2020-12-09 16:49:28 +00:00
drh
aeb6bc5628 Fix harmless compiler warnings associated with the recent FTS5 enhancements. 
FossilOrigin-Name: 1db7c751912beb57a697ac8e85b9c29e30da7b6c89207e9828bf08e56c58242f
 2020-12-04 17:05:16 +00:00
dan
b9ceb833f7 Ensure that fts5 function matchinfo() is registered before running tests that use it in fts5corrupt3.test. 
FossilOrigin-Name: 932e05e093192991589e70c2cbcc8a57fb3dcc6df1c2673962bfd06ba5cfdd97
 2020-12-04 16:49:25 +00:00
dan
3190b88e75 When merging prefix lists in fts5, use 16-way merges instead of 2-way merges. This faster. 
FossilOrigin-Name: 026a93508ec392ca5cd2578ae9eab64974f58beccda088e10d4cc951f237632f
 2020-12-03 21:22:37 +00:00
dan
64f1ef6abd Allow a search for an N character prefix in fts5 to use a prefix index of size N+1, if no prefix index of size N exists. 
FossilOrigin-Name: 78a7801d8fc9e58a62e5168e35b52b7440340549123fc6a537e2abd571f6fe7b
 2020-12-02 19:08:15 +00:00
dan
0fa433b40c Improve performance for fts5 column filters that filter for more than one column. e.g. "{col1 col2 col3}:phrase". 
FossilOrigin-Name: d8de2f236d43a88fac7550a0451951dd5a945eb304e32f82e662479cea7c2684
 2020-12-02 16:23:05 +00:00
dan
cad760d16e Fix a compiler warning in fts5_aux.c. 
FossilOrigin-Name: 8edb983bc87898eff2cd2e7e672a32a47c71b2be9d818513d339e95560d45b2b
 2020-11-27 19:40:13 +00:00
dan
6d19bf9460 Add missing cast to fts5 bm25() code. 
FossilOrigin-Name: 6ff9673847c0b4174d9435e93d19af0ee7406b1a12edeb6edec98697e1646824
 2020-11-27 16:15:55 +00:00
dan
3181706895 Remove a redundant "if( rc==SQLITE_OK )" from the fts5 bm25() code. 
FossilOrigin-Name: d85f4f27f58adcc75fc7d59e63af95b2a338052d8748a11f22ec1e48d1aff4cc
 2020-11-27 16:08:15 +00:00
dan
66efc393c5 Move an "if( rc==SQLITE_OK )" to outside a loop body in the fts5 bm25() code. 
FossilOrigin-Name: 14a4dcf3474566d072007a37d214c892397c21dd3b7f8b55ad0e5edfb7130dd6
 2020-11-27 16:05:31 +00:00
dan
1c5b23f170 Fix a case in the FTS5 integrity check where a corrupt database could cause a buffer overread. 
FossilOrigin-Name: a32b4f650d2d543bd2773cbc8655c1679a20b35ac9ec4d08c7754ddf6d972acb
 2020-11-27 15:28:26 +00:00
dan
ec8962869a Update mkunicode.tcl to match the change erroneously made to machine generated file fts5_unicode2.c in [b7b7bde9]. 
FossilOrigin-Name: 326d579d777fdede6bc64f9525248767f4730de4e50260b0387e614a9d006416
 2020-11-26 20:13:54 +00:00
drh
d36f588f31 Fix harmless compiler warnings about unused function parameters. 
FossilOrigin-Name: 25d067c270966d9506db8bedf280883e32b69050b14bdbbeda4bb2d9a362619c
 2020-11-25 16:28:04 +00:00
dan
d1d43efa4f Prevent fts5 tokenizer unicode61 from considering '\0' to be a token characters, even if other characters of class "Cc" are. 
FossilOrigin-Name: b7b7bde9b7a03665e3691c6d51118965f216d2dfb1617f138b9f9e60e418ed2f
 2020-10-26 13:24:36 +00:00
dan
f6b0464e36 Fix some fts5 problems with signed integer overflow causing segfaults in -ftrapv builds. 
FossilOrigin-Name: 7e17c2f4b7dc9b563d0b4da949bb134dc7c4fc9c86ce03891432a884ca6409d5
 2020-10-12 18:09:16 +00:00
dan
f46be6a1b9 Allow fts5 trigram tables created with detail=column or detail=none to optimize LIKE and GLOB queries. Allow case-insensitive tables to optimize GLOB as well as LIKE. 
FossilOrigin-Name: 64782463be62b72b5cd0bfaa7c9b69aa487d807c5fe0e65a272080b7739fd21b
 2020-10-05 16:41:56 +00:00
dan
12a6a1eaf9 Fix a segfault caused by running "column LIKE NULL" against an fts5 table using the trigram tokenizer. Fix for [e33ee62575fc22]. 
FossilOrigin-Name: 6e72a08de764077f2bba6f7e3b99ea29001941671a971f2ccf7ceeb9c682fb1a
 2020-10-03 17:06:02 +00:00
dan
95dca8d0cf FTS5 does not handle tokens that contain embedded nul characters. Prevent the trigram tokenizer from returning such tokens. Fix for [2ba5930b2]. 
FossilOrigin-Name: b1d048748c054575425a4bebf0c5d09962f9329d5ce6a978cf54e508b238584c
 2020-10-03 14:36:06 +00:00
dan
a22c1c84cc Add the -trigram option to the fts5txt2db.tcl script. 
FossilOrigin-Name: a394dd260252e7b7dcbc90fd321dfd40d9375f0598ea52b01001c568cbdbafce
 2020-10-01 17:15:12 +00:00
dan
ccf578d435 Add tests for the trigram tokenizer. Fix minor issues. 
FossilOrigin-Name: 897ced99b44085012aa44d3264940dcbd4c77b295a894a1b58fb2c03a0f7fee8
 2020-10-01 16:10:22 +00:00
dan
33a99fad08 Add experimental unicode-aware trigram tokenizer to fts5. And support for LIKE and GLOB optimizations for fts5 tables that use said tokenizer. 
FossilOrigin-Name: 0d7810c1aea93c0a3da1ccc4911dbce8a1b6e1dbfe1ab7e800289a0c783b5985
 2020-09-30 20:35:37 +00:00
dan
7548ab20e6 In fts5 integrity checks, do not compare the contents of the index against an external content table unless specifically requested. 
FossilOrigin-Name: 782163693f37aeb65209bebbaeb6659a36881b8c4b4bec778b366658488bf966
 2020-09-21 14:53:21 +00:00
dan
f2e151aeab Fix a problem in fts5 causing the in-memory hash table to be flushed more often than necessary. 
FossilOrigin-Name: 9a55601d072bcd1ac0f6f1633216c7c8b3ff3e037451e2cfd285cbe19b60b9fc
 2020-09-17 15:16:50 +00:00
dan
3acd3fb9d8 Fix a "jump depends on unitialized value" valgrind/asan error in fts5. 
FossilOrigin-Name: 66f94d623c16b4527e42f0519e32ec78eaf9ee6fd61ee7d0eb657cc59a8f7d68
 2020-09-17 14:16:02 +00:00
dan
86f477edaa Catch fts5 index corruption caused by issuing 'delete' commands with incorrect data earlier in some cases. Also fix a couple of test script problems. 
FossilOrigin-Name: b79f19edfd33c2a75f936c352668e14e81f35acf4f07edc27a21f941a7304b38
 2020-09-11 15:01:49 +00:00
dan
832aa023c6 Add fts5 test to confirm that for a table with columns a, b, c and d, "{a b}" and "-{c d}" are handled similarly. 
FossilOrigin-Name: 1a04920998368e56276fd0b100be8343609c6ff8a731cf8e26a0490f9c6dabdf
 2020-08-28 11:19:49 +00:00
dan
bbc9e1908a Fix an integer overflow bug in fts5 triggered by a corrupt record. 
FossilOrigin-Name: 49da8bdce17ced910b414d50e8df79800513dcf03a196197dc1ea8e47deeda92
 2020-07-13 11:06:30 +00:00
drh
ac9e184e1f Avoid an infinite recursion on an illegal recursive definition of an 
fts5vocab table.

FossilOrigin-Name: 109ee07433b274a39954cef62bf67d47bcda960df9bef56127210ebf1c3c104c
 2020-02-16 17:40:35 +00:00
drh
9ca9a53a42 Modify a couple of FTS test cases so that they work on big-endian platforms. 
FossilOrigin-Name: b20503aaf5b6595afafee6aa2bc6276354a62b0f583f6d94e31fedfdfc130347
 2020-01-29 12:37:59 +00:00
mistachkin
8122c6e9be Fix variable declaration issue seen with MSVC. 
FossilOrigin-Name: 8a6fe3066cf0599fcf2960e8fb3ae39e4e7a61ec81e8ec71d0b5910aee2c16f6
 2020-01-23 00:08:25 +00:00
dan
9f3e6fadde Handle blobs that are the return values of functions being cast to text in utf16 databases in the same way as blobs read directly from the database. Fix for [771fe617]. 
FossilOrigin-Name: e782096aa06fcf410c3a0a6ee26a9e4f44333da39b28b2efe675814a8652911a
 2020-01-02 16:24:22 +00:00
dan
9d41af23a4 Do not use HIDDEN columns for NATURAL joins. Fix for [7c0e06b16]. 
FossilOrigin-Name: ab09ef427181130be09a087b7e572ad4cfb6b3e1b459769ee5ebf046b3ead682
 2019-12-30 14:32:27 +00:00
dan
e752040c09 Fix an assert() in fts5 that could fail if an xSavepoint() call on another vtab fails. 
FossilOrigin-Name: a5d7f5d24a239f729de0b8aa5becf5af35ba87160565ee0713c335b8d1bbf12d
 2019-12-26 14:36:31 +00:00
dan
4fcce747e2 Fix a problem causing fts5 integrity-check failures if numeric values are inserted into a table within a utf-16 database. Fix for [752fdbf6]. 
FossilOrigin-Name: bae060f382e4386c0793db4aa280fe9cd4893307d17a1801a17218ae4dc031c2
 2019-12-24 20:51:01 +00:00
dan
3fee9de239 Fix another case where malformed utf-8 was being mishandled in fts5. Fix for [df46a6f3]. 
FossilOrigin-Name: 1c0a05b09a97e6e2e9b11c31ed6ec7e6484686614b587ebfd0cfe27d973ba461
 2019-12-24 16:20:05 +00:00
dan
84d17bdaa7 Fix an assert() added as part of commit [a11b393dc] that can fail if fts5 database records are corrupt. 
FossilOrigin-Name: 4630c1ec013a4c2e54a34c9a64b2c803ac3912450de660497eb34ee21c91f426
 2019-12-24 15:35:53 +00:00
dan
c26e78d275 Fix a spurious report of corruption that could be made by the fts5 integrity-check in SQLITE_DEBUG builds if the fts5 index contains malformed utf text. 
FossilOrigin-Name: a11b393dc2c882cf0b3c47c3405bf43ca1d6459605bd39cccce4d32da653a72d
 2019-12-24 14:27:03 +00:00
dan
7a1d7c3968 Fix "PRAGMA data_version" so that it works the same way with locking_mode=PERSIST and journal_mode=PERSIST configured. Fix for 7a458c2a5f. 
FossilOrigin-Name: 45748e2db028ffbda5d3e747493721a5a89af3fbc06823081a2f27f570e40e73
 2019-12-22 14:29:55 +00:00
dan
42ae633ab9 Fix two more cases in fts5 where sqlite3_value_bytes() was being called before sqlite3_value_text(). Fix for e431c355. 
FossilOrigin-Name: a1ba9a37d7a68a6d31f8197c6350589ebe6a12f4e3c193a178dd7ead8bcd565a
 2019-12-20 19:41:01 +00:00
drh
b2ab3dcff0 Fix harmless compiler warnings. 
FossilOrigin-Name: 2bbd014c900e0d71fd30ec647739b82c5dc3aa237a0203f5a1db50593dae199e
 2019-12-20 14:08:34 +00:00
dan
73b21382d4 Ensure fts5 consistently calls sqlite3_value_text() before sqlite3_value_bytes(). This matters when storing blobs in fts5 tables within utf-16 databases. Fix for [cf36cb4e02]. 
FossilOrigin-Name: c16305eba0eb14368f2631b9a82b7eb4483c93f3b31257ee7c33f9b00c344379
 2019-12-20 13:24:11 +00:00
dan
db5ed35609 Avoid a buffer overread in fts5 that could occur when parsing corrupt configuration records. 
FossilOrigin-Name: 355afd77df21a2265871ca6d075f26b1fa121c7c2682cf512281944ff0c2186d
 2019-12-10 03:40:11 +00:00
dan
c19e22f397 Fix possible null pointer dereferences in the fts5_expr() scalar function. 
FossilOrigin-Name: c5d44143599f3fe98492b2b900fa3d77925c7be545096251055ceeab899a41f1
 2019-12-09 02:20:37 +00:00
drh
2bbcaee833 Fix various documentation typos suggested by Mark Benningfield. 
No changes to code.

FossilOrigin-Name: 08f09dc4f986a27f4e4e69f8226dbfaf0cfe1269cefc2830bf0fbef7b9bbda48
 2019-11-26 14:24:12 +00:00
dan
7e88d6364c Fix a memory leak in fts5 error handling code. 
FossilOrigin-Name: de6637e15d95b7aee2caea8b8d5b9865f276cbbf152ecf0401c4c7bfdf6dee6f
 2019-11-23 08:36:35 +00:00
dan
6637332988 Prevent direct and indirect recursive content= options in fts3/4 and fts5. 
FossilOrigin-Name: 2eb997327c2c369c10b7835be80112d77cb2a7cfa9fe4d564042984c21d94698
 2019-11-21 14:20:59 +00:00
dan
9930cfe878 Fix a problem in fts3 causing it to report corruption if a doclist contains consectutive rowid entries that differ by more than 2^63. 
FossilOrigin-Name: 0b0a3048f9df88bbc8979f2ac5247bee7d7bc1e585faca252610d2fcbe36d51d
 2019-10-28 13:54:59 +00:00
dan
ef2df8f343 In fts5, fix a case of overreading a buffer by 1 byte when counting characters in malformed utf-8. Fix for [dd1f67bf]. 
FossilOrigin-Name: 8d964e1c21d4cea699023e02b0616a75c5859dd083c9365cdcbc0676ebbdaae4
 2019-10-24 20:35:27 +00:00
dan
52612bec3c Fix a resource leak in fts5 that could occur if an auxiliary function is called from within a query that does not use the full-text index. 
FossilOrigin-Name: b528bdcd45db1b783ecd9739c3d3c890f04de7003f079668970eafaf8e23b2f3
 2019-10-20 08:26:08 +00:00
dan
2d82269ca8 Fix a problem with row-value IN(...) operators and virtual tables. 
FossilOrigin-Name: aa57d7abac0bb92d4d5fd4e093a11cf8efc04e4eed748b2a400d01f137250649
 2019-10-14 15:15:50 +00:00
dan
ae55737fbf Do not allow users to effectively disable fts5 crisismerge operations by setting the crisismerge threshold to higher than the maximum allowable segment b-trees on a single level. Fix for [d392017c]. 
FossilOrigin-Name: 86e497209217abb7bcb491a023cd353f3c7c9c103ebd9f58dd8661b12cf3694c
 2019-10-09 18:36:32 +00:00
dan
a6bd1871d1 Disallow fts5 page sizes greater than 65536 bytes - as there are 16-bit offsets used in the page header. 
FossilOrigin-Name: 75775c5ab44e497cb19be10397229637f1374f05c3244e8f92d6c54fcea94f5f
 2019-10-09 15:26:45 +00:00
dan
fbd79cc585 Fix problems with using the fts5 'rebuild' command inside a transaction that contains other updates of the same table. Fix for [e258f008]. 
FossilOrigin-Name: 238e0835714696aba0631f288fcc30ec5fddb43893d469c6bf017f386b3cddee
 2019-10-08 13:34:24 +00:00
dan
b186a622ee Disallow page-sizes smaller than 32 bytes in fts5. Also ensure the fts5 integrity-check works even when "PRAGMA reverse_unordered_selects" is true. Fix for [265e935b26]. 
FossilOrigin-Name: 8ab0aebdb3c2d6fb3160b2c58ce6cc0495a6ddd960878a6395958c837f3d1b71
 2019-10-07 20:36:18 +00:00
dan
03e025e217 Fix a problem with running ALTER TABLE ADD COLUMN statements within a transaction that writes to one or more virtual tables. 
FossilOrigin-Name: 31e85fbbc4cfd09a98b4a6dc6624f863816f376c6a4eef1fafc657c8e1abbc36
 2019-10-07 18:43:21 +00:00