362d21614e
Fix a crash in FTS incremental phrase processing that can occur if the second or subsequent token is much more common in the dataset than the first.
...
FossilOrigin-Name: 0bf438fc30582a08fddfc3cec49366ee17ae2abe
2013-10-14 20:30:51 +00:00
b084582299
Fix harmless compiler warning.
...
FossilOrigin-Name: 4b130f88fba216e088f61252bbcdde57ec7ee6a9
2013-10-12 02:33:22 +00:00
4f99189051
Fix various harmless compiler warnings. Change the "warnings.sh" script to
...
work with STAT4 instead of STAT3.
FossilOrigin-Name: 7df06684ab36bfdad9e9aca6940b7a665c2a0cb5
2013-10-11 15:05:05 +00:00
86f72f0721
Fix compilation issue with MSVC.
...
FossilOrigin-Name: 36d64dc36f18c166b2c93c43579fa3bbb5cd545f
2013-10-07 21:49:16 +00:00
9211f8a6ce
Allow FTS4 multi-token phrases to use a combination of in-memory and incrementally loaded doclists. This allows phrases to (partially) benefit from incremental doclists without disabling the deferred token optimization.
...
FossilOrigin-Name: f6819c5f3363d358e7ef65fe6978f13991bd44af
2013-10-03 19:27:14 +00:00
eb26672d0e
Add a test to check that the new multi-token phrase optimization is actually helping.
...
FossilOrigin-Name: bc3a2ed5fb2402805928b0002457eebf06f87c47
2013-10-02 08:04:27 +00:00
ff574f4c76
Allow multi-token phrases to load doclists from the database incrementally. This allows queries that feature such phrases to benefit from the "docid<?" optimization.
...
FossilOrigin-Name: ea543f081d93ed1bf66c21ce2108ec94e349f4c5
2013-10-01 20:02:32 +00:00
73632e37c9
Merge trunk changes with this branch.
...
FossilOrigin-Name: e294a9c7c525d1da698259c2fef089579bfd1c0b
2013-09-30 18:16:07 +00:00
1f03e609ff
Fix a performance problem in the FTS4 auxiliary functions triggered by an OR clause in the full-text query.
...
FossilOrigin-Name: fa0f2f0e3e79ae653118b901e1cca7725dfaf249
2013-09-30 18:14:45 +00:00
19384380f3
Have fts4 full-text queries consider "docid<?" and similar constraints.
...
FossilOrigin-Name: 6622424a3a149edd35ba2ba0881aa41b4536417b
2013-09-30 11:42:19 +00:00
2505a5fee2
Fix an FTS4 problem introduced by [361084e1eb].
...
FossilOrigin-Name: 5f35c8cb8f8b67121c70f98c02c4aa7c25327690
2013-08-21 15:45:27 +00:00
2c18788ffa
Initialize a variable in fts3_write.c on the grounds that the argument required to show that it does not require initialization with is complicated. Add an assert() to where.c to silence a clang scan-build warning.
...
FossilOrigin-Name: d6c4d48a002a6d7057fccc30064ce0b049678f0c
2013-08-19 19:29:50 +00:00
290703091e
Add some extra assert() statements to silence a few clang warnings.
...
FossilOrigin-Name: 1c63e9515b5f5cb9169928b108d002dcd18ad4fd
2013-08-19 18:17:03 +00:00
1f10c28618
Modify several extensions to use the new exported function naming.
...
Fix some shared library compilation issues.
FossilOrigin-Name: 1e39f85077f1f2b96c3a656c5b6334bafb005908
2013-07-09 12:36:30 +00:00
ed20073b8d
Make a couple of implicit casts explicit to fix compiler warnings.
...
FossilOrigin-Name: 9676280a9387b8bfba82cfc0087318863d05d4f3
2013-07-05 10:46:08 +00:00
049d487e2e
Modify several extensions to use the new exported function naming. Fix some shared library compilation issues.
...
FossilOrigin-Name: f2ab8747825ab5131ffab174aa0ffe5e474f6811
2013-07-04 23:53:56 +00:00
4580cfb93e
Cut over the next generation query planner. Increase the version number
...
to 3.8.0.
FossilOrigin-Name: 0fe31f60cadc5fe5a9d87e110bfaed5fd026cba1
2013-06-26 11:43:18 +00:00
adbae616bd
Remove an unused function declaration from the FTS3 source code.
...
FossilOrigin-Name: 096ae1d8f9a08f92daedece6b0615f4d22b05023
2013-06-21 18:36:44 +00:00
8def92bac3
Add the "notindexed" option to fts4.
...
FossilOrigin-Name: 8ff2b8f5948ccddce70102e6d68d464c66e4f7ca
2013-06-21 17:30:47 +00:00
37ca04880a
Merge all changes from trunk.
...
FossilOrigin-Name: f2e15b1974e55373b5819e3b2326653f890f409f
2013-06-12 17:17:45 +00:00
5f0b33c2cb
Add the SQLITE_FTS3_MAX_EXPR_DEPTH compile time option.
...
FossilOrigin-Name: 24fc9d4438a5615dd20af5419456166df83a72ea
2013-06-11 14:22:11 +00:00
f2c9229f73
Up until now the fts4 "unicode61" tokenizer has treated all private use codepoints except the first and last of each of the three ranges as alphanumeric (eligible to be part of tokens). This commit fixes this so that all private use codepoints are considered alphanumeric. In other words, it fixes the handling of codepoints 0xE000, 0xF8FF, 0xF0000, 0xFFFFD, 0x100000 and 0x10FFFD.
...
FossilOrigin-Name: 6cfd9af5250029c0d275be027b4208c48954a8a1
2013-06-05 16:17:21 +00:00
73843cac73
Increase the estimated full-scan cost for an FTS3 table.
...
FossilOrigin-Name: 8d9f29f847f38612447c79a5fb366a9143ef6205
2013-06-04 13:07:44 +00:00
87ddfeb079
When writing to an FTS table, take an exclusive shared-cache lock on the %_segdir table before writing. Otherwise, an xCommit() call may report an SQLITE_LOCKED error.
...
FossilOrigin-Name: 3cd2da42e9403b1e6243ad53f3f2bbf89c0fb9b0
2013-05-27 10:11:53 +00:00
92692d32ad
Fix a couple of harmless compiler warnings in the fts3_tokenize virtual
...
table.
FossilOrigin-Name: 1fa8c457394c94864f7584e4c893ec09e685fba4
2013-05-08 17:06:28 +00:00
e8dee7d1ed
Modify the fts3tokenize table implementation so that it does not use the SQL function fts3_tokenizer. The user may have installed an authorizer callback that prohibits this.
...
FossilOrigin-Name: 0ba67b64de258883e4c43db09e131bb67083855e
2013-05-07 12:16:48 +00:00
acf9c2b555
Add an assert() to fts3_expr.c in order to silence a clang warning.
...
FossilOrigin-Name: d8dc2c5fb554eb5898ddc8d703c6a360af7fe023
2013-05-06 14:57:48 +00:00
3a01b600d3
Improve the error message issued when an FTS query exceeds the maximum allowable tree depth.
...
FossilOrigin-Name: f480b1fe6012f36c59cd0525efdc6df74143ccd0
2013-04-29 18:07:37 +00:00
181f4f789d
Fix an off-by-one in the code for limiting the depth of FTS expression trees.
...
FossilOrigin-Name: 72ac73189c3577740a77d2ea2fc7118391c0703f
2013-04-29 17:12:06 +00:00
6f77140f78
Fix harmless compiler warnings in the FTS expression parser.
...
FossilOrigin-Name: 3c78af8c535e16518f18733325f4cd1df7ec8282
2013-04-26 13:14:19 +00:00
aa29c86e49
Rebalance FTS expressions after parsing to limit recursion during evaluation. Avoid recursion when deleting FTS expression trees. Enforce a limit on the depth of an expression tree.
...
FossilOrigin-Name: f968d43f80cc2f236e7d09ba1e8278343e2b6976
2013-04-25 20:34:02 +00:00
da8caa0b2d
Fix harmless compiler warnings.
...
FossilOrigin-Name: 1a1cf5aa86734c832d845e07780262a178188d56
2013-04-22 23:38:50 +00:00
4d648c777b
Update MSCV makefile for the fts3tokenize virtual table. Clear some
...
harmless compiler warnings in the fts3tokenize code.
FossilOrigin-Name: 2b3fed09eb89f1e9d6a5f7487ef526b46eb51490
2013-04-22 17:07:56 +00:00
d7a959c694
Add virtual table module 'fts3tokenize' to fts3. fts3tokenize provides SQL access to FTS tokenizer implementations.
...
FossilOrigin-Name: c5a4b21a15bd8a184380817ff41657acbccb866d
2013-04-22 15:30:37 +00:00
a8f60381a9
Change the fts4aux virtual table module so that fts4aux tables created in the temp database may report on fts3/fts4 tables in any attached database.
...
FossilOrigin-Name: 546a187f1361dad340ed8f6c28dd89e6c19f9c33
2013-04-12 16:47:27 +00:00
1cb90c668c
In fts3, when filtering lists for hits in a specific column, edit the list in place in the same way as it is for NEAR filtering. Fix for [38b1ae018f].
...
FossilOrigin-Name: f85f9103cffa5c8ba6a63a68beb90817147ba080
2013-03-25 11:38:44 +00:00
48864df97d
Many spelling fixes in comments. No changes to code.
...
FossilOrigin-Name: 6f6e2d50941e444ebc83604daddcc034137a05b7
2013-03-21 21:20:32 +00:00
7b6be6f7bf
Fix an uninitialized variable detected by valgrind.
...
FossilOrigin-Name: ffd1e51490286abfaea68fd4b4b4cb967d87b04b
2012-12-04 00:53:08 +00:00
bba44d0d1b
Fix a problem causing the "number-of-documents" field maintained by FTS4 to be set incorrectly.
...
FossilOrigin-Name: e38fb02d5ea5daa6992df4dfbbeec92bf7b525f6
2012-11-27 15:56:38 +00:00
e62c0694de
Fix a bug in fts3 causing it to return SQLITE_NOMEM when NEAR, OR and some auxiliary functions were used together. Also a segfault in vtab.c that could follow an OOM condition.
...
FossilOrigin-Name: 33f4f9817e8b3cb97ce02a4c49bd586332da37b2
2012-10-25 19:25:39 +00:00
8749c1831a
Make sure substructure elements have proper alignment in the ICU tokenizers
...
of FTS2 and FTS3.
FossilOrigin-Name: aaa2d9b0db74d8452d9294de17cff786ab4ec7c8
2012-10-19 02:10:53 +00:00
1276cd0ba8
Add a comment to fts3_snippet.c.
...
FossilOrigin-Name: b5f5ba3103e6fcf117fb69d5379fa70f691a652d
2012-10-17 20:28:52 +00:00
6c42aa4a0e
Initialize variables used as outputs of custom tokenizer method calls in fts4.
...
FossilOrigin-Name: 514505a84bcc677518d60a42896598f85ed78265
2012-10-17 20:15:10 +00:00
d5578433ff
Fix all known instances of 'repeated the' style typos in comments. No changes to code.
...
FossilOrigin-Name: 9b19b847533f944f289d93dcdba29c0d67bf251c
2012-08-25 10:01:29 +00:00
4dc3d73d69
Add the SQLITE_DISABLE_FTS4_DEFERRED compile time option.
...
FossilOrigin-Name: e799222f3b8246e65657a758437914ece7069ba9
2012-08-20 17:24:48 +00:00
fe7b2bc9ff
Merge several compiler compatibility fixes to trunk.
...
FossilOrigin-Name: d3d491a5461e21d84c6425977b80deedc7fa8a3a
2012-06-21 14:05:15 +00:00
2ae26b759d
Fix a problem with identifying white-space characters outside of the ascii range in the ICU tokenizer.
...
FossilOrigin-Name: 892b74116a3b23268895b96433d18ef00c1433d8
2012-06-18 20:52:32 +00:00
c8bde37d19
Fix a few compilation issues that can occur with certain compilers (e.g. GCC 2.95.3, MSVC).
...
FossilOrigin-Name: f970a3de61fe2ebaf1778c4a3383cfdc52299162
2012-06-18 08:00:56 +00:00
1c30debd41
Fix an obscure memory leak in FTS3 that can come about when two memory
...
allocations are immediately adjacent to one another.
FossilOrigin-Name: 025227be5495f950c466dfabac140cba69e498be
2012-06-08 14:01:53 +00:00
25cdf46ae4
Add the "tokenchars=" and "separators=" options, for customizing the set of characters considered to be token separators, to the unicode61 tokenizer.
...
FossilOrigin-Name: e56fb462aa1f11bb23303ae0dc62815c21e26a52
2012-06-07 15:53:48 +00:00
2c897e3e5f
Disable FTS unicode61 by default. It is enabled by specifying compile time option SQLITE_ENABLE_FTS4_UNICODE61.
...
FossilOrigin-Name: eccd6b6580637084495b80e9232262188ba0cf8d
2012-06-06 19:51:27 +00:00
754d3adf7c
Have the FTS unicode61 strip out diacritics when tokenizing text. This can be disabled by specifying the tokenizer option "remove_diacritics=0".
...
FossilOrigin-Name: 790f76a5898dad1a955d40edddf11f7b0fec0ccd
2012-06-06 19:30:38 +00:00
a9cfaba95a
Omit the fts3 unicode character class routines from the build if fts3/4
...
is disabled.
FossilOrigin-Name: c00bb5d4601efc15933f222349e96a043b610a19
2012-05-28 12:22:00 +00:00
7946c53009
If SQLITE_DISABLE_FTS3_UNICODE is defined, do not build the "unicode61" tokenizer.
...
FossilOrigin-Name: e71495a817b479bc23c5403d99255e3f098eb054
2012-05-26 18:28:14 +00:00
501c74d3e1
Change the format of the tables used by sqlite3FtsUnicodeTolower() to make them a little smaller.
...
FossilOrigin-Name: b89d3834f6690073fca0fc22c18afa1fb280ea7d
2012-05-26 17:57:02 +00:00
cf9f6f1375
Add fault-injection tests that use the unicode61 tokenizer. Fix a problem revealed by the same.
...
FossilOrigin-Name: ed28c48a3dd7e766e60db0d96ef5460bf9913e6b
2012-05-26 15:44:08 +00:00
ab322bd21e
Change the name of the "unicode" tokenizer to "unicode61" to emphasize that the case folding and separator-character identification routines are based on unicode version 6.1.
...
FossilOrigin-Name: 8f3e60aa2253f21bcee5d03982cfdd7f16c00060
2012-05-26 14:54:50 +00:00
1c7016c9a5
Add special fast paths to sqlite3FtsUnicodeTolower() and Isalnum() for codepoints in the ASCII range.
...
FossilOrigin-Name: cf7b25d47687635a04f4347d45f135c686b9d758
2012-05-25 19:50:12 +00:00
80ed5a56a5
Fix comments in generated file fts3_unicode2.c.
...
FossilOrigin-Name: 3dc567ef4702d9a63d78d11ff705cb7f7359f7a6
2012-05-25 18:48:48 +00:00
3d403c71a8
Add an experimental tokenizer to fts4 - "unicode". This tokenizer works in the same way except that it understands unicode "simple case folding" and recognizes all characters not classified as "Letters" or "Numbers" by unicode as token separators.
...
FossilOrigin-Name: 0c13570ec78c6887103dc99b81b470829fa28385
2012-05-25 17:50:19 +00:00
5cfed52dd1
Have the FTS auxiliary functions correctly handle terms that appear in non-matching branches of the FTS expression. Fix for [bdc6bbbb38].
...
FossilOrigin-Name: 4d3e1673b309a3d6a6cac01d67dfd79a8007f991
2012-05-10 17:43:14 +00:00
83cc139231
Fix harmless compiler warnings on x64 MSVC, mostly in test code, but also in
...
tclsqlite.c and in the FTS4 module.
FossilOrigin-Name: 3281972eaa46cb57fd9f0387063f47430dc0a3b4
2012-04-19 18:04:28 +00:00
85e7243acf
Fix harmless static-analysis warnings, mosting having to do with memory
...
leaks in the command-line shell. Add a clang analysis of the command-line
shell to the "warnings-clang.sh" script. Other minor cleanups to the
command-line shell code.
FossilOrigin-Name: 93a0f452a7023898ad3d62ee81b39a80477c332f
2012-04-11 11:38:53 +00:00
3b06a2a056
Add #ifdefs to allow a test build to succeed even if SQLITE_ENABLE_FTS3 is
...
not defined.
FossilOrigin-Name: fb121980e48af368353431fd04924e414b65c852
2012-04-02 17:18:23 +00:00
a25a42cfe7
Change the ICU tokenizer so that it does not attempt to call strlen(NULL).
...
FossilOrigin-Name: 04298f1ac42c40cb2a48092b415acf96a08954b7
2012-03-31 11:58:23 +00:00
7da5fcb0b7
Fix MSVC compiler warnings in test code.
...
FossilOrigin-Name: cb7a850439c9a4a7887650d6b81d95ab8025de5b
2012-03-30 14:59:43 +00:00
4e245a4c35
Fix compiler warnings on GCC and MSVC and fix a C89-ism that
...
broke the build for MSVC.
FossilOrigin-Name: b451c0f97f0abe78ebe6c62ff489ec1ad8a1f767
2012-03-30 00:00:36 +00:00
cdb86dc437
Add output of PRAGMAs auto_vacuum and encoding to the "schema" command of the
...
fts3view utility program.
FossilOrigin-Name: e31076319363a46905836880765bae3bf204ed19
2012-03-27 18:00:05 +00:00
311ec02587
Remove the fts3merge.test script in favour of changing the fts4merge.test script so that it runs tests using both fts4 and fts3. Fix some problems with incr-merge and FTS3 tables.
...
FossilOrigin-Name: 5c447e226afca0d46b9ed994dea26a16a9ae168c
2012-03-27 15:00:06 +00:00
a1a9f0a0d8
In the fts3view utility, label the blank segments used to mark the end of a
...
segment sequence for a level/idx as "null". Improve the alignment of root
segment names.
FossilOrigin-Name: 04aea0245e4183fef3664609f5a6353b65d71a85
2012-03-27 14:54:44 +00:00
1ecb59a1e5
Enhance the fts3view tool with the big-segment command and fix a bug in the
...
display of doclists.
FossilOrigin-Name: e9436d8038e5a0d1ba992a77d1064d4a55595f57
2012-03-27 13:51:31 +00:00
5da0aa1603
Allow multiple incremental merges to proceed concurrently. This is required to prevent a large crisis-merge from occuring while an even larger incremental-merge is underway.
...
FossilOrigin-Name: 7ed9d2f24a650b424b97dfc19b8042c4cf09c82c
2012-03-27 11:48:02 +00:00
40aab4c56d
Minor correct errors in the file format description for FTS3/4 contained in
...
the fts3.c header comment.
FossilOrigin-Name: fb8aacdd8fbdc946cb271cc589f76b806387937d
2012-03-27 00:38:33 +00:00
280d5c33cd
Enhance fts3view to show decodes of segments and doclists.
...
FossilOrigin-Name: 6d09de231b68dd9520d99c65d133f26e90eb784f
2012-03-27 00:34:04 +00:00
36f6b891e6
Add the fts3view utility program.
...
FossilOrigin-Name: f936c8ea16d21345fd1622272dc7e9850acb2493
2012-03-26 21:57:53 +00:00
cbf9450404
Fix FTS3 so that it works even without SQLITE_DEBUG.
...
FossilOrigin-Name: a18c103121529c2e3c6a8ada16a4c40d14080670
2012-03-26 14:36:42 +00:00
6f4df8ac83
Modify the FTS integrity-check so that the checksums do not depend on the results of signed integer overflow, which is undefined in C.
...
FossilOrigin-Name: f907fc3fb387e74bb66babcbf050748cb253a6fa
2012-03-26 10:57:31 +00:00
84bce14f77
Add a comment to explain how the FTS integrity-check works.
...
FossilOrigin-Name: 64e8a116f39434a3b7347f01a47f88eef3276742
2012-03-26 10:47:03 +00:00
cbcd9f5357
Add an experimental integrity-check function to FTS.
...
FossilOrigin-Name: 40fc8804743dfb005991e9c5ef7b0ebcb3c2e731
2012-03-26 10:36:55 +00:00
6c2e7e19fa
Enable fts3 tables to use incremental merge by automatically creating the
...
%_stat table when it is needed.
FossilOrigin-Name: cc051fc0b2d89603b27b94cf2afdbda417ee9d94
2012-03-24 17:29:05 +00:00
61fa09dbf2
Fix a failing assert() in the FTS3_LOG_MERGES related code.
...
FossilOrigin-Name: 4220d52cb3426f1680b72d57ecc9f4ade029357d
2012-03-24 17:09:11 +00:00
0dfcdeb4f0
Add a comment to fts3SyncMethod() to justify the nMinMerge=64 constant.
...
FossilOrigin-Name: af55ca5fc6778cb6d1a79a17dfa2d4e567ea1ccc
2012-03-24 16:43:55 +00:00
790964ebc4
Fix a bug in debugging code enabled when FTS3_LOG_MERGES is defined.
...
FossilOrigin-Name: 2e06babf49de844d0e878d68114dbcc5ad4d6e54
2012-03-24 16:18:08 +00:00
45eddd6855
Remove the Fts3Table.mxLevel variable.
...
FossilOrigin-Name: 67a0cffc9d07be7f09dad9d019a18160711295cd
2012-03-24 16:11:21 +00:00
4b1e4dabc3
Modify the way the number of leaves written and the maximum relative level are calculated in the auto-incr-merge code.
...
FossilOrigin-Name: 0d841c957c6ec4afecb49504177c6279e09d7012
2012-03-24 14:45:19 +00:00
4ef9dfff97
An attempt at automatic incremental merging for FTS4.
...
FossilOrigin-Name: ed69434cd89084f4b57bd2cc4f5cc558904af565
2012-03-24 02:20:43 +00:00
d59de73e27
Fix a spurious SQLITE_CONSTRAINT error that may be returned by an incr-merge operation.
...
FossilOrigin-Name: ed7c17ea165f6348506bd23ebc58c427bb65d697
2012-03-23 18:26:11 +00:00
3501a91677
Remove an incorrect assert() statement. Fix a const-related warning.
...
FossilOrigin-Name: 96ed47493b3d46344fd2105642f31690aee06674
2012-03-23 14:38:49 +00:00
e81eaec754
Following an incr-merge operation that does not completely consume its input segments, store context in the rowid==1 row of the %_stat table that allows the next incr-merge to pick up where the previous left off.
...
FossilOrigin-Name: ab0a4f44fb67e9f0cb82297b80e728ca58cdb0fb
2012-03-22 16:48:12 +00:00
e239233982
Merge trunk changes into the fts4-incr-merge branch.
...
FossilOrigin-Name: f61d5fb0281381228eb1a12a233bacaeb26b12a3
2012-03-20 17:04:17 +00:00
d13a3bc772
Fix one more compiler warning missed by the previous check-in.
...
FossilOrigin-Name: bc03d99a78e90c02b69037e5f5f81537b5a3ac60
2012-03-19 14:57:49 +00:00
01ea399a84
Suppress harmless compiler warnings on windows in FTS4 and RTREE.
...
FossilOrigin-Name: 4fd68647c8d4b120e04d054617cef31001f44c6f
2012-03-19 14:51:19 +00:00
bbe0520329
Avoid a compiler warning (an incorrect compiler warning, at that) in vs2010.
...
FossilOrigin-Name: 7dd97f12cd268cac1241f0f3e8de94bb629b97c7
2012-03-19 14:28:43 +00:00
d1ab097d45
Fix various incorrect and missing comments and other style issues in and around the FTS incremental merge code.
...
FossilOrigin-Name: 7aabb62c8ccbd2b8d216e25226f06e5820dec38a
2012-03-17 16:56:57 +00:00
c8d7b24e93
Cherrypick the [3475092cff] fix for 32-bit overflow with large language-ids
...
into trunk.
FossilOrigin-Name: 2755edc7f1186ba8955f2c3550a90f40a55ddae2
2012-03-16 16:52:54 +00:00
81b35dc66c
Fix some integer overflow problems that can occur when using large langauge id values.
...
FossilOrigin-Name: 3475092cff862080a020d386076d739f0d22c9b2
2012-03-16 15:54:19 +00:00
e931b7f45f
Add a comment to the FTS getAbsoluteLevel() function. No actual code changes.
...
FossilOrigin-Name: 7e0f861beda4d74d0c3c9fb4abb3ddb5fee346bd
2012-03-16 14:54:07 +00:00
7a48480d07
Fix harmless compiler warnings.
...
FossilOrigin-Name: 74eadeec34c4b19cf5f8b7f648db3b7ad601a00e
2012-03-16 00:28:11 +00:00
4ab6f2b9c1
Modify incremental merge code to merge nMin segments at a time.
...
FossilOrigin-Name: cd34bc1af4ba608ea3b52bab55bcfe0086711900
2012-03-15 17:45:50 +00:00
5730ef599c
Add tests for incremental merge code.
...
FossilOrigin-Name: 570473729d6561d81e6e5f8884fd18487008636e
2012-03-14 20:01:52 +00:00
a2af0aecdb
Avoid allocating a large object on the stack in the incremental merge code. Use sqlite3_malloc() instead.
...
FossilOrigin-Name: 36ae510de45be44efd34cff242d02fb21b7419ac
2012-03-14 12:17:40 +00:00
8af1fc721e
Fix another bug in the incremental merge code.
...
FossilOrigin-Name: f97b12e0955c4c29f9c31a186d72d87f7407782e
2012-03-14 11:51:31 +00:00
604f58a686
Fix some bugs in the incremental merge code. Some remain.
...
FossilOrigin-Name: bff21683705a61b8b8672e0b44c287d1dc7c32a9
2012-03-13 19:56:34 +00:00
ab4f6385de
Minor commenting and stylistic changes only.
...
FossilOrigin-Name: a1747086c5e0c152fcf4bd9fa80a61b6f03f4a94
2012-03-09 12:52:43 +00:00
593c9824bc
Add the 'merge=?,?' command to fts4. This still needs some work.
...
FossilOrigin-Name: 741b8f897750eac3c9774fd65de7e40bb89781b1
2012-03-08 18:39:03 +00:00
fd6ddf91b0
Fix a problem compiling the test code in fts3_test.c when SQLITE_ENABLE_FTS3 is not defined.
...
FossilOrigin-Name: b00ccda307caae597c143ab0586f90acb77f79cf
2012-03-05 16:24:26 +00:00
3a1abde7c8
Merge the fts4-languageid branch with the trunk.
...
FossilOrigin-Name: 99a9073b5e411ce94f38ce49608baaa15de8b850
2012-03-05 15:33:32 +00:00
996073b3ae
Add the xLanguageid method to sqlite3_fts3_tokenizer versions 1 and greater.
...
FossilOrigin-Name: f8e9c445dd358c40e5a7bf3756b9f291909dbea7
2012-03-03 18:46:41 +00:00
9db1d56f02
Remove an obsolete comment from FTS3. No code changes.
...
FossilOrigin-Name: 74d224b14467d8a6915d3cf372f45e0d28708fa2
2012-03-02 20:34:57 +00:00
7395599031
Fix problems with combining content= and languageid= in a single fts4 table.
...
FossilOrigin-Name: 22491e7bc38aee43819b888e04241cb6a6ef73a3
2012-03-02 19:53:02 +00:00
8fb662b0be
Fix the FTS 'optimize' command on multi-lingual databases.
...
FossilOrigin-Name: 65fa693729a336e4d905ce72e6b9ccf4faa772bd
2012-03-02 12:26:19 +00:00
e3ddd1ac42
Fix a bug in merging FTS language tables for languages other than language 0.
...
FossilOrigin-Name: d281cb8984c911a4c0cce2ec299e1351d8e580e4
2012-03-02 11:48:50 +00:00
7fcb214b93
Add the "languageid=" option to fts4. This code is still largely untested and alsmost certainly buggy.
...
FossilOrigin-Name: bea257f70f10dd1111d79cabd1e1462dc651704d
2012-03-01 19:44:20 +00:00
548af8b69e
Fix a compiler warning in FTS4.
...
FossilOrigin-Name: b4cad1ce78e6bb967c472bec5c562a24b4dfebaf
2012-01-31 23:34:21 +00:00
585db0e23a
Fix an FTS problem triggered by querying for an N character prefix using an N+1 character prefix index after rows have been deleted from the FTS table.
...
FossilOrigin-Name: c05c3fd20d93f430140d762ead23bacd337ffb4d
2012-01-25 16:29:45 +00:00
45d29309a7
Fix typos in commands. Combine the ExpandBlob and expandBlob macros into one.
...
FossilOrigin-Name: fc9179e154751b51ddef429600d30a89d4a098bd
2012-01-08 22:18:33 +00:00
64803bd2fa
Fix minor issues with FTS and RTREE discovered by coverity.
...
FossilOrigin-Name: 1c27d842163e27c39bbe9409f50657b9de9ade6e
2011-12-22 17:31:17 +00:00
0c8cda6e0d
Fix a problem in fts3_write.c causing stack memory to be referenced after it is out of scope.
...
FossilOrigin-Name: f9c4a7c8f4e5821b47c1393d6272e32416d8886c
2011-12-22 15:30:46 +00:00
689e38f7cf
Add a missing va_end() macro to fts3.c.
...
FossilOrigin-Name: e6f825748a1d510c8f5529c79656bb5db9216231
2011-11-04 10:37:21 +00:00
042f88e5b4
Add new file ext/fts3/README.content, describing the experimental FTS4 content option.
...
FossilOrigin-Name: 13a9d085e1a5654a97b8d26bae7182ca6c0c237b
2011-10-31 11:36:29 +00:00
6ed18b7110
Fix harmless compiler warnings in FTS4.
...
FossilOrigin-Name: 1af4a25631c4077a9bfc64e15eb92fbf5146c89c
2011-10-19 18:21:47 +00:00
e459b07601
Merge the fts4-content branch with the trunk.
...
FossilOrigin-Name: 8a4077057ddeb08e8edc5f20a75abaaba7a278ba
2011-10-19 16:20:40 +00:00
9743948386
Have FTS3 ignore ^ prefixes. The ^ syntax is only supported on FTS4 tables.
...
FossilOrigin-Name: df36ac948179f37b432a88701b6c79299e073ce8
2011-10-19 15:52:48 +00:00
d17f70a624
Change the way ^ tokens work in FTS so that the filtering is done as part of reading the FTS index instead of waiting until an entire doclist has been retrieved and then filtering it.
...
FossilOrigin-Name: 9b58c59eb4efaa38ce50a3ce1b52f9ba578c71d6
2011-10-19 11:57:13 +00:00
50a7544d6f
Fix a problem in FTS to do with ^ tokens and the snippet() function.
...
FossilOrigin-Name: 2c03b24f4cc6f2c28c9d5b9984320d41b8486c32
2011-10-19 09:40:49 +00:00
3f1ea8d114
If a token within an FTS query is prefixed with a '^' character, it must be the first token in a column of data to match.
...
FossilOrigin-Name: 63ac33c860eb32ce96699f06bf83121cec2ffaca
2011-10-18 19:39:41 +00:00
a986d33fd3
Cherrypick patch [3126754c72] from the trunk into the content= branch.
...
FossilOrigin-Name: f9b5b217088a6aeb25eba184ab92d1a842a680a6
2011-10-18 12:49:59 +00:00
42495cd46f
Fix a floating-point exception that can occur when an FTS4 query contains a large number of tokens connected by AND or NEAR operators.
...
FossilOrigin-Name: 3126754c72351c724be29d75a194bfc3e7b67205
2011-10-18 12:44:04 +00:00
30ddce6ff4
Added the tool/warnings-clang.sh script. Changes so that there are no
...
warnings with either gcc or clang even including FTS4 and RTREE and
both with and without SQLITE_THREADSAFE=0.
FossilOrigin-Name: 39408702a989f907261c298bf0947f3e68bd10fe
2011-10-15 00:16:30 +00:00
086e4913b5
Fix a few minor and harmless clang warnings in FTS3 and RTREE.
...
FossilOrigin-Name: b3324f6cc27c3bfb32b12eacace2fc731c2dd644
2011-10-14 22:57:03 +00:00
8dab211632
Handle updating the only row of an FTS table correctly. Fix for [9fd058691].
...
FossilOrigin-Name: 7e24645be2fe0ffe092212e7bcfa5b4500305811
2011-10-13 17:16:45 +00:00
deb9473250
Change FTS4 so that if both the content=xxx option and column names are specified, the virtual table assumes that the named columns correspond to columns of table xxx.
...
FossilOrigin-Name: 289ee43179369fce2fde50870d72c445e184e896
2011-10-05 15:11:30 +00:00
8361b189c9
Fix a problem with IO error handling in the rebuild-index code.
...
FossilOrigin-Name: c6ba81fcad32192674bd510e607f787adc1f7038
2011-10-05 06:07:00 +00:00
49fc336438
Improve test coverage of fts3.c.
...
FossilOrigin-Name: 0f439944ab49a5691615bc170fdcf652055573df
2011-10-04 19:41:50 +00:00
cc5b81464a
Add tests to check that modifying the schema of an FTS content table does not cause a crash in the FTS module. Also disable the deferred token optimization for content=xxx FTS tables.
...
FossilOrigin-Name: be86c7061b68f403730bf63ea1f7dc0d9ceb0a3b
2011-10-04 16:37:35 +00:00
d8e454e19a
Add experimental 'content' option to FTS4.
...
FossilOrigin-Name: 1d27ea741f61c624e18bdc6a3b1c2d8574a64ddc
2011-10-04 11:22:59 +00:00
f9556a1a3d
Allocate the correct size for the output buffer in fts3DoclistOrMerge(). Fix for [56be976859].
...
FossilOrigin-Name: 07788c0f7f3740c1c280f6ce4dc68401c30bae6e
2011-09-13 19:08:43 +00:00
ec661058dd
Fix harmless compiler warnings on unix.
...
FossilOrigin-Name: 90b1aea17400bbda5ebc8ae4eb4e12127519e42e
2011-07-09 16:17:18 +00:00
f443e317f7
Remove accidental global variable "doclist" from fts3Int.h.
...
FossilOrigin-Name: f244b2f3959e86fac555b91438cfe90528ab1648
2011-07-07 07:37:53 +00:00
da7b45615d
Add a fix and tests for the FTS deferred token logic.
...
FossilOrigin-Name: 91daea7d2ec41f014fb30c6371aae447cc07f287
2011-06-28 11:58:09 +00:00
b8ae1afba1
Merge latest trunk changes with fts3-changes branch.
...
FossilOrigin-Name: 226686475c28cdc87e86d2d3f10014436a9a2b8f
2011-06-28 09:51:22 +00:00
0ea2517c4e
Changes to allow FTS to be compiled as a loadable module again.
...
FossilOrigin-Name: 29e69f389cb0078e125ba5814a68b3fe8cf634c6
2011-06-28 07:15:43 +00:00
5b217c308f
Changes to improve the selection of deferred tokens within phrases.
...
FossilOrigin-Name: 2c4bbd90e2fca593c186bf412b608aff8c9f9061
2011-06-27 11:15:53 +00:00
17fb042578
Fix some of the code issues (missing comments etc.) in the new FTS code.
...
FossilOrigin-Name: 8230d83120e0f4d217fde56e22c6f05aa5adee09
2011-06-23 17:09:51 +00:00
45de8d5f0f
If "PRAGMA page_size" commands are not authorized, the FTS module will assume
...
a page size of 1024.
FossilOrigin-Name: ba39382ef546b4c4ae15ed90ef0b6c87762bffef
2011-06-23 00:59:41 +00:00
2e2f36b4ed
Fix a couple of compiler issues on Windows due to warning clean up.
...
FossilOrigin-Name: 81e6f151d384fa1f1474175b39ad6dfa64f8da0d
2011-06-20 20:50:49 +00:00
7fd3392981
More compiler warning fixes.
...
FossilOrigin-Name: ed2dda9329ca42e9c0be1986c78b091051e7598f
2011-06-20 19:00:30 +00:00
051eb38a26
Fix various harmless compiler warnings in FTS3 and RTREE.
...
FossilOrigin-Name: fe62179efdea40887e23a16f898060b879013824
2011-06-20 17:24:29 +00:00
6764a70086
Fix a problem where FTS test code was not being included in the testfixture build when SQLITE_ENABLE_FTS4 was defined.
...
FossilOrigin-Name: e539d08a4dda073dc67dc39a104f89374d3bd8e9
2011-06-20 11:15:06 +00:00
c5f21892a8
Avoid loading doclists for infrequent terms that are part of phrases twice.
...
FossilOrigin-Name: 8f939723f742329cedba8930f71dff42004f3d0d
2011-06-17 17:37:31 +00:00
10fc4d8c45
Add a missing declaration to fts3Int.h.
...
FossilOrigin-Name: 3bfd4466f50711eb71d1a13231025ff4e1e76246
2011-06-17 16:04:39 +00:00
d4d21fea8e
Fix a problem with NEAR queries executed inside a transaction that writes the FTS table.
...
FossilOrigin-Name: 051c756c367837908f6691c0a36108e088c94f99
2011-06-16 16:06:05 +00:00
6b96771b82
Changes to #ifdefs so that the build goes correctly if the only FTS macro
...
defined is SQLITE_ENABLE_FTS4.
FossilOrigin-Name: a0b43a320e6491de7070966ed7c3ec55fd660a85
2011-06-16 00:54:45 +00:00
bf4bc8c254
Fix so that the TCL test harness works even if SQLITE_ENABLE_FTS is omitted.
...
FossilOrigin-Name: 63ebcb52a1909aca80f2fef3e982f8fb5929b73b
2011-06-15 13:02:21 +00:00
1aa4f3e529
Fix a couple of compiler warnings.
...
FossilOrigin-Name: 3899f3b95ed50773a72d907b39b451fefce69c9e
2011-06-15 12:43:36 +00:00
376d1e5c4f
Add an interface to better test incremental loading of doclists by FTS4. Also some tests for this and term prefix queries.
...
FossilOrigin-Name: 7a3813138d1a5c5d99f4756c79998831d779774f
2011-06-15 08:30:29 +00:00
2cf1a1de4b
Fix another bug caused by NEAR/matchinfo/order=DESC interaction.
...
FossilOrigin-Name: 04907fbadeb743c95cc9f3529e63ef388684799f
2011-06-14 09:00:27 +00:00
c72f9d98cb
Merge recent trunk changes into fts3-prefix-search branch.
...
FossilOrigin-Name: 135ce30f62ebd6a1b239c18dbbd9c926ea507db4
2011-06-14 07:22:30 +00:00
806cbfc214
Remove unused parameters from internal fts3 function.
...
FossilOrigin-Name: 06de3f2cbc27cdfd9f83218c9ea576f74f60d07b
2011-06-14 07:14:43 +00:00
2f77934385
Add tests for deferred tokens to fts3auto.test. Fix a problem with OR queries and deferred tokens.
...
FossilOrigin-Name: b9fb69e55bb05a8819688ee63987f2a05c33d59b
2011-06-13 17:00:12 +00:00
99ebad90e3
Fix a bug exposed by combining matchinfo(), NEAR and "ORDER BY rowid DESC".
...
FossilOrigin-Name: 5f6b87f420f21749aa7c72e020c50aca74890086
2011-06-13 09:11:01 +00:00
786b068967
Fix problems to do with using both OR and NEAR operators in a single expression.
...
FossilOrigin-Name: 4e8dd19eef04777d800977faf1859a405e396f30
2011-06-09 10:48:02 +00:00
abf2545ed9
Fix various issues to do with deferred tokens, NEAR expressions and matchinfo().
...
FossilOrigin-Name: 3972a787df5ec253b99b148385655e7b68d851fa
2011-06-08 18:39:07 +00:00
3eabcf5f46
Have NEAR queries use incremental merging. Fix issues surrounding the deferred token optimization.
...
FossilOrigin-Name: 9d10a6846b12a9cc8fd4fdc3affd931a27218b5a
2011-06-07 18:35:45 +00:00
ebcdc73f53
Merge the latest trunk changes into the fts3-prefix-search branch.
...
FossilOrigin-Name: 567dd84359218245d4e6887547e2a48881f2c8e0
2011-06-06 18:14:25 +00:00
5289b01524
Modify fts3rnd.test to run tests for both "ORDER BY docid ASC" and "ORDER BY docid DESC" with both order=ASC and order=DESC FTS tables. Fixes for some bugs found.
...
FossilOrigin-Name: 89f2f482e077241ac29a58eadf44a72a9c01f98c
2011-06-06 14:51:50 +00:00
9aab071780
Clean up the code for processing FTS4 options a bit.
...
FossilOrigin-Name: 0425138a2365d23b07d88fda2b1f458f112f389d
2011-06-06 06:55:38 +00:00
76e04431ef
Remove some unreachable code.
...
FossilOrigin-Name: 650e1a79eda5a2134a1fbd305ab1f205a57c0892
2011-06-04 20:13:24 +00:00
b46ee91729
Allow the "order=DESC" and "order=ASC" parameters in FTS4 "CREATE VIRTUAL TABLE" statements. Tables created with "order=DESC" store all doclists in descending order, which allows optimizations normally applied to "ORDER BY docid ASC" queries to be used with "ORDER BY docid DESC" queries instead.
...
FossilOrigin-Name: f6a0193f5a32603eb48bddc6297042dbd2ffe96e
2011-06-04 20:04:35 +00:00
126ba6c0ac
FTS changes: Remove unreachable code. Fix bugs. When processing a large doclist incrementally, read from disk incrementally too.
...
FossilOrigin-Name: a4c7e2820824e82580730c36f85aede2efa66754
2011-06-03 18:00:19 +00:00
e414854800
Changes to improve performance and support LIMIT clauses on fts3 tables. This branch is unstable for now.
...
FossilOrigin-Name: 28149a7882a1e9dfe4a75ec5b91d176ebe6284e9
2011-06-02 19:57:24 +00:00
382874fc5c
Minor changes made while planning a larger change.
...
FossilOrigin-Name: 84097a4c759b1d65890af885f137d3cb16eef584
2011-05-28 15:57:40 +00:00
7ee4fdd7ba
Fix some minor and harmless compiler warnings.
...
FossilOrigin-Name: a4755e7088c3cc7c5ea191ce37f3950472f523ec
2011-05-25 23:18:02 +00:00
98b08e71f2
If a prefix index of size N is not present, use a prefix index of size N+1 along with the terms index for queries for prefixes of length N.
...
FossilOrigin-Name: cc83991caae7c7d647432d5711b6cd80228c3002
2011-05-25 19:17:32 +00:00
73ae6e5e76
Merge trunk changes into experimental fts3-prefix-search branch.
...
FossilOrigin-Name: f0f0a03db214b68a37069f64c27ae8520220c900
2011-05-25 18:47:26 +00:00
9d1f874ab3
Change fts4 so that the prefix= parameter is passes a comma-separated list of integers. For each integer N, a separate index of all prefixes of length N bytes is created.
...
FossilOrigin-Name: be59bf49402d2e2f4b95fb6668849f3745cb7bf2
2011-05-25 18:34:53 +00:00
346506f97b
Changes to savepoint in virtual tables for simpler and more consistent
...
operation.
FossilOrigin-Name: 92f26a8b8f18832cb1f8ba7dd8e5b020e71a5883
2011-05-25 01:16:42 +00:00
09643ab745
If the fts4 option prefix=1 is specified, have the fts4 module maintain an index of prefixes as well as terms.
...
FossilOrigin-Name: b5bdc639898ee22eebedeb560810e94e74de8aa4
2011-05-24 18:49:45 +00:00
e485522462
Do not invoke the xRollbackTo or xRelease methods of a virtual table without
...
having first invoked an appropriate xSavepoint method. Add assert() statements
to FTS3/4 to verify that this is happening in all cases.
FossilOrigin-Name: 651ef24249d8c22c4f13e4c0bb98a60099cfd23a
2011-05-24 15:36:01 +00:00
133d7dab17
Add extended return code SQLITE_CORRUPT_VTAB. Returned when the tcontents of the sqlite tables used internally by a virtual table module are invalid or inconsistent.
...
FossilOrigin-Name: 8844e8bfb87314fb40ecb92705e8fff88f72bb38
2011-05-17 15:56:16 +00:00
807d0fcaba
Avoid including fts3_term.c in the amalgamation, as it contains test code only.
...
FossilOrigin-Name: f392b7ae0266b4c694836583cb91b10f2b6c0752
2011-05-17 14:41:36 +00:00
e578b5967c
Increase the iVersion for sqlite3_module needed for the xSavepoint,
...
xRelease, and xRollbackTo methods from 1 to 2. This is so that it will
conform to published documentation.
FossilOrigin-Name: 0f4954e9793332573d4ffa0c1e199180f4cb990a
2011-05-06 00:19:57 +00:00
7b0ced74c0
Merge vtab-conflict branch with trunk.
...
FossilOrigin-Name: 8f9666af5f9459cbd0305a86281d745ea6163acf
2011-05-04 17:23:19 +00:00
62d6c7e32b
Fix a couple of compiler warnings in the FTS code.
...
FossilOrigin-Name: 1a113359705d307efa1833b1dfa5542b045dae43
2011-05-04 16:30:05 +00:00
e1c8e13bf6
Fix a performance problem in queries that use "ORDER BY rowid DESC" and one or more FTS auxiliary functions.
...
FossilOrigin-Name: 95e09b20e9aad28f829c8950f3632debe473070a
2011-05-04 15:41:18 +00:00
0f599faa29
Optimize "ORDER BY rowid/docid DESC/ASC" clauses on FTS tables.
...
FossilOrigin-Name: 13395121e3d17ab6581dc5f6736ea324321a374c
2011-05-04 12:52:59 +00:00
3480a01dad
Add documentation for the newly introduced sqlite3_vtab_config() and on_conflict() API functions. Test that encountering an SQLITE_MISMATCH in fts3 does not corrupt the full text index.
...
FossilOrigin-Name: abdd70ae0424ccadb7edaf16e970c78b5257d23c
2011-04-27 16:02:46 +00:00
a311b80392
Extra tests for fts3. And fixes for conflict-handling related problems in fts3.
...
FossilOrigin-Name: fb4a355871d9482ccb28b6ba03b842b3cc87b696
2011-04-26 19:21:34 +00:00
b061d058cb
Add support for on conflict clauses to fts3/fts4.
...
FossilOrigin-Name: 6d2633a6d0a9bb88fb1a6adac0827dc51df2d4d2
2011-04-25 18:49:57 +00:00
355c53b6ef
Avoid passing NULL to the xOpen method of an FTS3/4 tokenizer.
...
FossilOrigin-Name: 0dd09fc034c127718366d3a3183e367d2f9fd82d
2011-04-19 06:43:22 +00:00
fcd71b6010
Suppress many harmless compiler warnings, mostly signed/unsigned comparisons
...
within asserts or unused parameters in extensions.
FossilOrigin-Name: 3eeb0ff78d04891b5fd1a3d99a9fb8cfbed77a81
2011-04-05 22:08:24 +00:00
0370493760
Fix a bug in fts4 to do with matchinfo and deferred tokens.
...
FossilOrigin-Name: 30d42dc66f19ef5cc5b27d6273eadf56619a3ce8
2011-03-23 17:10:43 +00:00
2fe88b7ca4
Ensure fts4aux can handle a table name in single or double quotes as a constructor argument.
...
FossilOrigin-Name: 929d62e496bb36a3ee0e19ec4609329d79aaeddc
2011-02-05 15:47:12 +00:00
bb7e77c758
Add test cases for fts4aux. Fix a bug affecting fts3 tables with multiple columns.
...
FossilOrigin-Name: dc511e60a65232a7087e12ff40b63506cf37a634
2011-02-05 14:37:57 +00:00
4c600ac389
Change fts4aux to (additionally) report on term frequency in individual columns of an fts table.
...
FossilOrigin-Name: 3996f92a9aca9ac2628be003eca83c8f954c71de
2011-02-04 18:56:25 +00:00
767f9a8f9e
Extra tests for fts4 compress/uncompress hooks. Fix some minor problems with the same.
...
FossilOrigin-Name: 80225abe79b0a7723b922ec129954438af95855a
2011-02-03 12:48:17 +00:00
16690ed167
Extra tests for the fts4aux module.
...
FossilOrigin-Name: cfc475690d85ea7e3547424289d9837f46ab7ef4
2011-02-03 10:56:00 +00:00
20d5f354e9
Optimize handling of equality and range constraints on the "term" column of an fts4aux table.
...
FossilOrigin-Name: 386701ded2bcee5309ec9f285d94e6eb1eade193
2011-02-02 17:30:43 +00:00
ee85a51092
Add missing file fts3_aux.c.
...
FossilOrigin-Name: 0147d9739f318a02721d3ae0e7be339a19231f60
2011-02-02 04:26:46 +00:00
0bfda98155
Fix a case in fts4 where a corrupt %_stat table could lead to a crash.
...
FossilOrigin-Name: 4ade96ce974244fc34bb97713d3cba10e3d33056
2011-02-01 17:55:48 +00:00
dan