Commit Graph

461 Commits

Author SHA1 Message Date
dan
49377a8fb7 Fix a problem with the fts5 auxiliary function API causing a crash when a query contained a phrase of zero terms.
FossilOrigin-Name: 0741812d7fcd558479e4849fbb3ba8d03738d018
2016-09-21 19:43:34 +00:00
dan
48a72ebfa3 Update the fts5vocab table to handle "ORDER BY term" efficiently.
FossilOrigin-Name: d4928fb5cd63a72808f01778fa5a11395385dedf
2016-09-21 14:41:09 +00:00
dan
37ff4d4c77 Fix a bug in the fts5 snippet function causing it to return text containing zero phrase instances.
FossilOrigin-Name: b174ed2bde43a1872374ce18b85ac9f067e86fbb
2016-08-24 18:50:47 +00:00
dan
783e778f38 Fix other minor problems with the fts5 snippet() function.
FossilOrigin-Name: b56b2a85cf377686727f3929208ede43f797cf76
2016-08-20 17:23:45 +00:00
dan
7ca1ea16ff Have the fts5 snippet() function avoid favouring snippets that begin with sentences that do not contain search terms. Add an extra bias in favour of the first sentence in the document.
FossilOrigin-Name: d8180af2adfc101dfce95a0498b5cd7b30643b30
2016-08-19 14:25:38 +00:00
dan
b39a5ac229 Adjust some tests to account for recent changes to the fts5 snippet function.
FossilOrigin-Name: 184ecbe9c0c3280a22cdeda5c7836a7b280c3e1f
2016-08-18 14:47:48 +00:00
dan
e7036202bc Bias the fts5 snippet() function to return snippets that look like they start at the start of sentences.
FossilOrigin-Name: 60de159476edbd48dc363f7f77f09c32ea68422f
2016-08-17 19:05:12 +00:00
dan
8c2b1fde58 Fix a problem in the fts5 snippet() auxiliary function.
FossilOrigin-Name: a861713cc6a3868a1c89240e8340bc7b2b9559da
2016-08-17 11:14:39 +00:00
dan
460d38f15e Fix an fts5 problem with corrupt database handling found by address-sanitizer.
FossilOrigin-Name: e22252e1da4cd9e41b970970a1c4f466aa6cc133
2016-08-13 10:34:12 +00:00
dan
3b96d1e2e4 Fix a buffer overread in fts5.
FossilOrigin-Name: fcfbee6c7d33a9ae7feb46044a0c2fe680460d39
2016-08-13 06:38:31 +00:00
drh
c73bd0a5cc Fix a harmless compiler warning in FTS5.
FossilOrigin-Name: 2ea0c8b46eefd4874f021f8dfd344be751f2034f
2016-08-11 01:24:56 +00:00
drh
8d57d7af23 Fix harmless compiler warning.
FossilOrigin-Name: 9a5a4f6e3bc265fecf79a7f63d14abbf239da636
2016-08-09 21:01:52 +00:00
dan
ccf03677a3 Minor update to the way fts5 column filters are parsed.
FossilOrigin-Name: 14864f2b8470fe98dbd17f59963bf1be8d4962f9
2016-08-09 19:48:37 +00:00
dan
882ef0b8c0 Have fts5 interpret column lists that begin with a "-" character as "match any column except" lists.
FossilOrigin-Name: e517545650631d1e8a7ee63c6646a8b183a0a894
2016-08-09 19:26:57 +00:00
drh
6da466e4f4 Fix documentation typos. No changes to code.
FossilOrigin-Name: ebc396a19fa79bea208ecda277ffff5d02166d0b
2016-08-07 18:52:11 +00:00
mistachkin
c08556844e Add macro necessary for building several extensions when the __stdcall calling convention is not in use.
FossilOrigin-Name: bc8c9b55ddf5227dc6b2116486f4a19bf54fce2b
2016-07-29 01:31:49 +00:00
mistachkin
2d45d7bfab Missed a couple Tcl command deletion callbacks.
FossilOrigin-Name: fc5c187602acceedf7152b6927616d134f779a8f
2016-07-28 23:58:25 +00:00
mistachkin
afe18262b4 Use the SQLITE_TCLAPI macro in several extensions that were missed in the previous check-in.
FossilOrigin-Name: df2543652b835353c03956dfbd3e35e5b191372e
2016-07-28 23:54:45 +00:00
mistachkin
52b1dbb5fc Allow the 'testfixture.exe' target to be compiled with the __stdcall calling convention.
FossilOrigin-Name: e8be3dfeabaa31b3490793cf8230faae1204be15
2016-07-28 14:37:04 +00:00
drh
07d9a03187 Fix a test case in FTS5 to make it compatible with Tcl 8.5.
FossilOrigin-Name: a495f8e77e14241150f79e746c7303284f04621a
2016-07-25 21:11:53 +00:00
dan
033ac6af7e Add extra test cases to verify the fix in [64ca1a835].
FossilOrigin-Name: bf98a2de7ed41a8acca89d7369b353f12182cfb4
2016-07-21 18:02:20 +00:00
dan
8414e81359 Add the largely untested SQLITE_FTS5_NO_WITHOUT_ROWID compile time option to fts5. For building a dynamically loadable extension that does not use WITHOUT ROWID.
FossilOrigin-Name: d0a1cf1c56c237617fb73cb31f4950365b1f3e9b
2016-07-15 19:17:19 +00:00
dan
0df5024e83 Fix an FTS5 problem (segfault or incorrect query results) with "... MATCH 'x OR y' ORDER BY rank" queries when either token 'x' or 'y' is completely absent from the dataset.
FossilOrigin-Name: 64ca1a835a89fd211078d2cd8f9b649e89be528d
2016-05-30 08:28:21 +00:00
dan
ba1079ba2a Update a comment in fts5.h. No changes to code.
FossilOrigin-Name: 14e53d0e2f62d82ae1d64a72fd9711548e3bf5ea
2016-05-09 19:03:42 +00:00
dan
170b66493d Fix the FTS5 xQueryPhrase() API function so that it recognizes column filters.
FossilOrigin-Name: 48505cde05d74748454dc135d4ec25954ecf020e
2016-05-09 18:05:44 +00:00
mistachkin
77fac879d2 More harmless compiler warning fixes.
FossilOrigin-Name: ab69527c1608da0b668f3b49e967661dd99cc3d4
2016-04-12 20:05:06 +00:00
drh
997de998a8 Remove an unused local variable from FTS5.
FossilOrigin-Name: 0ed693c29f184223cde3b3d51f0e06273e586803
2016-03-23 15:53:45 +00:00
dan
848b190e40 Explicitly limit the size of fts5 tokens to 32768 bytes.
FossilOrigin-Name: 70fc69eed9b09159899d7cbd1416a59d04210a63
2016-03-23 15:04:00 +00:00
dan
0a2f2b546a Fix some errors in fts5 test scripts.
FossilOrigin-Name: e1ab2d376a72786098125a41c1ea8140fcbd15c6
2016-03-22 15:01:54 +00:00
drh
13969f5af0 Fix harmless compiler warnings.
FossilOrigin-Name: 25d776e4523aefeec007943fe29aa17c23ccb301
2016-03-21 22:28:51 +00:00
dan
6737aab5cf Add tests to ensure that fts5 is now deterministic.
FossilOrigin-Name: 3d076094ce411bffcfeb76c0284d15d0f83bd459
2016-03-21 16:25:34 +00:00
dan
f5268268b7 Have fts5 cache the structure of its index in main memory. Use "PRAGMA data_version" to figure out when this cache should be invalidated.
FossilOrigin-Name: 902f221754f3cc6fb4ae049c135f85efce604ed3
2016-03-21 15:30:50 +00:00
dan
3c6707f72b Rearrange code so that tests pass whether SQLITE_DEBUG is defined or not.
FossilOrigin-Name: 89296a46c3b891ee0b637ff6f89d828d6e46e7b5
2016-03-21 15:18:13 +00:00
dan
ef480d37dc Change the way fts5 internally allocates segment ids in order to eliminated non-determinism from the module.
FossilOrigin-Name: d6e2637df16764aa9723a30ea2eb8a631d28cb2b
2016-03-21 09:56:19 +00:00
dan
cf9bce34ab Add test file fts5multiclient.test.
FossilOrigin-Name: 7832466f9177cca59455ba272802145297fcd53d
2016-03-19 14:47:28 +00:00
drh
860443da10 Fix FTS5 so that it works with SQLITE_OMIT_AUTORESET.
FossilOrigin-Name: b199637d81d7e2a767131ac03c7679b101fd459c
2016-03-18 00:39:40 +00:00
dan
c9460ab989 Fix some OOM-handling issues in the fts5 changes on this branch.
FossilOrigin-Name: 020a0bda59fe93b9361ceeed0d5a8ba4177380c8
2016-03-17 12:39:56 +00:00
dan
db8d320d63 Merge the changes on the reusable-pragma branch into this one.
FossilOrigin-Name: 6c4a17b963916d19c9433d7a251338042f32e9fa
2016-03-16 20:53:33 +00:00
dan
377ba38429 Add a cast to an implict (size_t -> int) conversion in fts5_expr.c.
FossilOrigin-Name: d9b5ff7aba3a8f6ca4505eac308fa9d510254c3e
2016-03-16 19:53:58 +00:00
dan
f55fb6615b Have fts5 cache the decoded structure of fts5 indexes in memory. Use "PRAGMA data_version" to detect stale caches.
FossilOrigin-Name: 33ef2210ef19e55c8d460bfe9d3dc146034c8acc
2016-03-16 19:48:10 +00:00
dan
7d85774da8 Fix a memory leak in fts5 that could occur following a syntax error in a query expression.
FossilOrigin-Name: 54b8968e335760105206b894672213c1d56771a4
2016-03-14 12:16:23 +00:00
dan
5cf98f101e Fix a problem handling 'NEAR("" token)' in fts5 found by fuzzing.
FossilOrigin-Name: 10a827ae5f3f322af836c15e581fdc958a433a5a
2016-03-12 19:33:47 +00:00
dan
e8c20120ce Fix handling of strings that contain zero tokens in fts5. And other problems found by fuzzing.
FossilOrigin-Name: 72b3ff0f0df83e62adda6584b4281cf086d45e45
2016-03-12 16:32:16 +00:00
dan
1bb85f3ed3 Add tests to cover untested branches in recent fts5 changes.
FossilOrigin-Name: fa0033edf6ddf3c6631fa95b343225dbc8ac9429
2016-03-10 15:12:47 +00:00
drh
6459ca0b8c Remove an unused local variable.
FossilOrigin-Name: 3c343c3d012367942037e64a9855b825cb459844
2016-03-10 14:22:42 +00:00
dan
4dbc65b29a Add an incremental optimize capability to fts5. Make the 'merge' command independent of the 'automerge' settings.
FossilOrigin-Name: 556671444c03e3afca072d0f5e9bea2657de6fd3
2016-03-09 20:54:14 +00:00
drh
5b38201708 Merge 3.11.1 changes into trunk.
FossilOrigin-Name: 7abc2dd953477797fa05421b061d73596aa025fb
2016-03-03 17:54:30 +00:00
dan
e18ec6acc2 Fix an error in an fts5 test script.
FossilOrigin-Name: b2a03e2d479aec2fc68dfc83a1fa34c740f44e13
2016-03-02 13:26:19 +00:00
dan
bce50c734b Do not attempt to run fts5bigtok.test or fts5merge2.test if FTS5 is not compiled in.
FossilOrigin-Name: f7ed373953c2dea0919a85dfb461dfef512cd5fc
2016-03-01 16:56:26 +00:00
drh
fb02bad04f Remove a forgotten "breakpoint" command from an FTS5 test script.
FossilOrigin-Name: 61927c8f3156c6e61b768318b3cd3bcc4144e17b
2016-03-01 15:52:41 +00:00
dan
b1e8d31edf Fix a problem in fts5 where a corrupt db could lead to a (huge) buffer overread. Cherrypick of [c9a30e117f].
FossilOrigin-Name: daef5869f4d62ebb24eb03b79fe4be0812fa0496
2016-03-01 14:51:36 +00:00
dan
45a0e4eb59 Fix an fts5 problem causing 'optimize' to corrupt the fts index under some circumstances. Cherrypick of [251d6473f7].
FossilOrigin-Name: 5b1b7ab5d67f3d691e52ba57dbc14c52d90c6631
2016-03-01 14:50:55 +00:00
dan
4e9d0d5418 Fix a problem in fts5 where a corrupt db could lead to a (huge) buffer overread.
FossilOrigin-Name: c9a30e117f2c6c9ef0cc0c6ca5227d2961715b8f
2016-02-29 17:34:16 +00:00
dan
437a54eaba Fix an fts5 problem causing 'optimize' to corrupt the fts index under some circumstances.
FossilOrigin-Name: 251d6473f7c9ad95adbdcc49cb8eaf7c0956764b
2016-02-29 17:16:26 +00:00
dan
9e2cfad18a Add an "#ifndef SQLITE_AMALGAMATION" guard around a typedef in fts5_test_mi.c to avoid redefining "u32" if it is compiled as part of the amalgamation. Cherrypick of [f9c49ba1].
FossilOrigin-Name: ce0f1ed517e387ca6a687b805088d7c529cd257b
2016-02-19 10:19:43 +00:00
dan
92b859292d Add an "#ifndef SQLITE_AMALGAMATION" guard around a typedef in fts5_test_mi.c to avoid redefining "u32" if it is compiled as part of the amalgamation.
FossilOrigin-Name: f9c49ba1edf13cb8c48d73f1b2e3f80c46b3c030
2016-02-19 10:18:36 +00:00
dan
557d0383ff Remove SQLITE_TEST macros from fts5_test_mi.c. Add the SQLITE_FTS5_ENABLE_TEST_MI compile time symbol - to make it easier to create builds that include the fts5 matchinfo demo by default.
FossilOrigin-Name: e2ef6667772eeac6eee3aed5d0609c2a0ef5d8cf
2016-02-18 16:16:43 +00:00
dan
a0c2d3c155 Remove an unnecessary "include <tcl.h>" line from fts5_test_mi.c.
FossilOrigin-Name: 31d8b69e9e0747e573516570bfe2770384e99134
2016-02-18 13:32:34 +00:00
dan
5e29f38d2c Remove the "#ifdef SQLITE_TEST" condition from fts5_test_mi.c. Update fts5_main.c so that if SQLITE_FTS5_TEST_MI is defined when it is built the fts5_test_mi.c entry point is automatically called. This makes it easier to build versions of fts5 that include the matchinfo() demo by default.
FossilOrigin-Name: 87e6e225aea3a4d10476d16eab4a6403a4a128c9
2016-02-17 14:48:26 +00:00
dan
87d6b35541 Improve test coverage of demo code in fts5_test_mi.c.
FossilOrigin-Name: 8a78050dc0e374a0df0f29716df60250e2c4b4f7
2016-02-17 14:11:06 +00:00
drh
45f31be85d Experimental changes to Lemon for improved parser performance.
FossilOrigin-Name: a65d583ce97b8c08157268bd054479cda3957a94
2016-02-16 21:19:49 +00:00
drh
7ff89807e0 Fix a harmless compiler warning in FTS5.
FossilOrigin-Name: e52c90758c53cbb0f74e0d8582e4c3c1bee04150
2016-02-15 23:46:36 +00:00
dan
53ff9c2972 Fix a potential buffer overread provoked by invalid utf-8 in fts5.
FossilOrigin-Name: a049fbbde5da2e43d41aa8c2b41f9eb21507ac76
2016-02-12 18:48:09 +00:00
dan
f2d328fa25 Fix a fairly obscure buffer overread in fts5.
FossilOrigin-Name: 130580207ab5cee762b2893808acef7c8afad027
2016-02-12 17:56:27 +00:00
drh
5d2e2f1267 Remove the unused fts5BlobCompare() routine.
FossilOrigin-Name: defc762dd11144402abd591278819768a080d5ea
2016-02-11 21:53:57 +00:00
dan
df9c59f74d Fix another unused variable warning in fts5 code.
FossilOrigin-Name: 61b4c120540afd80a4021b0d3990f13acc38a059
2016-02-11 18:18:33 +00:00
dan
2a44b3d877 Avoid a buffer overread when reading a corrupt fts5 structure record.
FossilOrigin-Name: facbc424e555061135aced7b134bf6c19f54e484
2016-02-11 18:08:38 +00:00
dan
22e8356368 Handle parser stack overflow when parsing fts5 query expressions. Fix some compiler warnings in fts5 code.
FossilOrigin-Name: bc3f7900d5a06829d123814a5ac7b951bcfc1560
2016-02-11 17:01:32 +00:00
drh
df3a907ecc Add JSON1 and FTS5 to the set of extensions subject to close compiler warning
analysis.  Fix some warnings in each.   More (harmless) warnings still exist
in FTS5.

FossilOrigin-Name: cfe2eb88b504f5e9b1351022036641b1ac4c3e78
2016-02-11 15:37:18 +00:00
dan
c2d0b56e6a Fix a signed/unsigned comparison warning in fts5.
FossilOrigin-Name: 85eb05ddbc3a7cff986af13c3dab6c0063e62171
2016-02-08 20:14:41 +00:00
dan
d120a1e3ab Add tests for and remove unreachable branches from fts5 in order to restore test coverage.
FossilOrigin-Name: 22589018ac3321f7bd89ce9fa69e57eae395e324
2016-02-05 21:09:26 +00:00
dan
b80b8f482c Further streamlining of fts5 prefix query code.
FossilOrigin-Name: ca91bd8ac70a5b3fef127364f73ec675e58bb92c
2016-02-05 19:18:02 +00:00
dan
4786cf5a20 Further improve performance of unindexed fts5 prefix queries.
FossilOrigin-Name: c9c6457d8ea911f6cc63967127e58da3146fd3ef
2016-02-04 19:45:19 +00:00
dan
eb715c62f9 Improve performance of fts5 prefix queries on detail=col tables.
FossilOrigin-Name: ca11f46db047e7f131cef3893f73824758a2076b
2016-02-03 20:04:59 +00:00
dan
b06347a5dc Add tests to restore full coverage of fts5 code.
FossilOrigin-Name: 063755c81574800e7db12a42e17d982a8c1e5181
2016-02-02 21:19:21 +00:00
dan
735ff4a809 Enhance the performance of fts5 AND and OR queries.
FossilOrigin-Name: 62ea9e5ab8bc1a20245beebceb5ea62dcd7ec84e
2016-02-02 17:40:41 +00:00
dan
df015945e3 Improve performance of fts5 queries.
FossilOrigin-Name: 2334e88244afe6387208be5d527aba9b5ddf4e4c
2016-02-01 20:12:41 +00:00
dan
52f5d1ba6b Performance enhancement for fts5 column filter queries on detail=full tables.
FossilOrigin-Name: 13fb4aa6a87c5c6258979953da82eedc1a7bf14f
2016-01-30 19:16:11 +00:00
dan
18205c781d Further minor performance improvements and code-size reductions related to fts5 column filters on detail=col tables.
FossilOrigin-Name: b4ac61aeee976296e7719949cd4fb496147a29e8
2016-01-26 20:08:50 +00:00
dan
1c16bccb79 Improve the performance of fts5 column filters on detail=col tables.
FossilOrigin-Name: 249a2d070c34bf884a04cb248b9691e239f2871c
2016-01-26 19:30:49 +00:00
dan
fee981e5b9 Enhance fts5txt2db.tcl, a script used to generate fts5/fts4 databases for performance testing.
FossilOrigin-Name: c646e40350e5aa91abcf52de61fb31275bad38f9
2016-01-26 17:08:22 +00:00
dan
e9eb1593f5 Fix an fts5 problem with using both xPhraseFirst() and xPhraseFirstColumn() within a single statement in detail=col mode.
FossilOrigin-Name: 72d53699bf0dcdb9d2a22e229989d7435f061399
2016-01-23 18:51:59 +00:00
dan
e17b646fd5 Merge trunk changes (including fixes for warnings in fts5) with this branch.
FossilOrigin-Name: ceccc9ad788fb4da9120915741995b9f088f85ff
2016-01-23 16:20:16 +00:00
dan
c66d03f1e1 Fix some signed/unsigned comparison compiler warnings in fts5.
FossilOrigin-Name: 3be336aa893f9eb0837d7d66c83bab1489792b9a
2016-01-23 15:57:06 +00:00
dan
8498f387be Fix a problem involving detail=col and column filters.
FossilOrigin-Name: 7558a0ad2276e91f2faced8ea405d9fdb4fa0c6e
2016-01-23 14:45:36 +00:00
dan
795502d26c Experimental performance enhancements for fts5.
FossilOrigin-Name: b5a57b812fd6a734cf2a342bf0b730ae18912d73
2016-01-22 19:48:34 +00:00
dan
bd5af9ea31 Add a -detail option to the loadfts5.tcl script. For creating tables with the detail= option set.
FossilOrigin-Name: 81e830e21f81b6428209afa0b8f42f2fd9bd7aab
2016-01-20 16:36:10 +00:00
dan
4684ffd495 Add further fts5 tests. 100% code coverage is finally restored.
FossilOrigin-Name: b914ece0d146cfc8adba4dc4e8633e14f174d6ea
2016-01-19 16:06:23 +00:00
dan
6d3c2889ec Add tests for fts5. Fix a crash that can occur in fts5 if the database content is corrupted.
FossilOrigin-Name: acaf426449bf6fd3140fd63141750ff69d1119a5
2016-01-18 17:48:28 +00:00
dan
0536a07c0e Add a debugging function to print human-readable versions of fts5 detail=none leaf pages.
FossilOrigin-Name: 8358af3658d888516cdef5f8c8d89e9bdee53f91
2016-01-18 09:08:56 +00:00
dan
3ed7029921 Add extra tests to cover untested branches in fts5.
FossilOrigin-Name: 61deab043dcea860070dba6b02601a7de627fff1
2016-01-16 21:06:41 +00:00
dan
8631402e6a Add further tests for fts5. Fix some problems with detail=col mode and auxiliary functions.
FossilOrigin-Name: de77d6026e8035c505a704e7b8cfe5af6579d35f
2016-01-16 18:58:51 +00:00
dan
25247435be Rationalize some code in fts5_storage.c. Add tests to cover recently added branches in fts5.
FossilOrigin-Name: 3b72df405ac9b3a71144f45317d32e25f0084c4b
2016-01-15 19:54:47 +00:00
mistachkin
1e38927c12 Fix a couple C99-isms in FTS5 that cause compile errors on MSVC.
FossilOrigin-Name: 584848d4396d7755027fab377c97e40fbcc90d21
2016-01-15 05:39:55 +00:00
dan
2cfe049f40 Add a test to verify that the fts3 and fts5 "unicode61" tokenizers are byte for byte compatible.
FossilOrigin-Name: 8ec8314354edc9d6f7d1c8a6370fd984a552a52f
2016-01-14 20:05:59 +00:00
drh
0ea94db612 Yet another change to FTS5 trying to get it to merge successfully into sessions.
FossilOrigin-Name: 8dedff3b9ac3e6bf9c131fee19f7d26dc1ebd61f
2016-01-14 14:33:36 +00:00
drh
d625adb8db Fix more compiler warnings in FTS5.
FossilOrigin-Name: f791fc7009d68941c60d7bf953bab19d86112133
2016-01-14 14:29:57 +00:00
dan
17535393b9 Fix another compiler warning in fts5 code.
FossilOrigin-Name: d17bc2c92f4d086280e49a3cc72993be7fee2da7
2016-01-14 14:19:50 +00:00
dan
f705e9deab Fix compiler warnings in fts5.
FossilOrigin-Name: 5a343cc0336bba056df4449e6cd2e3fb9e75a105
2016-01-14 14:15:54 +00:00
dan
7f50d69359 Add documentation for new API functions to the comments in fts5.h.
FossilOrigin-Name: 14d296fa7b43b4999a1fd070cf030f83fe1b94c3
2016-01-13 18:25:36 +00:00