assumed that the row had values in all columns, sigh. Fixes bug
http://www.sqlite.org/cvstrac/tktview?tn=2289 . (CVS 3833)
FossilOrigin-Name: 81be7290a4db7b74a533aaf95c7389eb4bde6a88
updates happen within a single transaction, there was a lot of wasted
encode/decode overhead due to segment merges. This code buffers
updates in memory and writes out larger level-0 segments. It only
works when documents are presented in ascending order by docid.
Comparing a test set running 100 documents per transaction, the total
runtime is cut almost in half. (CVS 3751)
FossilOrigin-Name: 0229cba69698ab4b44f8583ef50a87c49422f8ec
assertions when this occurs, and it's almost certainly not the right
thing to do in the first place. (CVS 3746)
FossilOrigin-Name: f6c3abdc6c5e916e5366ba28fb1cd06ca3554303
Collector) now handles the case where PLWriter (Position List Writer)
needed a local buffer. Change to using the associated DLWriter
(Document List Writer) buffer, which reduces the number of memory
copies needed in doclist processing, and brings PLWriter operation in
line with DLWriter operation. (CVS 3707)
FossilOrigin-Name: d04fa3a13a84f49074c673b8ee2fb6541da061b5
Currently, PLWriter (Position List Writer) creates a locally-owned
DataBuffer to write into. This is necessary to support doclist
collection during tokenization, where there is no obvious buffer to
write output to, but is not necessary for the other users of PLWriter.
This change adds a DLCollector (Doc List Collector) structure to
handle the tokenization case.
Also fix a potential memory leak in writeZeroSegment(). In case of
error from leafWriterStep(), the DataBuffer dl was being leaked. (CVS 3706)
FossilOrigin-Name: 1b9918e20767aebc9c1e7523027139e5fbc12688
malloc/calloc/realloc appropriately, and use sizeof(var) instead of
sizeof(type) to make certain that we don't get a mismatch between
them as the code rots. (CVS 3693)
FossilOrigin-Name: fbc53da8c645935c74e49af2ab2cf447dc72ba4e
When creating fts tables in an attached database, the backing tables
are created in database 'main'. This change propagates the
appropriate database name to the routines which build sql statements.
Note that I propagate the database name and table name separately. I
briefly considered just making the table name be "db.table", but it
didn't fit so well in the model used to store the table name and other
information, and having the db name passed separately seemed a bit
more transparent. (CVS 3631)
FossilOrigin-Name: 283385d20724f0144f38de89bd179715ee5e738b
Calling UPDATE against an fts table in a UTF-16 database inserts
corrupted data into the database. The UTF-8 data is being inserted
directly. This appears to happen because sqlite3_ value_text()
destructively coerces a value to UTF-8, and it's never converted back
when updating the table. This works around the problem by rearranging
things so that the update happens before the coercion. (CVS 3596)
FossilOrigin-Name: 4f2ab4b6320ffc621900049b41f50bc30d76d7f5
The virtual table interface allows for a cursor to field multiple
xFilter() calls. For instance, if a join is done with a virtual
table, there could be a call for each row which potentially matches.
Unfortunately, fulltextFilter() assumes that it has a fresh cursor,
and overwrites a prepared statement and a malloc'ed pointer, resulting
in unfinalized statements and a memory leak.
This change hacks the code to manually clean up offending items in
fulltextFilter(), emphasis on "hacks", since it's a fragile fix
insofar as future additions to fulltext_cursor could continue to have
the problem. (CVS 3521)
FossilOrigin-Name: 18142fdb6d1f5bfdbb1155274502b9a602885fcb
that this is of marginal utility when encoding terms resulting from
regular English text, it turns out to be very useful when encoding
inputs with very large terms. (CVS 3520)
FossilOrigin-Name: c8151a998ec2423b417566823dc9957c7d5d782c
between leaf nodes, instead of storing the entire leftmost term of the
rightmost child, store only that portion of the leftmost term
necessary to distinguish it from the rightmost term of the leftmost
child. (CVS 3513)
FossilOrigin-Name: f6e0b080dcfaf554b2c05df5e7d4db69d012fba3
LeafWriter to use empty data buffer (instead of empty term) to detect
an empty block. Code to validate interior nodes. Moderate revisions
to leaf-node and doclist validation. Recast leafWriterStep() in terms
of LeafWriterStepMerge(). (CVS 3512)
FossilOrigin-Name: f30771d5c7ef2b502af95d81a18796b75271ada4
where excessively large terms keep the tree from finding a single
root. A downside is that this could result in large interior nodes in
the presence of large terms, which may be prone to fragmentation,
though if the nodes were smaller that would translate into more levels
in the tree, which would also have that problem. (CVS 3510)
FossilOrigin-Name: 64b7e3406134ac4891113b9bb432ad97504268bb
( http://www.sqlite.org/cvstrac/chngview?cn=3486 ) broke test fts2a-5.3.
This change should make the expected result more obvious. (CVS 3489)
FossilOrigin-Name: cde383eb467de0d752e94a22cd2f890c2dc599cc
http://www.sqlite.org/cvstrac/tktview?tn=2036,35 describes some cases
where we were passing memset() a length which was the sizeof a
pointer, rather than the structure pointed to. Instead, wrap this
idiom up in CLEAR() and SCRAMBLE() macros. (CVS 3488)
FossilOrigin-Name: 5878add0839f9c5bec77caae2361ec20cb60b48b
distinguish reading from a static buffer from writing to a dynamic
buffer. This allows n-way doclist merging, and in-place merging of
segment leaf nodes, which together cut segment merge times in half. (CVS 3486)
FossilOrigin-Name: af5bfb986e39248abbfc6fff2e13c6f9e634a751
was writing out a segment made up of a single leaf node containing the
\0 header. LeafReader assumed that leaf nodes always contained at
least one term, so assertions would fail.
While it would be possible to support reading and merging empty
segments, there's no reason to do so. While this change could have
been done in writeZeroSegment(), I put it in leafWriterFlush() so that
it would work right if segmentMerge() created an empty segment, which
could happen with future changes to how deleted documents are handled. (CVS 3484)
FossilOrigin-Name: fed79beec7da24a26ae94494bdc0c98dd102bc06
updates. Groups of documents form segments which are encoded in a
btree layered over a table of blocks, with various tricks to make
merges fast. This performs 20x-25x faster than fts1 when loading the
Enron corpus, and is only slightly slower for queries. (CVS 3474)
FossilOrigin-Name: 85272b2f5394e37916afb1d509e7296810d976f5
docListRestrictColumn() generates a DL_POSITIONS doclist, which means
that after the first doclist is processed, the second doclist is
initialized as DL_POSITIONS, but with DL_POSITIONS_OFFSETS data.
(Note that DL_DEFAULT is now DL_POSITIONS, which masks this bug.) (CVS 3467)
FossilOrigin-Name: 144e3f11e22c6efd6f2d960599ab2d93542db406
We handle an UPDATE to a row by performing an UPDATE on the content table and by building new position lists for each term which appears in either the old or new versions of the row. We write these position lists all at once; this is presumably more efficient than a delete followed by an insert (which would first write empty position lists, then new position lists). (CVS 3434)
FossilOrigin-Name: 757fa22400b363212b4d5f648bdc9fcbd9a7f152
method of a virtual table. In FTS1, use strcmp instead of strcasecmp.
Ticket #1981. (CVS 3428)
FossilOrigin-Name: efa8fb32a596c7232bb1754b3231e4f2421df75b
table. Offsets are retrieved using a special "offsets" function whose
first argument is the magic column. Snippets will ultimately be retrieved
in the same way. (CVS 3427)
FossilOrigin-Name: 5e35dc1ffadfe7fa47673d052501ee79903eead9
a string containing byte offset information for all matching terms.
Also added a large test case based on SQLite mailing list entries. (CVS 3417)
FossilOrigin-Name: f25cfa1aec0e4c1fe07176039a1b7f4e6a2c66ec
names in the spec that are SQL keywords or have special characters, etc.
Also added support for additional control lines. Column names can be
followed by a type specifier (which is ignored.) (CVS 3410)
FossilOrigin-Name: adb780e0dc8bc7dcd1102efbfa4bc17eefdf968e
