If we want to add or remove arguments from the callback functions, it requires
changing the callback interface all over the place. By letting the callback simply
expect a single struct argument, it would clean things up a bit.
ok christos
Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)
The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.
Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.
The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).
To enforce using the new tokenizer, a schema version bump is needed
Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.
This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>
Thanks to christos@ for multiple rounds of reviews of the tokenizer code.
Earlier, a white space separated string was generated containing all the section
numbers passed through command line arguments. Later on that would have to be
tokenized and processed. Instead of that, use a NULL terminated array of strings.
Thanks to christos@ for reviewing and suggesting further improvements.
- Don't ever page unless asked for with -p
- Introduce "legacy mode" (-l)
1. searches only name and name_desc, prints name(section) - name_description
2. turns off escape formatting (can be forced on with -i)
3. turns off context printing (can be forced on with -c)
- Parse the environment $APROPOS variable as an argument vector.
With these changes one can simply 'export APROPOS=-l' and get the old apropos
behavior.
handled there.
- underline the name, section, and description so that it is prettier.
- change to bold terminal the terminal highlighting to match with less
the path from the _mandb variable from man.conf now.
Set _mandb in man.conf to same value as was used before.
From Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>.
This code has been developed by Abhinav Upadhyay as part of Google's Summer
of Code 2011. It uses libmandoc to parse man pages and builds a Full
Text Index in a SQLite database. The combination of indexing the full
manual page, filtering out stop words and ranking individual matches
based on the section gives a much improved user experience.
The old makewhatis and friends are kept under MKMAKEMANDB=no for now.
