Using the custom tokenizer means one cannot interactively query the database
through the SQLite shell, thus thwarting the purpose of the debug build option.
Thanks to leot@ for reporting it.
(While there change the debug macro from DEBUG to APROPOS_DEBUG)
Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)
The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.
Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.
The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).
To enforce using the new tokenizer, a schema version bump is needed
Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.
This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>
Thanks to christos@ for multiple rounds of reviews of the tokenizer code.
Originally, MKCRYPTO was introduced because the United States
classified cryptography as a munition and restricted its export. The
export controls were substantially relaxed fifteen years ago, and are
essentially irrelevant for software with published source code.
In the intervening time, nobody bothered to remove the option after
its motivation -- the US export restriction -- was eliminated. I'm
not aware of any other operating system that has a similar option; I
expect it is mainly out of apathy for churn that we still have it.
Today, cryptography is an essential part of modern computing -- you
can't use the internet responsibly without cryptography.
The position of the TNF board of directors is that TNF makes no
representation that MKCRYPTO=no satisfies any country's cryptography
regulations.
My personal position is that the availability of cryptography is a
basic human right; that any local laws restricting it to a privileged
few are fundamentally immoral; and that it is wrong for developers to
spend effort crippling cryptography to work around such laws.
As proposed on tech-crypto, tech-security, and tech-userlevel to no
objections:
https://mail-index.netbsd.org/tech-crypto/2017/05/06/msg000719.htmlhttps://mail-index.netbsd.org/tech-security/2017/05/06/msg000928.htmlhttps://mail-index.netbsd.org/tech-userlevel/2017/05/06/msg010547.html
P.S. Reviewing all the uses of MKCRYPTO in src revealed a lot of
*bad* crypto that was conditional on it, e.g. DES in telnet... That
should probably be removed too, but on the grounds that it is bad,
not on the grounds that it is (nominally) crypto.
Earlier, a white space separated string was generated containing all the section
numbers passed through command line arguments. Later on that would have to be
tokenized and processed. Instead of that, use a NULL terminated array of strings.
Thanks to christos@ for reviewing and suggesting further improvements.
mandoc(3) already generates the text node representing the value for the .Nm macro.
Doing our own parsing for .Nm on top of that leads to large duplication of text
in the database. This gets specially worse for man pages with large NAME sections,
such as queue(3).
Similar to apropos(1), whatis did not utilise the mandb_links table till now.
Therefore, if it was asked about one of the links to a man page, it would
error out. This change teaches whatis(1) to look up both the FTS table
as well as the links table, thus ensuring that it is able to answer queries
about MLINKS as well.
Comparision between outputs before this change and after this change:
#Before change
$ whatis realloc
realloc: not found
#after change
$ ./whatis realloc
realloc(3) - general memory allocation operations
realloc(3) - general purpose memory allocation functions
realloc(9) - general-purpose kernel memory allocator
apropos(1) only indexes the first .Nm entry from the NAME section in the full
text index. Rest of the .Nm entries are stored in a separate table: mandb_links.
Till now apropos(1) did not use the mandb_links table. So whenever a query
was being made for one of the man page links, such as realloc(3), it was showing
malloc(3) in the results but not as the first result. And, also the result would
show up as malloc(3), rather than realloc(3) (which can be confusing).
With this change, for single keyword queries, apropos(1) would now utilise the
mandb_links table as well. If the query is for one of the links of a man page,
it would show as the first result. Also, the result would show up as the name
of the link rather than the original man page name. For example, if the query
was for realloc, the output would be realloc(3), rather than malloc(3).
Following are some example queries showing difference in the output before this
change and after this change:
#Before changes
$ apropos -n 5 -M realloc
reallocarr (3) reallocate array
reallocarray (3) reallocate memory for an array of elements checking
for overflow
fgetwln (3) get a line of wide characters from a stream
fgetln (3) get a line from a stream
posix_memalign (3) aligned memory allocation
#After changes
$ ./apropos -n 5 -M realloc
realloc (3) general memory allocation operations
realloc (3) general purpose memory allocation functions
realloc (9) general-purpose kernel memory allocator
reallocarr (3) reallocate array
reallocarray (3) reallocate memory for an array of elements checking
for overflow
#Before changes
$ apropos -n 5 -M TAILQ_REMOVE
SLIST_HEAD (3) implementations of singly-linked lists, lists, simple
queues, tail queues, and singly-linked tail queues
#After changes
$ ./apropos -n 5 -M TAILQ_REMOVE
TAILQ_REMOVE (3) implementations of singly-linked lists, lists,
simple queues, tail queues, and singly-linked tail queues
#Before changes
$ apropos -n 5 -M falloc
filedesc (9) file descriptor tables and operations
file (9) operations on file entries
#After changes
$ ./apropos -n 5 -M falloc
falloc (9) file descriptor tables and operations
file (9) operations on file entries
ok christos@
There can be symlinks which are pointing to man pages not installed in
one of the _default locations mentioned in man.conf or MANPATH. For example
there are man pages in /usr/pkg/man which are symlinked to pages in
/usr/pkg/lib/perl5/man. If we ignore symlinks, we would not be able to
index such pages installed outside the default set of directories.
(Also, the symlink test was incorecct, so we never noticed this issue)
Ok christos@, wiz@
sections multiple times. This started to happen because, pmdoc_Sh(), the handler function
responsible for parsing the Sh macros, used to recursively go through all the child
nodes and then the next nodes starting from top level Sh block node.
Now, once it has processed all the child nodes of the top level block node,
it moves to the next node, which is the top level block node of the next section and
in this way one call to pmdoc_Sh() was causing a complete pass through the
man page. Since, mandoc(3) calls pmdoc_Sh() for each .Sh macro in the man
page, it would result in parsing some of the sections multiple times.
This never happened with the previous versions of mandoc, so we never noticed.
I've fixed this by starting the parse sequence of the Sh macro from its body, which gurantees
that we will stop once that section ends.
ok christos@
than parsing it by hand.
With the latest mandoc(3), the .Nd macro was getting parsed twice. This fixes
that problem and cleans up the code as well.
ok christos@
With the latest API, mdoc_validate()/man_validate() needs to be called before
reading the roff_man.meta field, otherwise it is NULL.
Also, if a man page doesn't specify machine architecture, don't default to '?'
, let it be stored as null in the db. Otherwise, the output of apropos(1) shows
the names of the results as \?/<title>
One part is responsible for generating the SQL query
The other part is responsible for executing the generated query.
While there, also remove a comment which is not valid anymore.
And, don't call the snippet function when doing legacy mode search as we are
not using the full text feature there.
makemandb(8), man(1) already use -C as an option to take man.conf path,
so use the same option for whatis(1) and apropos(1) for consitency.
apropos was using -C/-c to disable/enable context of the search
matches, change that to -M/-m respectively.
jmcneill