Aren
/
NetBSD
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
NetBSD/usr.sbin
History
abhinav 188f922ddf Add a custom tokenizer which does not stem certain keywords. 
Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)

The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.

Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.

The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).

To enforce using the new tokenizer, a schema version bump is needed

Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.

This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>

Thanks to christos@ for multiple rounds of reviews of the tokenizer code.
 		2017-06-18 16:24:10 +00:00
..
ac New sentence, new line. 2016-12-21 22:49:14 +00:00
accton
…
acpitools Drop pointless packed attributes. 2016-10-04 22:22:16 +00:00
altq PR/50824: David Binderman: Limit scanf width 2016-02-17 19:51:29 +00:00
apm This change arm, armeb, earm, earmeb, earmhf, earmhfeb so all builds that 2013-05-02 03:56:38 +00:00
apmd This change arm, armeb, earm, earmeb, earmhf, earmhfeb so all builds that 2013-05-02 03:56:38 +00:00
arp whitespace police 2017-06-06 19:59:10 +00:00
bad144 Fix the 3 programs that use DTYPE_ (disklabel disk types) constants from the 2015-01-02 19:46:02 +00:00
bootp Drop main() prototype. 2017-05-04 16:26:09 +00:00
bta2dpd Streaming channel is now opened in blocking mode again. 2017-05-27 10:04:57 +00:00
btattach a couple more stray \n 2015-06-16 23:54:58 +00:00
btconfig add details of later (to v4.2) specification versions and 2015-11-28 09:25:45 +00:00
btdevctl the Bluetooth rc.d setup was simplified some time ago, so 2015-06-12 09:43:05 +00:00
bthcid need <sys/stat.h> for chmod(2) 2017-01-10 20:56:41 +00:00
btpand The last argument before ... must not be default promoted, otherwise the 2016-10-04 21:40:31 +00:00
catman Call the path for makewhatis _PATH_MAKEWHATIS instead of _PATH_WHATIS, 2016-05-29 22:33:39 +00:00
chroot
…
cnwctl avoid pointer gymnastics 2013-10-19 17:05:58 +00:00
cpuctl Update from the latest Intel SDM: 2016-10-11 04:16:28 +00:00
crash Protect the right part. 2017-04-10 12:05:52 +00:00
dev_mkdb Remove many HAVE_GCC || HAVE_PCC conditionals as the options also apply 2012-08-10 12:10:27 +00:00
diskpart Fix the 3 programs that use DTYPE_ (disklabel disk types) constants from the 2015-01-02 19:46:02 +00:00
dumpfs Print FS_TRIM too. 2013-09-03 04:02:13 +00:00
dumplfs If the number of inodes per block is zero, bail instead of using 2016-08-12 08:22:13 +00:00
edquota Grr. The ATF quota tests demand failing silently in certain circumstances. 2012-08-14 04:53:43 +00:00
eeprom Finish ANSIfication. Drop extra spaces from __P removal. Add more 2013-07-02 11:59:46 +00:00
envstat add HYT-221/271/939 humidity/temperature I2C sensor 2014-05-18 11:46:23 +00:00
eshconfig
…
etcupdate From Jim Bernard in PR install/50615: install_checksum expects a path 2016-01-03 08:44:59 +00:00
extattrctl
…
faithd use new scopeid functions 2013-10-19 17:16:25 +00:00
flashctl
…
fssconfig Include rump-vs-host ops files 2016-07-31 02:15:54 +00:00
fusermount
…
fwctl Merge riastradh-drm2 to HEAD. 2014-03-18 18:20:35 +00:00
gpioctl Fix pasto for GPIO_PIN_EVENTS/GPIO_PIN_LEVEL/GPIO_PIN_FALLING 2016-04-05 10:58:04 +00:00
grfconfig PR/50867: David Binderman: Fix parsing loop. 2016-02-29 18:59:52 +00:00
gspa
…
hdaudioctl Split hdaudio and PCI attach glue. Even though the driver was written with 2015-03-28 14:09:58 +00:00
i2cscan Whitespace. 2015-11-26 23:29:05 +00:00
ifmcstat Remove mkludge stuffs 2017-02-23 07:57:09 +00:00
ifwatchd Use RO_MSGFILTER. 2017-04-11 14:15:08 +00:00
inetd Increase buffer size reported to strlcpy() to be one larger than the 2017-02-15 02:48:31 +00:00
installboot Bump date for previous. 2017-03-15 11:22:47 +00:00
intrctl fix column alignment of "intrctl list [-c]" 2016-11-02 11:03:33 +00:00
iopctl
…
iostat Enhance disk metrics by calculating a weighted sum that is incremented 2017-03-05 23:07:12 +00:00
ip6addrctl usage is dead 2015-12-27 12:42:52 +00:00
ipwctl Merge riastradh-drm2 to HEAD. 2014-03-18 18:20:35 +00:00
irdaattach
…
isdn fix indent 2016-03-02 19:28:56 +00:00
isibootd
…
iteconfig
…
iwictl Merge riastradh-drm2 to HEAD. 2014-03-18 18:20:35 +00:00
kgmon for 64 bit mips platforms where we built userland largely as n32 by 2016-12-23 10:19:57 +00:00
kvm_mkdb
…
lastlogin
…
ldpd Use RO_MSGFILTER. 2017-04-12 17:02:51 +00:00
link
…
lmcconfig Fix typo ("then" instead of "than") 2013-12-09 09:35:16 +00:00
lockstat move all the necessay code inside ifdef 2013-10-19 17:10:17 +00:00
lpr Drop main() prototype. 2017-05-04 16:26:09 +00:00
lptctl Mark up author with .An. 2014-09-19 16:06:16 +00:00
mailwrapper New sentence, new line. 2014-09-19 16:05:55 +00:00
makefs leave the size alone and set the create_size to include the offset. It 2017-04-14 15:40:35 +00:00
makemandb Add a custom tokenizer which does not stem certain keywords. 2017-06-18 16:24:10 +00:00
map-mbone
…
mdconfig
…
mdsetimage Need <stdint.h> for uintmax_t. 2016-12-18 18:32:24 +00:00
memswitch Fix typo. display.contrast is 0-15, not 0-1. 2015-09-21 08:04:43 +00:00
mld6query PR/51353: Patrick Mackey: mld6query "-r" option is not recognised 2016-07-20 23:19:18 +00:00
mlxctl s/LIButil/LIBUTIL/ in DPADD 2016-12-10 13:55:55 +00:00
mmcformat Merge riastradh-drm2 to HEAD. 2014-03-18 18:20:35 +00:00
mopd delete generated file 2016-06-08 02:58:19 +00:00
mountd Remove xref to signal(7). 2016-12-24 08:26:57 +00:00
moused Use .An -nosplit to avoid unwanted line breaks in the middle of a sentence. 2016-08-16 02:04:01 +00:00
mrinfo Add couple FALLTHROUGHs CIDs: 976559, 976560, 976561 2016-11-17 09:29:01 +00:00
mrouted Fix bug in mrouted about aging neighbors. 2015-02-05 16:50:19 +00:00
mscdlabel need <sys/types.h> 2017-01-10 21:03:36 +00:00
mtrace PR/50966: David Binderman: Use all 3 of b, p, n to determine if route changed 2016-03-13 19:47:59 +00:00
mtree Fix typo. 2017-02-22 14:15:15 +00:00
ndbootd
…
ndiscvt improve error messages (remove \n, use __func__, etc) 2015-06-16 23:04:13 +00:00
ndp remove unused ancient code and widen v6 address so that columns are aligned. 2016-09-19 19:13:14 +00:00
netgroup_mkdb
…
nfsd Don't exit if we have udp only sockets, instead wait for all the workers 2016-08-23 13:10:12 +00:00
npf set "testing mode" for ioctl I/O 2017-05-17 18:55:13 +00:00
ofctl Implement OF_finddevice, now ofctl -p <path to node> works 2015-12-23 13:42:24 +00:00
paxctl New sentence, new line. 2016-11-08 08:21:52 +00:00
pcictl apply the previous typo in another place as well. 2016-09-27 16:44:04 +00:00
perfused remove dup function 2016-10-19 01:31:05 +00:00
pf Enable carp packets early during boot, to avoid gratuitous failovers. 2017-05-26 15:40:27 +00:00
postinstall add missing pam items (cron, racoon) 2017-06-16 22:58:38 +00:00
powerd Fix spelling of "occurs". 2017-02-21 15:15:19 +00:00
psrset
…
pstat Drop main() prototype. 2017-05-04 16:26:09 +00:00
puffs Remove MKCRYPTO option. 2017-05-21 15:28:36 +00:00
pvcsif
…
pvctxctl
…
pwd_mkdb Drop main() prototype. 2017-05-04 16:26:09 +00:00
quot From Michael Plass: 2016-07-28 08:24:58 +00:00
quotacheck improve error messages (remove \n, use __func__, etc) 2015-06-16 23:04:13 +00:00
quotaon
…
quotarestore Don't use const foo const as type, one const is enough. 2012-09-13 21:44:49 +00:00
racoon Remove MKCRYPTO_RC5. Unconditionally include RC5 in libcrypto.so. 2017-05-21 14:20:44 +00:00
racoonctl
…
rarpd Use RO_MSGFILTER. 2017-04-12 16:47:39 +00:00
rbootd Drop main() prototype. 2017-05-04 16:26:09 +00:00
rdate Drop main() prototype. 2017-05-04 16:26:09 +00:00
repquota improve error messages (remove \n, use __func__, etc) 2015-06-16 23:04:13 +00:00
rip6query
…
rmt
…
route6d Separate nexthop caches from the routing table 2016-04-04 07:37:07 +00:00
rpc.bootparamd
…
rpc.lockd this file uses the <sys/queue.h> macros so it should include 2017-06-08 03:25:03 +00:00
rpc.pcnfsd Add a statuslen argument to get_pr_status(), and use it 2014-03-29 18:54:36 +00:00
rpc.statd remove my copyright. 2017-06-03 14:44:12 +00:00
rpc.yppasswdd Drop main() prototype. 2017-05-04 16:26:09 +00:00
rpcbind Limit size of buffer: 2017-05-03 21:28:00 +00:00
rtadvd Use RO_MSGFILTER. 2017-04-11 14:29:23 +00:00
rtsold Give an example how to use dhcpcd instead of rtsol 2014-10-12 19:53:34 +00:00
rwhod include <pwd.h> 2012-11-04 22:32:01 +00:00
sa New sentence, new line. 2014-09-19 16:05:55 +00:00
schedctl Fix illegal snprintf usage noted in PR 47976 by simplifying it away: 2014-07-27 04:46:48 +00:00
screenblank
…
sdpd newer versions of the Bluetooth Core specification state that 2015-11-24 21:11:39 +00:00
service Only match first instance of name and rcvar in file. 2017-06-05 09:20:05 +00:00
services_mkdb need <sys/stat.h> 2017-01-10 21:04:06 +00:00
sesd Sync usage with man page. 2013-04-07 18:49:35 +00:00
sliplogin Add USE_FORT support. 2016-07-13 13:10:04 +00:00
spray
…
srtconfig Drop main() prototype. 2017-05-04 16:26:09 +00:00
sti Improve DESCRIPTION, fix up English usage. 2012-08-14 08:08:29 +00:00
sunlabel Drop main() prototype. 2017-05-04 16:26:09 +00:00
sup Drop main() prototype. 2017-05-04 16:26:09 +00:00
sysinst Drop main() prototype. 2017-05-04 16:26:09 +00:00
syslogd Remove MKCRYPTO option. 2017-05-21 15:28:36 +00:00
tadpolectl Make usage static and dead. 2015-10-02 20:42:15 +00:00
tcpdchk eliminate %m complexity. __format_arg__ thinks it is handling a format for 2015-10-14 15:50:48 +00:00
tcpdmatch
…
tcpdrop
…
timed improve error messages (remove \n, use __func__, etc) 2015-06-16 23:04:13 +00:00
tpctl Mark usage as dead. 2013-11-29 23:01:05 +00:00
tprof
…
traceroute PR/50821: David Binderman: remove "i <" i < 2016-02-17 19:57:01 +00:00
traceroute6 CID 978587: Removed unnecessary check against NULL. 2016-11-17 09:21:34 +00:00
trpt for 64 bit mips platforms where we built userland largely as n32 by 2016-12-23 10:19:57 +00:00
unlink
…
usbdevs Fix previous... handle hub ports 2016-09-07 08:09:59 +00:00
user Document file format better. From Travis Paul and Matthew Bauer. 2016-02-09 14:14:02 +00:00
veriexecgen
…
videomode
…
vipw
…
vnconfig Sort sections. Use more markup. 2016-04-10 09:49:14 +00:00
wakeonlan Fix Dd argument. 2017-02-26 22:18:58 +00:00
wiconfig Kill another strict aliasing violation. 2016-06-15 13:47:26 +00:00
wlanctl remove -v no/op 2014-04-22 15:55:16 +00:00
wsconscfg Drop main() prototype. 2017-05-04 16:26:09 +00:00
wsfontload remove extraneous malloc.h 2013-04-05 03:18:26 +00:00
wsmoused
…
wsmuxctl
…
ypbind Add missing .El. From Henning Petersen in PR 49227. 2014-09-20 11:16:05 +00:00
yppoll
…
ypserv Drop main() prototype. 2017-05-04 16:26:09 +00:00
ypset
…
zdump we keep version in a separate file now 2016-10-20 17:39:08 +00:00
zic don't try to run a non-existing nbcat 2017-01-06 17:25:32 +00:00
Makefile Remove MKCRYPTO option. 2017-05-21 15:28:36 +00:00
Makefile.inc Define _KERNTYPES for things that need it. 2016-01-23 21:22:45 +00:00