mirrors/mc
1
0
Fork 0
mirror of https://github.com/MidnightCommander/mc synced 2024-12-31 16:44:24 +03:00
Code Issues Packages Projects Releases Wiki Activity

Ticket #3616: speed up of utf-8 normalization.

Browse Source 
When content of a large directory is being sorted by file names, a
significant amount of CPU time is spent in str_utf8_normalize() that is
called from str_utf8_create_key_gen().

For example, /usr/bin/ contains 5437 files on my Archlinux box. Running
mc /usr/bin/ /usr/bin/ takes approx. 75 000 000 CPU instructions to sort
file names, or 25% of total program run time. From these 75 000 000
instructions, 42 500 000 instruction are spent in str_utf8_normalize().

str_utf8_normalize() uses g_utf8_normalize() to do the work.
g_utf8_normalize() is a heavyweight function, that converts UTF-8 into
UCS-4, does the normalization and then converts UCS-4 back into UTF-8.

Since file names are composed of ASCII characters in most cases, we can
speed up str_utf8_normalize() by checking if the heavyweight Unicode
normalization is actually needed. Normalization of ASCII string is
no-op, so it is effectively "normalized" by just strdup().

With this patch, running mc /usr/bin/ /usr/bin/ requires just 37 000 000
instructions to sort the file names (down from 75 000 000) and 4 500 000
instuctions to do str_utf8_normalize() (down from 42 500 000).

Signed-off-by: Andrew Borodin <aborodin@vmail.ru>
This commit is contained in:
devzero 2017-07-28 13:19:54 +03:00 committed by Andrew Borodin
parent eab2556ed8
commit 7f9b333861
1 changed files with 19 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
lib/strutil/strutilutf8.c
View File

@ -1080,6 +1080,25 @@ str_utf8_normalize (const char *text)
const char *start;
const char *end;
/* g_utf8_normalize() is a heavyweight function, that converts UTF-8 into UCS-4,
* does the normalization and then converts UCS-4 back into UTF-8.
* Since file names are composed of ASCII characters in most cases, we can speed up
* utf8 normalization by checking if the heavyweight Unicode normalization is actually
* needed. Normalization of ASCII string is no-op.
*/
/* find out whether text is ASCII only */
for (end = text; *end != '\0'; end++)
if ((*end & 0x80) != 0)
{
/* found 2nd byte of utf8-encoded symbol */
break;
}
/* if text is ASCII-only, return copy, normalize otherwise */
if (*end == '\0')
return g_strndup (text, end - text);
fixed = g_string_sized_new (4);
start = text;