These are just some random thoughts of mine. Your mileage may vary.
bzip2-0.9.0
uses exactly the same file format as the previous
version, bzip2-0.1
. This decision was made in the interests of
stability. Creating yet another incompatible compressed file format
would create further confusion and disruption for users.
Nevertheless, this is not a painless decision. Development
work since the release of bzip2-0.1
in August 1997
has shown complexities in the file format which slow down
decompression and, in retrospect, are unnecessary. These are:
bzip2
's existing algorithm for
most inputs, and the randomisation mechanism protects
adequately against bad cases. I didn't think it was
a good tradeoff to make. Partly this is due to the fact
that I was not flooded with email complaints about
bzip2-0.1
's performance on repetitive data, so
perhaps it isn't a problem for real inputs.
Probably the best long-term solution
is to use the existing sorting
algorithm initially, and fall back to a O(N (log N)^2)
algorithm if the standard algorithm gets into difficulties.
This can be done without much difficulty; I made
a prototype implementation of it some months now.
decompress.c
through the C preprocessor
and you'll see what I mean. Much of this complexity
could have been avoided if the compressed size of
each block of data was recorded in the data stream.
It would be fair to say that the bzip2
format was frozen
before I properly and fully understood the performance
consequences of doing so.
Improvements which I have been able to incorporate into 0.9.0, despite using the same file format, are:
bzip2-0.9.0
now reads and writes files with fread
and fwrite
; version 0.1 used putc
and getc
.
Duh! I'm embarrassed at my own moronicness (moronicity?) on this
one.
Further ahead, it would be nice to be able to do random access into files. This will require some careful design of compressed file formats.
After some consideration, I have decided not to use
GNU autoconf
to configure 0.9.0.
autoconf
, admirable and wonderful though it is,
mainly assists with portability problems between Unix-like
platforms. But bzip2
doesn't have much in the way
of portability problems on Unix; most of the difficulties appear
when porting to the Mac, or to Microsoft's operating systems.
autoconf
doesn't help in those cases, and brings in a
whole load of new complexity.
Most people should be able to compile the library and program under Unix straight out-of-the-box, so to speak, especially if you have a version of GNU C available.
There are a couple of __inline__
directives in the code. GNU C
(gcc
) should be able to handle them. If your compiler doesn't
like them, just #define
__inline__
to be null. One
easy way to do this is to compile with the flag -D__inline__=
,
which should be understood by most Unix compilers.
If you still have difficulties, try compiling with the macro
BZ_STRICT_ANSI
defined. This should enable you to build the
library in a strictly ANSI compliant environment. Building the program
itself like this is dangerous and not supported, since you remove
bzip2
's checks against compressing directories, symbolic links,
devices, and other not-really-a-file entities. This could cause
filesystem corruption!
One other thing: if you create a bzip2
binary for public
distribution, please try and link it statically (gcc -s
). This
avoids all sorts of library-version issues that others may encounter
later on.
I tried pretty hard to make sure bzip2
is
bug free, both by design and by testing. Hopefully
you'll never need to read this section for real.
Nevertheless, if bzip2
dies with a segmentation
fault, a bus error or an internal assertion failure, it
will ask you to email me a bug report. Experience with
version 0.1 shows that almost all these problems can
be traced to either compiler bugs or hardware problems.
bzip2
, and I've run across two such examples myself.
2.7.X versions of GNU C are known to generate bad code from
time to time, at high optimisation levels.
If you get problems, try using the flags
-O2
-fomit-frame-pointer
-fno-strength-reduce
.
You should specifically not use -funroll-loops
.
You may notice that the Makefile runs four tests as part of
the build process. If the program passes all of these, it's
a pretty good (but not 100%) indication that the compiler has
done its job correctly.
bzip2
crashes randomly, and the crashes are not
repeatable, you may have a flaky memory subsystem. bzip2
really hammers your memory hierarchy, and if it's a bit marginal,
you may get these problems. Ditto if your disk or I/O subsystem
is slowly failing. Yup, this really does happen.
Try using a different machine of the same type, and see if
you can repeat the problem.
bzip2
tells
you your file is corrupted on decompression, and you
obtained the file via FTP, there is a possibility that you
forgot to tell FTP to do a binary mode transfer. That absolutely
will cause the file to be non-decompressible. You'll have to transfer
it again.
If you've incorporated libbzip2
into your own program
and are getting problems, please, please, please, check that the
parameters you are passing in calls to the library, are
correct, and in accordance with what the documentation says
is allowable. I have tried to make the library robust against
such problems, but I'm sure I haven't succeeded.
Finally, if the above comments don't help, you'll have to send me a bug report. Now, it's just amazing how many people will send me a bug report saying something like
bzip2 crashed with segmentation fault on my machine
and absolutely nothing else. Needless to say, a such a report is totally, utterly, completely and comprehensively 100% useless; a waste of your time, my time, and net bandwidth. With no details at all, there's no way I can possibly begin to figure out what the problem is.
The rules of the game are: facts, facts, facts. Don't omit them because "oh, they won't be relevant". At the bare minimum:
Machine type. Operating system version. Exact version ofbzip2
(dobzip2 -V
). Exact version of the compiler used. Flags passed to the compiler.
However, the most important single thing that will help me is the file that you were trying to compress or decompress at the time the problem happened. Without that, my ability to do anything more than speculate about the cause, is limited.
Please remember that I connect to the Internet with a modem, so you should contact me before mailing me huge files.
bzip2
is a resource hog. It soaks up large amounts of CPU cycles
and memory. Also, it gives very large latencies. In the worst case, you
can feed many megabytes of uncompressed data into the library before
getting any compressed output, so this probably rules out applications
requiring interactive behaviour.
These aren't faults of my implementation, I hope, but more an intrinsic property of the Burrows-Wheeler transform (unfortunately). Maybe this isn't what you want.
If you want a compressor and/or library which is faster, uses less
memory but gets pretty good compression, and has minimal latency,
consider Jean-loup
Gailly's and Mark Adler's work, zlib-1.1.2
and
gzip-1.2.4
. Look for them at
http://www.cdrom.com/pub/infozip/zlib
and
http://www.gzip.org
respectively.
For something faster and lighter still, you might try Markus F X J
Oberhumer's LZO
real-time compression/decompression library, at
http://wildsau.idv.uni-linz.ac.at/mfx/lzo.html
.
If you want to use the bzip2
algorithms to compress small blocks
of data, 64k bytes or smaller, for example on an on-the-fly disk
compressor, you'd be well advised not to use this library. Instead,
I've made a special library tuned for that kind of use. It's part of
e2compr-0.40
, an on-the-fly disk compressor for the Linux
ext2
filesystem. Look at
http://www.netspace.net.au/~reiter/e2compr
.
A record of the tests I've done.
First, some data sets:
egcs-1.0.2
, gcc-2.8.1
, KDE Beta 4, GTK, Octave, etc.
827 megabytes.
.tar
files) of the entire GNU distribution, plus a couple of
Linux distributions. 2400 megabytes.
The tests conducted are as follows. Each test means compressing (a copy of) each file in the data set, decompressing it and comparing it against the original.
First, a bunch of tests with block sizes, internal buffer sizes and randomisation lengths set very small, to detect any problems with the blocking, buffering and randomisation mechanisms. This required modifying the source code so as to try to break it.
Then some tests with unmodified source code.
-1
.
-s
, decompress with flag -s
.
bzip2-0.1pl2
compressing,
bzip2-0.9.0
decompressing, all settings normal.
bzip2-0.9.0
compressing,
bzip2-0.1pl2
decompressing, all settings normal.
.tar
files with
bzip2
compiled with Purify.
These tests were conducted on a 205 MHz Cyrix 6x86MX machine, running Linux 2.0.32. They represent nearly a week of continuous computation. All tests completed successfully.
bzip2
is not research work, in the sense that it doesn't present
any new ideas. Rather, it's an engineering exercise based on existing
ideas.
Four documents describe essentially all the ideas behind bzip2
:
Michael Burrows and D. J. Wheeler: "A block-sorting lossless data compression algorithm" 10th May 1994. Digital SRC Research Report 124. ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz If you have trouble finding it, try searching at the New Zealand Digital Library, http://www.nzdl.org. Daniel S. Hirschberg and Debra A. LeLewer "Efficient Decoding of Prefix Codes" Communications of the ACM, April 1990, Vol 33, Number 4. You might be able to get an electronic copy of this from the ACM Digital Library. David J. Wheeler Program bred3.c and accompanying document bred3.ps. This contains the idea behind the multi-table Huffman coding scheme. ftp://ftp.cl.cam.ac.uk/pub/user/djw3/ Jon L. Bentley and Robert Sedgewick "Fast Algorithms for Sorting and Searching Strings" Available from Sedgewick's web page, www.cs.princeton.edu/~rs
The following paper gives valuable additional insights into the algorithm, but is not immediately the basis of any code used in bzip2.
Peter Fenwick: Block Sorting Text Compression Proceedings of the 19th Australasian Computer Science Conference, Melbourne, Australia. Jan 31 - Feb 2, 1996. ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps
Kunihiko Sadakane's sorting algorithm, mentioned above, is available from:
http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
The Manber-Myers suffix array construction algorithm is described in a paper available from:
http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
Go to the first, previous, next, last section, table of contents.