commit all archived files
This commit is contained in:
parent
e929740825
commit
357448b5b9
|
|
@ -0,0 +1,67 @@
|
|||
# compression-performance-test
|
||||
|
||||
|
||||
|
||||
A test suite for general purpose lossless data compressors.
|
||||
|
||||
This project contains a set of scripts to compare various compression software, with a target metric of maximising the compression ratio for a given compression speed. Decompression speed and memory usage were not tested.
|
||||
|
||||
## Test Environment
|
||||
|
||||
This project also includes a sample output, tested on an Intel Sandy Bridge desktop PC under Cygwin64.
|
||||
|
||||
The tested compressors are (in alphabetical order):
|
||||
- `brotli` (compiled from git head on cygwin64)
|
||||
- `bzip2` (version 1.0.6, from cygwin64 package manager)
|
||||
- `cat`
|
||||
- `gzip` (version 1.7, from cygwin64 package manager)
|
||||
- `lz4` (1.7.2, compiled from git on cygwin64)
|
||||
- `lz5` (1.5.0, compiled from git on cygwin64)
|
||||
- `rar` (5.4.0 windows64 evaluation binary)
|
||||
- `xz` (5.2.2, from cygwin64 package manager)
|
||||
- `zpaq` (7.15, compiled from release archive on cygwin64)
|
||||
- `zstd` (1.1.0, compiled from git for mingw-w64)
|
||||
|
||||
The test corpus files are:
|
||||
- `debian-security-tracker-20160924.json`, a 15MB json file from https://security-tracker.debian.org/tracker/
|
||||
- `helloworld.exe`, a 1.6MB hello-world binary for x86_64, written in Golang
|
||||
- `openftp4_all_20160918.log`, a 10MB unsorted list of IPv4 addresses from https://github.com/massivedynamic/openftp4
|
||||
- `veekun-pokedex-20160328.tar`, a 15MB csv corpus from https://github.com/veekun/pokedex
|
||||
|
||||
## Test Results
|
||||
|
||||
- `cat` always provides the fastest compression.
|
||||
- `zpaq` always provides the best compression ratio, always at the slowest compression time. If you have infinite time to compress data and simply want the best ratio, you should always choose zpaq.
|
||||
|
||||
Otherwise, there's a frontier between these two extremes (Matt Mahoney, the author of zpaq, refers to this as a Pareto Frontier). Along this frontier, different compressors 'win' or 'lose', depending both on time requirements and both the data content.
|
||||
|
||||
The test file, listed with its time/ratio optimal compressors:
|
||||
|
||||
- `debian-security-tracker-20160924.json`: cat > lz4/lz5 (briefly) > zstd > brotli > zpaq.
|
||||
- `helloworld.exe`: cat > lz4/lz5 (briefly) > zstd > brotli > xz (punctuated with rar) > zpaq.
|
||||
- `openftp4_all_20160918.log`: cat > lz4/lz5 (briefly) > brotli (briefly) > zstd (over a large range of times) > bzip2 (surprisingly) > zpaq.
|
||||
- `veekun-pokedex-20160328.tar`: cat > lz4 > zstd > xz (over a large range of times) > zpaq.
|
||||
|
||||
Observations:
|
||||
|
||||
- The only time `rar` made an optimal showing was during the .exe test. This probably demonstrates use of custom filters for specific file formats.
|
||||
- `gzip` was always outclassed at every level.
|
||||
- `lz4`/`lz5` have an advantage at very low compression levels (e.g. -2/-3) but are immediately outclassed for the entire rest of their range.
|
||||
- `bzip2` takes a very similar amount of time to compress, regardless of the target compression ratio.
|
||||
- The internet community seems to consider `bzip2` obsolete in favour of `xz`. This might be true in general, but there are still cases where `bzip2` shines above it.
|
||||
|
||||
<html>
|
||||
<div>
|
||||
<img src="https://code.ivysaur.me/img/compression-performance-test_0.png" style="max-width:100%;"><br>
|
||||
<img src="https://code.ivysaur.me/img/compression-performance-test_1.png" style="max-width:100%;"><br>
|
||||
<img src="https://code.ivysaur.me/img/compression-performance-test_2.png" style="max-width:100%;"><br>
|
||||
<img src="https://code.ivysaur.me/img/compression-performance-test_3.png" style="max-width:100%;"><br>
|
||||
</div>
|
||||
</html>
|
||||
|
||||
## Changelog
|
||||
|
||||
2016-09-24 v1
|
||||
- Initial release
|
||||
- [⬇️ compression-performance-test-v1.zip](dist-archive/compression-performance-test-v1.zip) *(5.92 KiB)*
|
||||
|
||||
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Reference in New Issue