GitHub - klauspost/compress: Optimized Go Compression Packages (original) (raw)

compress

This package provides various compression algorithms.

Go Reference Go Sourcegraph Badge

package usage

Use go get github.com/klauspost/compress@latest to add it to your project.

This package will support the current Go version and 2 versions back.

Use the links above for more information on each.

changelog

Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.

Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.

While the release has been extensively tested, it is recommended to testing when upgrading.

See changes to v1.14.x

deflate usage

The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:

Typical speed is about 2x of the standard library packages.

old import new import Documentation
compress/gzip github.com/klauspost/compress/gzip gzip
compress/zlib github.com/klauspost/compress/zlib zlib
archive/zip github.com/klauspost/compress/zip zip
compress/flate github.com/klauspost/compress/flate flate

You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.

The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.

Currently there is only minor speedup on decompression (mostly CRC32 calculation).

Memory usage is typically 1MB for a Writer. stdlib is in the same range. If you expect to have a lot of concurrently allocated Writers consider using the stateless compress described below.

For compression performance, see: this spreadsheet.

To disable all assembly add -tags=noasm. This works across all packages.

Stateless compression

This package offers stateless compression as a special option for gzip/deflate. It will do compression but without maintaining any state between Write calls.

This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.

This is only relevant in cases where you expect to run many thousands of compressors concurrently, but with very little activity. This is not intended for regular web servers serving individual requests.

Because of this, the size of actual Write calls will affect output size.

In gzip, specify level -3 / gzip.StatelessCompression to enable.

For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation

A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:

// replace 'ioutil.Discard' with your output.
gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
if err != nil {
    return err
}
defer gzw.Close()

w := bufio.NewWriterSize(gzw, 4096)
defer w.Flush()

// Write to 'w'

This will only use up to 4KB in memory when the writer is idle.

Compression is almost always worse than the fastest compression level and each write will allocate (a little) memory.

Other packages

Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):

license

This code is licensed under the same conditions as the original Go code. See LICENSE file.