GitHub - jdidion/xphyle: Python library that facilitates opening, reading, and writing files (and file-like entities like URLs and streams) agnostic of compression format. (production) (original) (raw)

xphyle: extraordinarily simple file handling

xphyle is a small python library that makes it easy to open compressed files. Most importantly, xphyle will use the appropriate program (e.g. 'gzip') to compress/decompress a file if it is available on your system; this is almost always faster than using the corresponding python library. xphyle also provides methods that simplify common file I/O operations.

Recent version of xphyle (4.0.0+) require python 3.6. Older versions of xphyle support python 3.4+.

Please note that xphyle may work on Windows, but it is not tested.

Installation

Building from source

Clone this repository and run

Example usages:

from xphyle import * from xphyle.paths import STDIN, STDOUT

Open a compressed file...

myfile = xopen('infile.gz')

...or a compressed stream

e.g. gzip -c afile | python my_program.py

stdin = xopen(STDIN)

Easily write to the stdin of a subprocess

with open_('|cat', 'wt') as process: process.write('foo')

We have to tell xopen what kind of compression

to use when writing to stdout

stdout = xopen(STDOUT, compression='gz')

The open_ method ensures that the file is usable with the with keyword.

Print all lines in a compressed file...

with open_('infile.gz') as myfile: for line in myfile: print(line)

... or a compressed URL

with open_('http://foo.com/myfile.gz') as myfile: for line in myfile: print(line)

Transparently handle paths and file objects

def dostuff(path_or_file): with open_(path_or_file) as myfile: for line in myfile: print(line)

Read all lines in a compressed file into a list

from xphyle.utils import read_lines lines = list(read_lines('infile.gz'))

Sum the rows in a compressed file where each line is an integer value

total = sum(read_lines('infile.gz', convert=int))

See the Documentation for full usage information.

Supported compression formats

• gzip (uses igzip or pigz if available)
• bgzip
• bzip2 (uses pbzip2 if available)
• lzma
• zstd

Issues

Please report bugs and request enhancements using the issue tracker.

Roadmap

Future releases are mapped out using GitHub Projects.

Citing xphyle

Didion, JP (2017) xphyle: Extraordinarily simple file handling. Journal of Open Source Software; doi:10.21105/joss.00255

Acknowledgements

• Dependencies scanned by PyUp.io
• Thanks to @ctb for reviewing the xphyle paper