Feather File Format — Apache Arrow v20.0.0 (original) (raw)

Feather is a portable file format for storing Arrow tables or data frames (from languages like Python or R) that utilizes the Arrow IPC formatinternally. Feather was created early in the Arrow project as a proof of concept for fast, language-agnostic data frame storage for Python (pandas) and R. There are two file format versions for Feather:

• Version 2 (V2), the default version, which is exactly represented as the Arrow IPC file format on disk. V2 files support storing all Arrow data types as well as compression with LZ4 or ZSTD. V2 was first made available in Apache Arrow 0.17.0.
• Version 1 (V1), a legacy version available starting in 2016, replaced by V2. V1 files are distinct from Arrow IPC files and lack many features, such as the ability to store all Arrow data types. V1 files also lack compression support. We intend to maintain read support for V1 for the foreseeable future.

The pyarrow.feather module contains the read and write functions for the format. write_feather() accepts either aTable or pandas.DataFrame object:

import pyarrow.feather as feather feather.write_feather(df, '/path/to/file')

read_feather() reads a Feather file as apandas.DataFrame. read_table() reads a Feather file as a Table. Internally, read_feather()simply calls read_table() and the result is converted to pandas:

Result is pandas.DataFrame

read_df = feather.read_feather('/path/to/file')

Result is pyarrow.Table

read_arrow = feather.read_table('/path/to/file')

These functions can read and write with file-paths or file-like objects. For example:

with open('/path/to/file', 'wb') as f: feather.write_feather(df, f)

with open('/path/to/file', 'rb') as f: read_df = feather.read_feather(f)

A file input to read_feather must support seeking.

Using Compression#

As of Apache Arrow version 0.17.0, Feather V2 files (the default version) support two fast compression libraries, LZ4 (using the frame format) and ZSTD. LZ4 is used by default if it is available (which it should be if you obtained pyarrow through a normal package manager):

Uses LZ4 by default

feather.write_feather(df, file_path)

Use LZ4 explicitly

feather.write_feather(df, file_path, compression='lz4')

Use ZSTD

feather.write_feather(df, file_path, compression='zstd')

Do not compress

feather.write_feather(df, file_path, compression='uncompressed')

Note that the default LZ4 compression generally yields much smaller files without sacrificing much read or write performance. In some instances, LZ4-compressed files may be faster to read and write than uncompressed due to reduced disk IO requirements.

Writing Version 1 (V1) Files#

For compatibility with libraries without support for Version 2 files, you can write the version 1 format by passing version=1 to write_feather. We intend to maintain read support for V1 for the foreseeable future.