pyarrow.RecordBatch — Apache Arrow v20.0.0 (original) (raw)
class pyarrow.RecordBatch#
Bases: _Tabular
Batch of rows of columns of equal length
Warning
Do not call this class’s constructor directly, use one of theRecordBatch.from_* functions instead.
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"]
Constructing a RecordBatch from arrays:
pa.RecordBatch.from_arrays([n_legs, animals], names=names) pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,2,4,4,5,100] animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Constructing a RecordBatch from pandas DataFrame:
import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2021, 2022], ... 'month': [3, 5, 7, 9], ... 'day': [1, 5, 9, 13], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) pa.RecordBatch.from_pandas(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string
year: [2020,2022,2021,2022] month: [3,5,7,9] day: [1,5,9,13] n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
pa.RecordBatch.from_pandas(df).to_pandas() year month day n_legs animals 0 2020 3 1 2 Flamingo 1 2022 5 5 4 Horse 2 2021 7 9 5 Brittle stars 3 2022 9 13 100 Centipede
Constructing a RecordBatch from pylist:
pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, ... {'n_legs': 4, 'animals': 'Dog'}] pa.RecordBatch.from_pylist(pylist).to_pandas() n_legs animals 0 2 Flamingo 1 4 Dog
You can also construct a RecordBatch using pyarrow.record_batch():
pa.record_batch([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
pa.record_batch(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string
year: [2020,2022,2021,2022] month: [3,5,7,9] day: [1,5,9,13] n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
__init__(*args, **kwargs)#
Methods
Attributes
__dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True)#
Return the dataframe interchange object implementing the interchange protocol.
Parameters:
nan_as_nullbool, default False
Whether to tell the DataFrame to overwrite null values in the data with NaN (or NaT).
Whether to allow memory copying when exporting. If set to False it would cause non-zero-copy exports to fail.
Returns:
DataFrame interchange object
The object which consuming library can use to ingress the dataframe.
Notes
Details on the interchange protocol:https://data-apis.org/dataframe-protocol/latest/index.html nan_as_null currently has no effect; once support for nullable extension dtypes is added, this value should be propagated to columns.
add_column(self, int i, field_, column)#
Add column to RecordBatch at position i.
A new record batch is returned with the column added, the original record batch object is left unchanged.
Parameters:
iint
Index to place the column at.
If a string is passed then the type is deduced from the column data.
columnArray or value coercible to array
Column data.
Returns:
New record batch with the passed column added.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) batch = pa.RecordBatch.from_pandas(df)
Add column:
year = [2021, 2022, 2019, 2021] batch.add_column(0,"year", year) pyarrow.RecordBatch year: int64 n_legs: int64 animals: string
year: [2021,2022,2019,2021] n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
Original record batch is left unchanged:
batch pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
append_column(self, field_, column)#
Append column at end of columns.
Parameters:
If a string is passed then the type is deduced from the column data.
columnArray or value coercible to array
Column data.
Returns:
Table or RecordBatch
New table or record batch with the passed column added.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Append column at the end:
year = [2021, 2022, 2019, 2021] table.append_column('year', [year]) pyarrow.Table n_legs: int64 animals: string year: int64
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]] year: [[2021,2022,2019,2021]]
cast(self, Schema target_schema, safe=None, options=None)#
Cast record batch values to another schema.
Parameters:
target_schemaSchema
Schema to cast to, the names and order of fields must match.
Check for overflows or other unsafe conversions.
optionsCastOptions, default None
Additional checks pass by CastOptions
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) batch = pa.RecordBatch.from_pandas(df) batch.schema n_legs: int64 animals: string -- schema metadata -- pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...
Define new schema and cast batch values:
my_schema = pa.schema([ ... pa.field('n_legs', pa.duration('s')), ... pa.field('animals', pa.string())] ... ) batch.cast(target_schema=my_schema) pyarrow.RecordBatch n_legs: duration[s] animals: string
n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
column(self, i)#
Select single column from Table or RecordBatch.
Parameters:
The index or name of the column to retrieve.
Returns:
columnArray (for RecordBatch) or ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Select a column by numeric index:
table.column(0) <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 4, 5, 100 ] ]
Select a column by its name:
table.column("animals") <pyarrow.lib.ChunkedArray object at ...> [ [ "Flamingo", "Horse", "Brittle stars", "Centipede" ] ]
column_names#
Names of the Table or RecordBatch columns.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa table = pa.Table.from_arrays([[2, 4, 5, 100], ... ["Flamingo", "Horse", "Brittle stars", "Centipede"]], ... names=['n_legs', 'animals']) table.column_names ['n_legs', 'animals']
columns#
List of all columns in numerical order.
Returns:
columnslist of Array (for RecordBatch) or list of ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.columns [<pyarrow.lib.ChunkedArray object at ...> [ [ null, 4, 5, null ] ], <pyarrow.lib.ChunkedArray object at ...> [ [ "Flamingo", "Horse", null, "Centipede" ] ]]
copy_to(self, destination)#
Copy the entire RecordBatch to destination device.
This copies each column of the record batch to create a new record batch where all underlying buffers for the columns have been copied to the destination MemoryManager.
Parameters:
destinationpyarrow.MemoryManager or pyarrow.Device
The destination device to copy the array to.
Returns:
device_type#
The device type where the arrays in the RecordBatch reside.
Returns:
DeviceAllocationType
drop_columns(self, columns)#
Drop one or more columns and return a new Table or RecordBatch.
Parameters:
Field name(s) referencing existing column(s).
Returns:
Table or RecordBatch
A tabular object without the column(s).
Raises:
If any of the passed column names do not exist.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Drop one column:
table.drop_columns("animals") pyarrow.Table n_legs: int64
n_legs: [[2,4,5,100]]
Drop one or more columns:
table.drop_columns(["n_legs", "animals"]) pyarrow.Table ...
drop_null(self)#
Remove rows that contain missing values from a Table or RecordBatch.
See pyarrow.compute.drop_null() for full usage.
Returns:
Table or RecordBatch
A tabular object with the same schema, with rows containing no missing values.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [None, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.drop_null() pyarrow.Table year: double n_legs: int64 animals: string
year: [[2022,2021]] n_legs: [[4,100]] animals: [["Horse","Centipede"]]
equals(self, other, bool check_metadata=False)#
Check if contents of two record batches are equal.
Parameters:
otherpyarrow.RecordBatch
RecordBatch to compare against.
check_metadatabool, default False
Whether schema metadata equality should be checked as well.
Returns:
are_equalbool
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch_0 = pa.record_batch([]) batch_1 = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"], ... metadata={"n_legs": "Number of legs per animal"}) batch.equals(batch) True batch.equals(batch_0) False batch.equals(batch_1) True batch.equals(batch_1, check_metadata=True) False
field(self, i)#
Select a schema field by its column name or numeric index.
Parameters:
The index or name of the field to retrieve.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.field(0) pyarrow.Field<n_legs: int64> table.field(1) pyarrow.Field<animals: string>
filter(self, mask, null_selection_behavior='drop')#
Select rows from the table or record batch based on a boolean mask.
The Table can be filtered based on a mask, which will be passed topyarrow.compute.filter() to perform the filtering, or it can be filtered through a boolean Expression
Parameters:
maskArray or array-like or Expression
The boolean mask or the Expression to filter the table with.
null_selection_behaviorstr, default “drop”
How nulls in the mask should be handled, does nothing if an Expression is used.
Returns:
filteredTable or RecordBatch
A tabular object of the same schema, with only the rows selected by applied filtering
Examples
Using a Table (works similarly for RecordBatch):
import pyarrow as pa table = pa.table({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
Define an expression and select rows:
import pyarrow.compute as pc expr = pc.field("year") <= 2020 table.filter(expr) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2019]] n_legs: [[2,5]] animals: [["Flamingo","Brittle stars"]]
Define a mask and select rows:
mask=[True, True, False, None] table.filter(mask) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022]] n_legs: [[2,4]] animals: [["Flamingo","Horse"]] >>> table.filter(mask, null_selection_behavior='emit_null') pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022,null]] n_legs: [[2,4,null]] animals: [["Flamingo","Horse",null]]
static from_arrays(list arrays, names=None, schema=None, metadata=None)#
Construct a RecordBatch from multiple pyarrow.Arrays
Parameters:
arrayslist of pyarrow.Array
One for each field in RecordBatch
Names for the batch fields. If not passed, schema must be passed
Schema for the created batch. If not passed, names must be passed
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"]
Construct a RecordBatch from pyarrow Arrays using names:
pa.RecordBatch.from_arrays([n_legs, animals], names=names) pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,2,4,4,5,100] animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Construct a RecordBatch from pyarrow Arrays using schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
classmethod from_pandas(cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None)#
Convert pandas.DataFrame to an Arrow RecordBatch
Parameters:
schemapyarrow.Schema, optional
The expected schema of the RecordBatch. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
preserve_indexbool, optional
Whether to store the index as an additional column in the resultingRecordBatch. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Usepreserve_index=True to force it to be stored as a column.
If greater than 1, convert columns to Arrow in parallel using indicated number of threads. By default, this followspyarrow.cpu_count() (may use up to system CPU count threads).
columnslist, optional
List of column to be converted. If None, use all columns.
Returns:
Examples
import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2021, 2022], ... 'month': [3, 5, 7, 9], ... 'day': [1, 5, 9, 13], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
Convert pandas DataFrame to RecordBatch:
import pyarrow as pa pa.RecordBatch.from_pandas(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string
year: [2020,2022,2021,2022] month: [3,5,7,9] day: [1,5,9,13] n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
Convert pandas DataFrame to RecordBatch using schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.RecordBatch.from_pandas(df, schema=my_schema) pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
Convert pandas DataFrame to RecordBatch specifying columns:
pa.RecordBatch.from_pandas(df, columns=["n_legs"]) pyarrow.RecordBatch n_legs: int64
n_legs: [2,4,5,100]
classmethod from_pydict(cls, mapping, schema=None, metadata=None)#
Construct a Table or RecordBatch from Arrow arrays or columns.
Parameters:
mappingdict or Mapping
A mapping of strings to Arrays or Python lists.
If not passed, will be inferred from the Mapping values.
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Table or RecordBatch
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) pydict = {'n_legs': n_legs, 'animals': animals}
Construct a Table from a dictionary of arrays:
pa.Table.from_pydict(pydict) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
pa.Table.from_pydict(pydict).schema n_legs: int64 animals: string
Construct a Table from a dictionary of arrays with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_pydict(pydict, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from a dictionary of arrays with pyarrow schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.Table.from_pydict(pydict, schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
classmethod from_pylist(cls, mapping, schema=None, metadata=None)#
Construct a Table or RecordBatch from list of rows / dictionaries.
Parameters:
mappinglist of dicts of rows
A mapping of strings to row values.
If not passed, will be inferred from the first row of the mapping values.
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Table or RecordBatch
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, ... {'n_legs': 4, 'animals': 'Dog'}]
Construct a Table from a list of rows:
pa.Table.from_pylist(pylist) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4]] animals: [["Flamingo","Dog"]]
Construct a Table from a list of rows with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_pylist(pylist, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from a list of rows with pyarrow schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.Table.from_pylist(pylist, schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
static from_struct_array(StructArray struct_array)#
Construct a RecordBatch from a StructArray.
Each field in the StructArray will become a column in the resultingRecordBatch.
Parameters:
struct_arrayStructArray
Array to construct the record batch from.
Returns:
Examples
import pyarrow as pa struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'}, ... {'year': 2022, 'n_legs': 4}]) pa.RecordBatch.from_struct_array(struct).to_pandas() animals n_legs year 0 Parrot 2 NaN 1 None 4 2022.0
get_total_buffer_size(self)#
The sum of bytes in each buffer referenced by the record batch
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.get_total_buffer_size() 120
is_cpu#
Whether the RecordBatch’s arrays are CPU-accessible.
itercolumns(self)#
Iterator over all columns in their numerical order.
Yields:
Array (for RecordBatch) or ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) for i in table.itercolumns(): ... print(i.null_count) ... 2 1
nbytes#
Total number of bytes consumed by the elements of the record batch.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will only be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.nbytes 116
num_columns#
Number of columns
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.num_columns 2
num_rows#
Number of rows
Due to the definition of a RecordBatch, all columns have the same number of rows.
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.num_rows 6
remove_column(self, int i)#
Create new RecordBatch with the indicated column removed.
Parameters:
iint
Index of column to remove.
Returns:
New record batch without the column.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) batch = pa.RecordBatch.from_pandas(df) batch.remove_column(1) pyarrow.RecordBatch n_legs: int64
n_legs: [2,4,5,100]
rename_columns(self, names)#
Create new record batch with columns renamed to provided names.
Parameters:
nameslist[str] or dict[str, str]
List of new column names or mapping of old column names to new column names.
If a mapping of old to new column names is passed, then all columns which are found to match a provided old column name will be renamed to the new column name. If any column names are not found in the mapping, a KeyError will be raised.
Returns:
Raises:
If any of the column names passed in the names mapping do not exist.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) batch = pa.RecordBatch.from_pandas(df) new_names = ["n", "name"] batch.rename_columns(new_names) pyarrow.RecordBatch n: int64 name: string
n: [2,4,5,100] name: ["Flamingo","Horse","Brittle stars","Centipede"] >>> new_names = {"n_legs": "n", "animals": "name"} >>> batch.rename_columns(new_names) pyarrow.RecordBatch n: int64 name: string
n: [2,4,5,100] name: ["Flamingo","Horse","Brittle stars","Centipede"]
replace_schema_metadata(self, metadata=None)#
Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata
Parameters:
Returns:
shallow_copyRecordBatch
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100])
Constructing a RecordBatch with schema and metadata:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64())], ... metadata={"n_legs": "Number of legs per animal"}) batch = pa.RecordBatch.from_arrays([n_legs], schema=my_schema) batch.schema n_legs: int64 -- schema metadata -- n_legs: 'Number of legs per animal'
Shallow copy of a RecordBatch with deleted schema metadata:
batch.replace_schema_metadata().schema n_legs: int64
schema#
Schema of the RecordBatch and its columns
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.schema n_legs: int64 animals: string
select(self, columns)#
Select columns of the RecordBatch.
Returns a new RecordBatch with the specified columns, and metadata preserved.
Parameters:
columnslist-like
The column names or integer indices to select.
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"])
Select columns my indices:
batch.select([1]) pyarrow.RecordBatch animals: string
animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
Select columns by names:
batch.select(["n_legs"]) pyarrow.RecordBatch n_legs: int64
n_legs: [2,2,4,4,5,100]
serialize(self, memory_pool=None)#
Write RecordBatch to Buffer as encapsulated IPC message, which does not include a Schema.
To reconstruct a RecordBatch from the encapsulated IPC message Buffer returned by this function, a Schema must be passed separately. See Examples.
Parameters:
memory_poolMemoryPool, default None
Uses default memory pool if not specified
Returns:
serializedBuffer
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) buf = batch.serialize() buf <pyarrow.Buffer address=0x... size=... is_cpu=True is_mutable=True>
Reconstruct RecordBatch from IPC message Buffer and original Schema
pa.ipc.read_record_batch(buf, batch.schema) pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,2,4,4,5,100] animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
set_column(self, int i, field_, column)#
Replace column in RecordBatch at position.
Parameters:
iint
Index to place the column at.
If a string is passed then the type is deduced from the column data.
columnArray or value coercible to array
Column data.
Returns:
New record batch with the passed column set.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) batch = pa.RecordBatch.from_pandas(df)
Replace a column:
year = [2021, 2022, 2019, 2021] batch.set_column(1,'year', year) pyarrow.RecordBatch n_legs: int64 year: int64
n_legs: [2,4,5,100] year: [2021,2022,2019,2021]
shape#
Dimensions of the table or record batch: (#rows, #columns).
Returns:
Number of rows and number of columns.
Examples
import pyarrow as pa table = pa.table({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table.shape (4, 2)
slice(self, offset=0, length=None)#
Compute zero-copy slice of this RecordBatch
Parameters:
offsetint, default 0
Offset from start of record batch to slice
Length of slice (default is until end of batch starting from offset)
Returns:
slicedRecordBatch
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) batch.to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede batch.slice(offset=3).to_pandas() n_legs animals 0 4 Horse 1 5 Brittle stars 2 100 Centipede batch.slice(length=2).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot batch.slice(offset=3, length=1).to_pandas() n_legs animals 0 4 Horse
sort_by(self, sorting, **kwargs)#
Sort the Table or RecordBatch by one or multiple columns.
Parameters:
sortingstr or list[tuple(name, order)]
Name of the column to use to sort (ascending), or a list of multiple sorting conditions where each entry is a tuple with column name and sorting order (“ascending” or “descending”)
**kwargsdict, optional
Additional sorting options. As allowed by SortOptions
Returns:
Table or RecordBatch
A new tabular object sorted according to the sort keys.
Examples
Table (works similarly for RecordBatch)
import pandas as pd import pyarrow as pa df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021], ... 'n_legs': [2, 2, 4, 4, 5, 100], ... 'animal': ["Flamingo", "Parrot", "Dog", "Horse", ... "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.sort_by('animal') pyarrow.Table year: int64 n_legs: int64 animal: string
year: [[2019,2021,2021,2020,2022,2022]] n_legs: [[5,100,4,2,4,2]] animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]
take(self, indices)#
Select rows from a Table or RecordBatch.
See pyarrow.compute.take() for full usage.
Parameters:
indicesArray or array-like
The indices in the tabular object whose rows will be returned.
Returns:
Table or RecordBatch
A tabular object with the same schema, containing the taken rows.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.take([1,3]) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2022,2021]] n_legs: [[4,100]] animals: [["Horse","Centipede"]]
to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, unicode maps_as_pydicts=None, types_mapper=None, bool coerce_temporal_nanoseconds=False)#
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
Parameters:
memory_poolMemoryPool, default None
Arrow MemoryPool to use for allocations. Uses the default memory pool if not passed.
categorieslist, default empty
List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
strings_to_categoricalbool, default False
Encode string (UTF8) and binary types to pandas.Categorical.
zero_copy_onlybool, default False
Raise an ArrowException if this function call would require copying the underlying data.
integer_object_nullsbool, default False
Cast integers with nulls to objects
date_as_objectbool, default True
Cast dates to objects. If False, convert to datetime64 dtype with the equivalent time unit (if supported). Note: in pandas version < 2.0, only datetime64[ns] conversion is supported.
timestamp_as_objectbool, default False
Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful in pandas version 1.x if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). Non-nanosecond timestamps are supported in pandas version 2.0. If False, all timestamps are converted to datetime64 dtype.
Whether to parallelize the conversion using multiple threads.
deduplicate_objectsbool, default True
Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
ignore_metadatabool, default False
If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
split_blocksbool, default False
If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
self_destructbool, default False
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts. This can change the ordering of (key, value) pairs, and will deduplicate multiple keys, resulting in a possible loss of data.
If ‘lossy’, this key deduplication results in a warning printed when detected. If ‘strict’, this instead results in an exception being raised when detected.
types_mapperfunction, default None
A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.
coerce_temporal_nanosecondsbool, default False
Only applicable to pandas version >= 2.0. A legacy option to coerce date32, date64, duration, and timestamp time units to nanoseconds when converting to pandas. This is the default behavior in pandas version 1.x. Set this option to True if you’d like to use this coercion when using pandas version >= 2.0 for backwards compatibility (not recommended otherwise).
Returns:
pandas.Series or pandas.DataFrame depending on type of object
Examples
import pyarrow as pa import pandas as pd
Convert a Table to pandas DataFrame:
table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) batch pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
import pyarrow as pa n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 isinstance(n_legs.to_pandas(), pd.Series) True
to_pydict(self, *, maps_as_pydicts=None)#
Convert the Table or RecordBatch to a dict or OrderedDict.
Parameters:
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.
If ‘lossy’, whenever duplicate keys are detected, a warning will be printed. The last seen value of a duplicate key will be in the Python dictionary. If ‘strict’, this instead results in an exception being raised when detected.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) table = pa.Table.from_arrays([n_legs, animals], names=["n_legs", "animals"]) table.to_pydict() {'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}
to_pylist(self, *, maps_as_pydicts=None)#
Convert the Table or RecordBatch to a list of rows / dictionaries.
Parameters:
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.
If ‘lossy’, whenever duplicate keys are detected, a warning will be printed. The last seen value of a duplicate key will be in the Python dictionary. If ‘strict’, this instead results in an exception being raised when detected.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa data = [[2, 4, 5, 100], ... ["Flamingo", "Horse", "Brittle stars", "Centipede"]] table = pa.table(data, names=["n_legs", "animals"]) table.to_pylist() [{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...
to_string(self, *, show_metadata=False, preview_cols=0)#
Return human-readable string representation of Table or RecordBatch.
Parameters:
show_metadatabool, default False
Display Field-level and Schema-level KeyValueMetadata.
preview_colsint, default 0
Display values of the columns for the first N columns.
Returns:
to_struct_array(self)#
Convert to a struct array.
to_tensor(self, bool null_to_nan=False, bool row_major=True, MemoryPool memory_pool=None)#
Convert to a Tensor.
RecordBatches that can be converted have fields of type signed or unsigned integer or float, including all bit-widths.
null_to_nan is False by default and this method will raise an error in case any nulls are present. RecordBatches with nulls can be converted with null_to_nanset to True. In this case null values are converted to NaN and integer type arrays are promoted to the appropriate float type.
Parameters:
null_to_nanbool, default False
Whether to write null values in the result as NaN.
Whether resulting Tensor is row-major or column-major
memory_poolMemoryPool, default None
For memory allocations, if required, otherwise use default pool
Examples
import pyarrow as pa batch = pa.record_batch( ... [ ... pa.array([1, 2, 3, 4, None], type=pa.int32()), ... pa.array([10, 20, 30, 40, None], type=pa.float32()), ... ], names = ["a", "b"] ... )
batch pyarrow.RecordBatch a: int32 b: float
a: [1,2,3,4,null] b: [10,20,30,40,null]
Convert a RecordBatch to row-major Tensor with null values written as ``NaN``s
batch.to_tensor(null_to_nan=True) <pyarrow.Tensor> type: double shape: (5, 2) strides: (16, 8) batch.to_tensor(null_to_nan=True).to_numpy() array([[ 1., 10.], [ 2., 20.], [ 3., 30.], [ 4., 40.], [nan, nan]])
Convert a RecordBatch to column-major Tensor
batch.to_tensor(null_to_nan=True, row_major=False) <pyarrow.Tensor> type: double shape: (5, 2) strides: (8, 40) batch.to_tensor(null_to_nan=True, row_major=False).to_numpy() array([[ 1., 10.], [ 2., 20.], [ 3., 30.], [ 4., 40.], [nan, nan]])
validate(self, *, full=False)#
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=Truefor thorough validation checks (potentially O(n)).
Parameters:
If True, run expensive checks, otherwise cheap checks only.
Raises:
ArrowInvalid