pyarrow.Table — Apache Arrow v20.0.0 (original) (raw)
class pyarrow.Table#
Bases: _Tabular
A collection of top-level named, equal length Arrow arrays.
Warning
Do not call this class’s constructor directly, use one of the from_*methods instead.
Examples
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"]
Construct a Table from arrays:
pa.Table.from_arrays([n_legs, animals], names=names) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Construct a Table from a RecordBatch:
batch = pa.record_batch([n_legs, animals], names=names) pa.Table.from_batches([batch]) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Construct a Table from pandas DataFrame:
import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) pa.Table.from_pandas(df) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022,2019,2021]] n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Construct a Table from a dictionary of arrays:
pydict = {'n_legs': n_legs, 'animals': animals} pa.Table.from_pydict(pydict) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
pa.Table.from_pydict(pydict).schema n_legs: int64 animals: string
Construct a Table from a dictionary of arrays with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_pydict(pydict, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from a list of rows:
pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, {'year': 2021, 'animals': 'Centipede'}] pa.Table.from_pylist(pylist) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,null]] animals: [["Flamingo","Centipede"]]
Construct a Table from a list of rows with pyarrow schema:
my_schema = pa.schema([ ... pa.field('year', pa.int64()), ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"year": "Year of entry"}) pa.Table.from_pylist(pylist, schema=my_schema).schema year: int64 n_legs: int64 animals: string -- schema metadata -- year: 'Year of entry'
Construct a Table with pyarrow.table():
pa.table([n_legs, animals], names=names) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
__init__(*args, **kwargs)#
Methods
Attributes
__dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True)#
Return the dataframe interchange object implementing the interchange protocol.
Parameters:
nan_as_nullbool, default False
Whether to tell the DataFrame to overwrite null values in the data with NaN (or NaT).
Whether to allow memory copying when exporting. If set to False it would cause non-zero-copy exports to fail.
Returns:
DataFrame interchange object
The object which consuming library can use to ingress the dataframe.
Notes
Details on the interchange protocol:https://data-apis.org/dataframe-protocol/latest/index.html nan_as_null currently has no effect; once support for nullable extension dtypes is added, this value should be propagated to columns.
add_column(self, int i, field_, column)#
Add column to Table at position.
A new table is returned with the column added, the original table object is left unchanged.
Parameters:
iint
Index to place the column at.
If a string is passed then the type is deduced from the column data.
columnArray, list of Array, or values coercible to arrays
Column data.
Returns:
New table with the passed column added.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Add column:
year = [2021, 2022, 2019, 2021] table.add_column(0,"year", [year]) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2021,2022,2019,2021]] n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Original table is left unchanged:
table pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
append_column(self, field_, column)#
Append column at end of columns.
Parameters:
If a string is passed then the type is deduced from the column data.
columnArray or value coercible to array
Column data.
Returns:
Table or RecordBatch
New table or record batch with the passed column added.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Append column at the end:
year = [2021, 2022, 2019, 2021] table.append_column('year', [year]) pyarrow.Table n_legs: int64 animals: string year: int64
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]] year: [[2021,2022,2019,2021]]
cast(self, Schema target_schema, safe=None, options=None)#
Cast table values to another schema.
Parameters:
target_schemaSchema
Schema to cast to, the names and order of fields must match.
Check for overflows or other unsafe conversions.
optionsCastOptions, default None
Additional checks pass by CastOptions
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.schema n_legs: int64 animals: string -- schema metadata -- pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...
Define new schema and cast table values:
my_schema = pa.schema([ ... pa.field('n_legs', pa.duration('s')), ... pa.field('animals', pa.string())] ... ) table.cast(target_schema=my_schema) pyarrow.Table n_legs: duration[s] animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
column(self, i)#
Select single column from Table or RecordBatch.
Parameters:
The index or name of the column to retrieve.
Returns:
columnArray (for RecordBatch) or ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Select a column by numeric index:
table.column(0) <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 4, 5, 100 ] ]
Select a column by its name:
table.column("animals") <pyarrow.lib.ChunkedArray object at ...> [ [ "Flamingo", "Horse", "Brittle stars", "Centipede" ] ]
column_names#
Names of the Table or RecordBatch columns.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa table = pa.Table.from_arrays([[2, 4, 5, 100], ... ["Flamingo", "Horse", "Brittle stars", "Centipede"]], ... names=['n_legs', 'animals']) table.column_names ['n_legs', 'animals']
columns#
List of all columns in numerical order.
Returns:
columnslist of Array (for RecordBatch) or list of ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.columns [<pyarrow.lib.ChunkedArray object at ...> [ [ null, 4, 5, null ] ], <pyarrow.lib.ChunkedArray object at ...> [ [ "Flamingo", "Horse", null, "Centipede" ] ]]
combine_chunks(self, MemoryPool memory_pool=None)#
Make a new table by combining the chunks this table has.
All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.
Parameters:
memory_poolMemoryPool, default None
For memory allocations, if required, otherwise use default pool.
Returns:
Examples
import pyarrow as pa n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) animals = pa.chunked_array([["Flamingo", "Parrot", "Dog"], ["Horse", "Brittle stars", "Centipede"]]) names = ["n_legs", "animals"] table = pa.table([n_legs, animals], names=names) table pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,2,4],[4,5,100]] animals: [["Flamingo","Parrot","Dog"],["Horse","Brittle stars","Centipede"]] >>> table.combine_chunks() pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,2,4,4,5,100]] animals: [["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]]
drop(self, columns)#
Drop one or more columns and return a new table.
Alias of Table.drop_columns, but kept for backwards compatibility.
Parameters:
Field name(s) referencing existing column(s).
Returns:
New table without the column(s).
drop_columns(self, columns)#
Drop one or more columns and return a new Table or RecordBatch.
Parameters:
Field name(s) referencing existing column(s).
Returns:
Table or RecordBatch
A tabular object without the column(s).
Raises:
If any of the passed column names do not exist.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Drop one column:
table.drop_columns("animals") pyarrow.Table n_legs: int64
n_legs: [[2,4,5,100]]
Drop one or more columns:
table.drop_columns(["n_legs", "animals"]) pyarrow.Table ...
drop_null(self)#
Remove rows that contain missing values from a Table or RecordBatch.
See pyarrow.compute.drop_null() for full usage.
Returns:
Table or RecordBatch
A tabular object with the same schema, with rows containing no missing values.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [None, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.drop_null() pyarrow.Table year: double n_legs: int64 animals: string
year: [[2022,2021]] n_legs: [[4,100]] animals: [["Horse","Centipede"]]
equals(self, Table other, bool check_metadata=False)#
Check if contents of two tables are equal.
Parameters:
otherpyarrow.Table
Table to compare against.
check_metadatabool, default False
Whether schema metadata equality should be checked as well.
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) names=["n_legs", "animals"] table = pa.Table.from_arrays([n_legs, animals], names=names) table_0 = pa.Table.from_arrays([]) table_1 = pa.Table.from_arrays([n_legs, animals], ... names=names, ... metadata={"n_legs": "Number of legs per animal"}) table.equals(table) True table.equals(table_0) False table.equals(table_1) True table.equals(table_1, check_metadata=True) False
field(self, i)#
Select a schema field by its column name or numeric index.
Parameters:
The index or name of the field to retrieve.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.field(0) pyarrow.Field<n_legs: int64> table.field(1) pyarrow.Field<animals: string>
filter(self, mask, null_selection_behavior='drop')#
Select rows from the table or record batch based on a boolean mask.
The Table can be filtered based on a mask, which will be passed topyarrow.compute.filter() to perform the filtering, or it can be filtered through a boolean Expression
Parameters:
maskArray or array-like or Expression
The boolean mask or the Expression to filter the table with.
null_selection_behaviorstr, default “drop”
How nulls in the mask should be handled, does nothing if an Expression is used.
Returns:
filteredTable or RecordBatch
A tabular object of the same schema, with only the rows selected by applied filtering
Examples
Using a Table (works similarly for RecordBatch):
import pyarrow as pa table = pa.table({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
Define an expression and select rows:
import pyarrow.compute as pc expr = pc.field("year") <= 2020 table.filter(expr) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2019]] n_legs: [[2,5]] animals: [["Flamingo","Brittle stars"]]
Define a mask and select rows:
mask=[True, True, False, None] table.filter(mask) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022]] n_legs: [[2,4]] animals: [["Flamingo","Horse"]] >>> table.filter(mask, null_selection_behavior='emit_null') pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022,null]] n_legs: [[2,4,null]] animals: [["Flamingo","Horse",null]]
flatten(self, MemoryPool memory_pool=None)#
Flatten this Table.
Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.
Parameters:
memory_poolMemoryPool, default None
For memory allocations, if required, otherwise use default pool
Returns:
Examples
import pyarrow as pa struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'}, ... {'year': 2022, 'n_legs': 4}]) month = pa.array([4, 6]) table = pa.Table.from_arrays([struct,month], ... names = ["a", "month"]) table pyarrow.Table a: struct<animals: string, n_legs: int64, year: int64> child 0, animals: string child 1, n_legs: int64 child 2, year: int64 month: int64
a: [ -- is_valid: all not null -- child 0 type: string ["Parrot",null] -- child 1 type: int64 [2,4] -- child 2 type: int64 [null,2022]] month: [[4,6]]
Flatten the columns with struct field:
table.flatten() pyarrow.Table a.animals: string a.n_legs: int64 a.year: int64 month: int64
a.animals: [["Parrot",null]] a.n_legs: [[2,4]] a.year: [[null,2022]] month: [[4,6]]
static from_arrays(arrays, names=None, schema=None, metadata=None)#
Construct a Table from Arrow arrays.
Parameters:
arrayslist of pyarrow.Array or pyarrow.ChunkedArray
Equal-length arrays that should form the table.
Names for the table columns. If not passed, schema must be passed.
Schema for the created table. If not passed, names must be passed.
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"]
Construct a Table from arrays:
pa.Table.from_arrays([n_legs, animals], names=names) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Construct a Table from arrays with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_arrays([n_legs, animals], ... names=names, ... metadata=my_metadata) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
pa.Table.from_arrays([n_legs, animals], ... names=names, ... metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from arrays with pyarrow schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"animals": "Name of the animal species"}) pa.Table.from_arrays([n_legs, animals], ... schema=my_schema) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
pa.Table.from_arrays([n_legs, animals], ... schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- animals: 'Name of the animal species'
static from_batches(batches, Schema schema=None)#
Construct a Table from a sequence or iterator of Arrow RecordBatches.
Parameters:
batchessequence or iterator of RecordBatch
Sequence of RecordBatch to be converted, all schemas must be equal.
If not passed, will be inferred from the first RecordBatch.
Returns:
Examples
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"] batch = pa.record_batch([n_legs, animals], names=names) batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede
Construct a Table from a RecordBatch:
pa.Table.from_batches([batch]) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
Construct a Table from a sequence of RecordBatches:
pa.Table.from_batches([batch, batch]) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100],[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"],["Flamingo","Horse","Brittle stars","Centipede"]]
classmethod from_pandas(cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None, bool safe=True)#
Convert pandas.DataFrame to an Arrow Table.
The column types in the resulting Arrow Table are inferred from the dtypes of the pandas.Series in the DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the case of object, we need to guess the datatype by looking at the Python objects in this Series.
Be aware that Series of the object dtype don’t carry enough information to always lead to a meaningful Arrow type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only contains None/nan objects, the type is set to null. This behavior can be avoided by constructing an explicit schema and passing it to this function.
Parameters:
schemapyarrow.Schema, optional
The expected schema of the Arrow Table. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
preserve_indexbool, optional
Whether to store the index as an additional column in the resultingTable. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Usepreserve_index=True to force it to be stored as a column.
If greater than 1, convert columns to Arrow in parallel using indicated number of threads. By default, this followspyarrow.cpu_count() (may use up to system CPU count threads).
columnslist, optional
List of column to be converted. If None, use all columns.
Check for overflows or other unsafe conversions.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) pa.Table.from_pandas(df) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
classmethod from_pydict(cls, mapping, schema=None, metadata=None)#
Construct a Table or RecordBatch from Arrow arrays or columns.
Parameters:
mappingdict or Mapping
A mapping of strings to Arrays or Python lists.
If not passed, will be inferred from the Mapping values.
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Table or RecordBatch
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) pydict = {'n_legs': n_legs, 'animals': animals}
Construct a Table from a dictionary of arrays:
pa.Table.from_pydict(pydict) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
pa.Table.from_pydict(pydict).schema n_legs: int64 animals: string
Construct a Table from a dictionary of arrays with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_pydict(pydict, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from a dictionary of arrays with pyarrow schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.Table.from_pydict(pydict, schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
classmethod from_pylist(cls, mapping, schema=None, metadata=None)#
Construct a Table or RecordBatch from list of rows / dictionaries.
Parameters:
mappinglist of dicts of rows
A mapping of strings to row values.
If not passed, will be inferred from the first row of the mapping values.
metadatadict or Mapping, default None
Optional metadata for the schema (if inferred).
Returns:
Table or RecordBatch
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, ... {'n_legs': 4, 'animals': 'Dog'}]
Construct a Table from a list of rows:
pa.Table.from_pylist(pylist) pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4]] animals: [["Flamingo","Dog"]]
Construct a Table from a list of rows with metadata:
my_metadata={"n_legs": "Number of legs per animal"} pa.Table.from_pylist(pylist, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
Construct a Table from a list of rows with pyarrow schema:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) pa.Table.from_pylist(pylist, schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
static from_struct_array(struct_array)#
Construct a Table from a StructArray.
Each field in the StructArray will become a column in the resultingTable.
Parameters:
struct_arrayStructArray or ChunkedArray
Array to construct the table from.
Returns:
Examples
import pyarrow as pa struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'}, ... {'year': 2022, 'n_legs': 4}]) pa.Table.from_struct_array(struct).to_pandas() animals n_legs year 0 Parrot 2 NaN 1 None 4 2022.0
get_total_buffer_size(self)#
The sum of bytes in each buffer referenced by the table.
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.get_total_buffer_size() 76
group_by(self, keys, use_threads=True)#
Declare a grouping over the columns of the table.
Resulting grouping can then be used to perform aggregations with a subsequent aggregate() method.
Parameters:
Name of the columns that should be used as the grouping key.
Whether to use multithreading or not. When set to True (the default), no stable ordering of the output is guaranteed.
Returns:
Examples
import pandas as pd import pyarrow as pa df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021], ... 'n_legs': [2, 2, 4, 4, 5, 100], ... 'animal': ["Flamingo", "Parrot", "Dog", "Horse", ... "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.group_by('year').aggregate([('n_legs', 'sum')]) pyarrow.Table year: int64 n_legs_sum: int64
year: [[2020,2022,2021,2019]] n_legs_sum: [[2,6,104,5]]
is_cpu#
Whether all ChunkedArrays are CPU-accessible.
itercolumns(self)#
Iterator over all columns in their numerical order.
Yields:
Array (for RecordBatch) or ChunkedArray (for Table)
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) for i in table.itercolumns(): ... print(i.null_count) ... 2 1
join(self, right_table, keys, right_keys=None, join_type='left outer', left_suffix=None, right_suffix=None, coalesce_keys=True, use_threads=True)#
Perform a join between this table and another one.
Result of the join will be a new Table, where further operations can be applied.
Parameters:
right_tableTable
The table to join to the current one, acting as the right table in the join operation.
The columns from current table that should be used as keys of the join operation left side.
right_keysstr or list[str], default None
The columns from the right_table that should be used as keys on the join operation right side. When None use the same key names as the left table.
join_typestr, default “left outer”
The kind of join that should be performed, one of (“left semi”, “right semi”, “left anti”, “right anti”, “inner”, “left outer”, “right outer”, “full outer”)
Which suffix to add to left column names. This prevents confusion when the columns in left and right tables have colliding names.
Which suffix to add to the right column names. This prevents confusion when the columns in left and right tables have colliding names.
coalesce_keysbool, default True
If the duplicated keys should be omitted from one of the sides in the join result.
Whether to use multithreading or not.
Returns:
Examples
import pandas as pd import pyarrow as pa df1 = pd.DataFrame({'id': [1, 2, 3], ... 'year': [2020, 2022, 2019]}) df2 = pd.DataFrame({'id': [3, 4], ... 'n_legs': [5, 100], ... 'animal': ["Brittle stars", "Centipede"]}) t1 = pa.Table.from_pandas(df1) t2 = pa.Table.from_pandas(df2)
Left outer join:
t1.join(t2, 'id').combine_chunks().sort_by('year') pyarrow.Table id: int64 year: int64 n_legs: int64 animal: string
id: [[3,1,2]] year: [[2019,2020,2022]] n_legs: [[5,null,null]] animal: [["Brittle stars",null,null]]
Full outer join:
t1.join(t2, 'id', join_type="full outer").combine_chunks().sort_by('year') pyarrow.Table id: int64 year: int64 n_legs: int64 animal: string
id: [[3,1,2,4]] year: [[2019,2020,2022,null]] n_legs: [[5,null,null,100]] animal: [["Brittle stars",null,null,"Centipede"]]
Right outer join:
t1.join(t2, 'id', join_type="right outer").combine_chunks().sort_by('year') pyarrow.Table year: int64 id: int64 n_legs: int64 animal: string
year: [[2019,null]] id: [[3,4]] n_legs: [[5,100]] animal: [["Brittle stars","Centipede"]]
Right anti join
t1.join(t2, 'id', join_type="right anti") pyarrow.Table id: int64 n_legs: int64 animal: string
id: [[4]] n_legs: [[100]] animal: [["Centipede"]]
join_asof(self, right_table, on, by, tolerance, right_on=None, right_by=None)#
Perform an asof join between this table and another one.
This is similar to a left-join except that we match on nearest key rather than equal keys. Both tables must be sorted by the key. This type of join is most useful for time series data that are not perfectly aligned.
Optionally match on equivalent keys with “by” before searching with “on”.
Result of the join will be a new Table, where further operations can be applied.
Parameters:
right_tableTable
The table to join to the current one, acting as the right table in the join operation.
onstr
The column from current table that should be used as the “on” key of the join operation left side.
An inexact match is used on the “on” key, i.e. a row is considered a match if and only if left_on - tolerance <= right_on <= left_on.
The input dataset must be sorted by the “on” key. Must be a single field of a common type.
Currently, the “on” key must be an integer, date, or timestamp type.
The columns from current table that should be used as the keys of the join operation left side. The join operation is then done only for the matches in these columns.
toleranceint
The tolerance for inexact “on” key matching. A right row is considered a match with the left row right.on - left.on <= tolerance. Thetolerance may be:
- negative, in which case a past-as-of-join occurs;
- or positive, in which case a future-as-of-join occurs;
- or zero, in which case an exact-as-of-join occurs.
The tolerance is interpreted in the same units as the “on” key.
right_onstr or list[str], default None
The columns from the right_table that should be used as the on key on the join operation right side. When None use the same key name as the left table.
right_bystr or list[str], default None
The columns from the right_table that should be used as keys on the join operation right side. When None use the same key names as the left table.
Returns:
nbytes#
Total number of bytes consumed by the elements of the table.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will only be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.nbytes 72
num_columns#
Number of columns in this table.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.num_columns 2
num_rows#
Number of rows in this table.
Due to the definition of a table, all columns have the same number of rows.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table = pa.Table.from_pandas(df) table.num_rows 4
remove_column(self, int i)#
Create new Table with the indicated column removed.
Parameters:
iint
Index of column to remove.
Returns:
New table without the column.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.remove_column(1) pyarrow.Table n_legs: int64
n_legs: [[2,4,5,100]]
rename_columns(self, names)#
Create new table with columns renamed to provided names.
Parameters:
nameslist[str] or dict[str, str]
List of new column names or mapping of old column names to new column names.
If a mapping of old to new column names is passed, then all columns which are found to match a provided old column name will be renamed to the new column name. If any column names are not found in the mapping, a KeyError will be raised.
Returns:
Raises:
If any of the column names passed in the names mapping do not exist.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) new_names = ["n", "name"] table.rename_columns(new_names) pyarrow.Table n: int64 name: string
n: [[2,4,5,100]] name: [["Flamingo","Horse","Brittle stars","Centipede"]] >>> new_names = {"n_legs": "n", "animals": "name"} >>> table.rename_columns(new_names) pyarrow.Table n: int64 name: string
n: [[2,4,5,100]] name: [["Flamingo","Horse","Brittle stars","Centipede"]]
replace_schema_metadata(self, metadata=None)#
Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.
Parameters:
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Constructing a Table with pyarrow schema and metadata:
my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) table= pa.table(df, my_schema) table.schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal' pandas: ...
Create a shallow copy of a Table with deleted schema metadata:
table.replace_schema_metadata().schema n_legs: int64 animals: string
Create a shallow copy of a Table with new schema metadata:
metadata={"animals": "Which animal"} table.replace_schema_metadata(metadata = metadata).schema n_legs: int64 animals: string -- schema metadata -- animals: 'Which animal'
schema#
Schema of the table and its columns.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.schema n_legs: int64 animals: string -- schema metadata -- pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' ...
select(self, columns)#
Select columns of the Table.
Returns a new Table with the specified columns, and metadata preserved.
Parameters:
columnslist-like
The column names or integer indices to select.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.select([0,1]) pyarrow.Table year: int64 n_legs: int64
year: [[2020,2022,2019,2021]] n_legs: [[2,4,5,100]] >>> table.select(["year"]) pyarrow.Table year: int64
year: [[2020,2022,2019,2021]]
set_column(self, int i, field_, column)#
Replace column in Table at position.
Parameters:
iint
Index to place the column at.
If a string is passed then the type is deduced from the column data.
columnArray, list of Array, or values coercible to arrays
Column data.
Returns:
New table with the passed column set.
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Replace a column:
year = [2021, 2022, 2019, 2021] table.set_column(1,'year', [year]) pyarrow.Table n_legs: int64 year: int64
n_legs: [[2,4,5,100]] year: [[2021,2022,2019,2021]]
shape#
Dimensions of the table or record batch: (#rows, #columns).
Returns:
Number of rows and number of columns.
Examples
import pyarrow as pa table = pa.table({'n_legs': [None, 4, 5, None], ... 'animals': ["Flamingo", "Horse", None, "Centipede"]}) table.shape (4, 2)
slice(self, offset=0, length=None)#
Compute zero-copy slice of this Table.
Parameters:
offsetint, default 0
Offset from start of table to slice.
Length of slice (default is until end of table starting from offset).
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.slice(length=3) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2020,2022,2019]] n_legs: [[2,4,5]] animals: [["Flamingo","Horse","Brittle stars"]] >>> table.slice(offset=2) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2019,2021]] n_legs: [[5,100]] animals: [["Brittle stars","Centipede"]] >>> table.slice(offset=2, length=1) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2019]] n_legs: [[5]] animals: [["Brittle stars"]]
sort_by(self, sorting, **kwargs)#
Sort the Table or RecordBatch by one or multiple columns.
Parameters:
sortingstr or list[tuple(name, order)]
Name of the column to use to sort (ascending), or a list of multiple sorting conditions where each entry is a tuple with column name and sorting order (“ascending” or “descending”)
**kwargsdict, optional
Additional sorting options. As allowed by SortOptions
Returns:
Table or RecordBatch
A new tabular object sorted according to the sort keys.
Examples
Table (works similarly for RecordBatch)
import pandas as pd import pyarrow as pa df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021], ... 'n_legs': [2, 2, 4, 4, 5, 100], ... 'animal': ["Flamingo", "Parrot", "Dog", "Horse", ... "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.sort_by('animal') pyarrow.Table year: int64 n_legs: int64 animal: string
year: [[2019,2021,2021,2020,2022,2022]] n_legs: [[5,100,4,2,4,2]] animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]
take(self, indices)#
Select rows from a Table or RecordBatch.
See pyarrow.compute.take() for full usage.
Parameters:
indicesArray or array-like
The indices in the tabular object whose rows will be returned.
Returns:
Table or RecordBatch
A tabular object with the same schema, containing the taken rows.
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa import pandas as pd df = pd.DataFrame({'year': [2020, 2022, 2019, 2021], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df) table.take([1,3]) pyarrow.Table year: int64 n_legs: int64 animals: string
year: [[2022,2021]] n_legs: [[4,100]] animals: [["Horse","Centipede"]]
to_batches(self, max_chunksize=None)#
Convert Table to a list of RecordBatch objects.
Note that this method is zero-copy, it merely exposes the same data under a different API.
Parameters:
max_chunksizeint, default None
Maximum number of rows for each RecordBatch chunk. Individual chunks may be smaller depending on the chunk layout of individual columns.
Returns:
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Convert a Table to a RecordBatch:
table.to_batches()[0].to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede
Convert a Table to a list of RecordBatches:
table.to_batches(max_chunksize=2)[0].to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse table.to_batches(max_chunksize=2)[1].to_pandas() n_legs animals 0 5 Brittle stars 1 100 Centipede
to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, unicode maps_as_pydicts=None, types_mapper=None, bool coerce_temporal_nanoseconds=False)#
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
Parameters:
memory_poolMemoryPool, default None
Arrow MemoryPool to use for allocations. Uses the default memory pool if not passed.
categorieslist, default empty
List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
strings_to_categoricalbool, default False
Encode string (UTF8) and binary types to pandas.Categorical.
zero_copy_onlybool, default False
Raise an ArrowException if this function call would require copying the underlying data.
integer_object_nullsbool, default False
Cast integers with nulls to objects
date_as_objectbool, default True
Cast dates to objects. If False, convert to datetime64 dtype with the equivalent time unit (if supported). Note: in pandas version < 2.0, only datetime64[ns] conversion is supported.
timestamp_as_objectbool, default False
Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful in pandas version 1.x if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). Non-nanosecond timestamps are supported in pandas version 2.0. If False, all timestamps are converted to datetime64 dtype.
Whether to parallelize the conversion using multiple threads.
deduplicate_objectsbool, default True
Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
ignore_metadatabool, default False
If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
split_blocksbool, default False
If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
self_destructbool, default False
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts. This can change the ordering of (key, value) pairs, and will deduplicate multiple keys, resulting in a possible loss of data.
If ‘lossy’, this key deduplication results in a warning printed when detected. If ‘strict’, this instead results in an exception being raised when detected.
types_mapperfunction, default None
A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.
coerce_temporal_nanosecondsbool, default False
Only applicable to pandas version >= 2.0. A legacy option to coerce date32, date64, duration, and timestamp time units to nanoseconds when converting to pandas. This is the default behavior in pandas version 1.x. Set this option to True if you’d like to use this coercion when using pandas version >= 2.0 for backwards compatibility (not recommended otherwise).
Returns:
pandas.Series or pandas.DataFrame depending on type of object
Examples
import pyarrow as pa import pandas as pd
Convert a Table to pandas DataFrame:
table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) batch pyarrow.RecordBatch n_legs: int64 animals: string
n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"]
batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
import pyarrow as pa n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 isinstance(n_legs.to_pandas(), pd.Series) True
to_pydict(self, *, maps_as_pydicts=None)#
Convert the Table or RecordBatch to a dict or OrderedDict.
Parameters:
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.
If ‘lossy’, whenever duplicate keys are detected, a warning will be printed. The last seen value of a duplicate key will be in the Python dictionary. If ‘strict’, this instead results in an exception being raised when detected.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa n_legs = pa.array([2, 2, 4, 4, 5, 100]) animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) table = pa.Table.from_arrays([n_legs, animals], names=["n_legs", "animals"]) table.to_pydict() {'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}
to_pylist(self, *, maps_as_pydicts=None)#
Convert the Table or RecordBatch to a list of rows / dictionaries.
Parameters:
maps_as_pydictsstr, optional, default None
Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].
If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.
If ‘lossy’, whenever duplicate keys are detected, a warning will be printed. The last seen value of a duplicate key will be in the Python dictionary. If ‘strict’, this instead results in an exception being raised when detected.
Returns:
Examples
Table (works similarly for RecordBatch)
import pyarrow as pa data = [[2, 4, 5, 100], ... ["Flamingo", "Horse", "Brittle stars", "Centipede"]] table = pa.table(data, names=["n_legs", "animals"]) table.to_pylist() [{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...
to_reader(self, max_chunksize=None)#
Convert the Table to a RecordBatchReader.
Note that this method is zero-copy, it merely exposes the same data under a different API.
Parameters:
max_chunksizeint, default None
Maximum number of rows for each RecordBatch chunk. Individual chunks may be smaller depending on the chunk layout of individual columns.
Returns:
RecordBatchReader
Examples
import pyarrow as pa import pandas as pd df = pd.DataFrame({'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) table = pa.Table.from_pandas(df)
Convert a Table to a RecordBatchReader:
table.to_reader() <pyarrow.lib.RecordBatchReader object at ...>
reader = table.to_reader() reader.schema n_legs: int64 animals: string -- schema metadata -- pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ... reader.read_all() pyarrow.Table n_legs: int64 animals: string
n_legs: [[2,4,5,100]] animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
to_string(self, *, show_metadata=False, preview_cols=0)#
Return human-readable string representation of Table or RecordBatch.
Parameters:
show_metadatabool, default False
Display Field-level and Schema-level KeyValueMetadata.
preview_colsint, default 0
Display values of the columns for the first N columns.
Returns:
to_struct_array(self, max_chunksize=None)#
Convert to a chunked array of struct type.
Parameters:
max_chunksizeint, default None
Maximum number of rows for ChunkedArray chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.
Returns:
unify_dictionaries(self, MemoryPool memory_pool=None)#
Unify dictionaries across all chunks.
This method returns an equivalent table, but where all chunks of each column share the same dictionary values. Dictionary indices are transposed accordingly.
Columns without dictionaries are returned unchanged.
Parameters:
memory_poolMemoryPool, default None
For memory allocations, if required, otherwise use default pool
Returns:
Examples
import pyarrow as pa arr_1 = pa.array(["Flamingo", "Parrot", "Dog"]).dictionary_encode() arr_2 = pa.array(["Horse", "Brittle stars", "Centipede"]).dictionary_encode() c_arr = pa.chunked_array([arr_1, arr_2]) table = pa.table([c_arr], names=["animals"]) table pyarrow.Table animals: dictionary<values=string, indices=int32, ordered=0>
animals: [ -- dictionary: ["Flamingo","Parrot","Dog"] -- indices: [0,1,2], -- dictionary: ["Horse","Brittle stars","Centipede"] -- indices: [0,1,2]]
Unify dictionaries across both chunks:
table.unify_dictionaries() pyarrow.Table animals: dictionary<values=string, indices=int32, ordered=0>
animals: [ -- dictionary: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"] -- indices: [0,1,2], -- dictionary: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"] -- indices: [3,4,5]]
validate(self, *, full=False)#
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=Truefor thorough validation checks (potentially O(n)).
Parameters:
If True, run expensive checks, otherwise cheap checks only.
Raises:
ArrowInvalid