BUG: read_csv with index_col option on pyarrow dtype cause error. · Issue #53229 · pandas-dev/pandas (original) (raw)

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Suppose that we have a file call test.csv, which contains the following lines:

When calling the following lines, we get error:

import pandas as pd import pyarrow as pa

df = pd.read_csv( 'test.csv', dtype={ 'col1': pd.ArrowDtype(pa.string()), # this cause problem 'col2': pd.ArrowDtype(pa.uint8()), # this cause problem }, engine='pyarrow', dtype_backend='pyarrow', index_col='col1', # without this is okay )

Error message:

KeyError: "Only a column name can be used for the key in a dtype mappings argument. 'col1' not found in columns."

However, when using pandas built-in dtypes is okay.

import pandas as pd import pyarrow as pa

df = pd.read_csv( 'test.csv', dtype={ 'col1': 'string', # this is okay 'col2': 'uint8', # this is okay }, engine='pyarrow', dtype_backend='pyarrow', index_col='col1', )

Issue Description

A weird scenario where read_csv failed with mysterious error.
When passing dtype option with pyarrow dtypes along with index_col option, we get a mysterious KeyError.
But replacing pyarrow dtypes with pandas bulit-in dtype is okay.

Expected Behavior

No error should happen.
In particular, index_col should be able to work with pyarrow dtypes.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.9.16.final.0
python-bits : 64
OS : Darwin
OS-release : 22.3.0
Version : Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : None
Cython : 0.29.34
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.0
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.1
pandas_datareader: None
bs4 : 4.12.2
bottleneck : 1.3.7
brotli :
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : 1.2.1
pyxlsb : 1.0.10
s3fs : None
scipy : 1.9.3
snappy :
sqlalchemy : None
tables : 3.8.0
tabulate : None
xarray : 2023.4.2
xlrd : 2.0.1
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None