BUG: Trying to use convert_dtypes with dtype_backend='pyarrow' specified on empty categorical series results in error or null[pyarrow] dtype · Issue #59934 · pandas-dev/pandas (original) (raw)

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

example 1

ser1 = pd.Series(pd.Categorical([None] * 5)) converted1 = ser1.convert_dtypes(dtype_backend="pyarrow")

example 2

ser2 = pd.Series(pd.Categorical([None] * 5, categories=["S1", "S2"])) converted2 = ser2.convert_dtypes(dtype_backend="pyarrow")

Issue Description

Trying to convert categorical series with empty categories results in pyarrow.lib.ArrowNotImplementedError: NumPyConverter doesn't implement <null> conversion. .
Trying to convert categorical series with non-empty categories returns null[pyarrow] dtype.

This is inconsistent with regular behavior when converting categoricals - if series is not empty, convert_dtypes call returns categorical dtype (essentially ignoring the requested pyarrow backend conversion).

I encountered this issue when trying to read files (like parquet, spss, ...) where some categorical variables are empty...

Expected Behavior

Returning categorical dtype with propagated categories, and not raising errors.

Installed Versions

INSTALLED VERSIONS

commit : 00855f8
python : 3.12.0
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 3.0.0.dev0+1536.g00855f81bd
numpy : 1.26.4
dateutil : 2.9.0.post0
pip : 24.2
Cython : 3.0.11
sphinx : 8.0.2
IPython : 8.27.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : 1.4.0
fastparquet : 2024.5.0
fsspec : 2024.9.0
html5lib : 1.1
hypothesis : 6.112.2
gcsfs : 2024.9.0post1
jinja2 : 3.1.4
lxml.etree : 5.3.0
matplotlib : 3.9.2
numba : 0.60.0
numexpr : 2.10.1
odfpy : None
openpyxl : 3.1.5
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 17.0.0
pyreadstat : 1.2.7
pytest : 8.3.3
python-calamine : None
pytz : 2024.2
pyxlsb : 1.0.10
s3fs : 2024.9.0
scipy : 1.14.1
sqlalchemy : 2.0.35
tables : 3.10.1
tabulate : 0.9.0
xarray : 2024.9.0
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : 0.23.0
tzdata : 2024.2
qtpy : None
pyqt5 : None