BUG: pd.DataFrame.to_hdf does not support multiindex frames with string[pyarrow_numpy] · Issue #55088 · pandas-dev/pandas (original) (raw)
Pandas version checks
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of pandas.
- I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd import numpy as np data = pd.DataFrame( np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD') ).set_index(['A', 'B'])
This action will be successful as it works on numpy objects
data.to_hdf('test.hdf', key='test', format='table')
If we force the usage of pyarrow string, the same will fail
pd.options.future.infer_string = True
data_arrow = pd.DataFrame( np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD') ).set_index(['A', 'B'])
data_arrow.to_hdf('test_arrow.hdf', key='test', format='table') # This will raise ValueError("cannot have non-object label DataIndexableCol")
Issue Description
Looks like using string[pyarrow_numpy] as dtype and to_hdf with a multiindex frame throws a value error. The error comes from pytables.py and more specifically the check below.
| def validate_names(self) -> None: |
|---|
| if not is_object_dtype(Index(self.values).dtype): |
| # TODO: should the message here be more specifically non-str? |
| raise ValueError("cannot have non-object label DataIndexableCol") |
The is_object_dtype always checks against np.object_.
| def is_object_dtype(arr_or_dtype) -> bool: |
|---|
| """ |
| Check whether an array-like or dtype is of the object dtype. |
| Parameters |
| ---------- |
| arr_or_dtype : array-like or dtype |
| The array-like or dtype to check. |
| Returns |
| ------- |
| boolean |
| Whether or not the array-like or dtype is of the object dtype. |
| Examples |
| -------- |
| >>> from pandas.api.types import is_object_dtype |
| >>> is_object_dtype(object) |
| True |
| >>> is_object_dtype(int) |
| False |
| >>> is_object_dtype(np.array([], dtype=object)) |
| True |
| >>> is_object_dtype(np.array([], dtype=int)) |
| False |
| >>> is_object_dtype([1, 2, 3]) |
| False |
| """ |
| return _is_dtype_type(arr_or_dtype, classes(np.object_)) |
Expected Behavior
I would expect that is_object_dtype allows for string[pyrrow_numpy] (if and only if downstream behavior is supported by that dtype). If that action is not supported with string[pyarrow_numpy] I would expected a more informative error.
Installed Versions
INSTALLED VERSIONS
commit : 723feb9
python : 3.9.13.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252
pandas : 2.2.0dev0+208.g723feb984e
numpy : 1.22.4
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : 0.29.36
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : 2.8.5
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None