BUG: read_csv with engine pyarrow doesn't handle parsing multiple date columns · Issue #47961 · pandas-dev/pandas (original) (raw)

Pandas version checks

Reproducible Example

import pandas as pd import io

data = """idx,date,time 2,2000-01-01, 00:00:00 3,2000-01-01, 00:01:00 4,2000-01-01, 00:02:00 """

df = pd.read_csv(io.StringIO(data), parse_dates={'date2': ['date', 'time']}) print(df) df = pd.read_csv(io.StringIO(data), parse_dates={'date2': ['date', 'time']}, engine='pyarrow')

Issue Description

Without , engine='pyarrow', it works fine. With it, we get

Traceback (most recent call last):
  File "/home/megorelli/pandas-dev/t.py", line 10, in <module>
    df = pd.read_csv(io.StringIO(data), parse_dates={'date2': ['date', 'time']}, engine='pyarrow')
  File "/home/megorelli/pandas-dev/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/megorelli/pandas-dev/pandas/io/parsers/readers.py", line 943, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/megorelli/pandas-dev/pandas/io/parsers/readers.py", line 605, in _read
    return parser.read(nrows)
  File "/home/megorelli/pandas-dev/pandas/io/parsers/readers.py", line 1748, in read
    df = self._engine.read()  # type: ignore[attr-defined]
  File "/home/megorelli/pandas-dev/pandas/io/parsers/arrow_parser_wrapper.py", line 152, in read
    return self._finalize_output(frame)
  File "/home/megorelli/pandas-dev/pandas/io/parsers/arrow_parser_wrapper.py", line 108, in _finalize_output
    frame.columns, frame = self._do_date_conversions(frame.columns, frame)
  File "/home/megorelli/pandas-dev/pandas/core/generic.py", line 5852, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/properties.pyx", line 69, in pandas._libs.properties.AxisProperty.__set__
    obj._set_axis(self.axis, value)
  File "/home/megorelli/pandas-dev/pandas/core/generic.py", line 771, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/home/megorelli/pandas-dev/pandas/core/internals/managers.py", line 218, in set_axis
    self._validate_set_axis(axis, new_labels)
  File "/home/megorelli/pandas-dev/pandas/core/internals/base.py", line 70, in _validate_set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 1 elements, new values have 2 elements

Expected Behavior

                date2  idx
0 2000-01-01 00:00:00    2
1 2000-01-01 00:01:00    3
2 2000-01-01 00:02:00    4

Installed Versions

/home/megorelli/pandas-dev/.venv/lib/python3.9/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 690a77fe19f9229ff02987abbd487a63c2cbe1b8
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.114-16025-ge75506b9d98e
Version : #1 SMP PREEMPT Sat Jul 16 18:52:19 PDT 2022
machine : aarch64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.0.dev0+1231.g690a77fe19.dirty
numpy : 1.22.4
pytz : 2022.1
dateutil : 2.8.2
setuptools : 62.1.0
pip : 22.0.4
Cython : 0.29.30
pytest : 7.1.2
hypothesis : 6.47.1
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.4.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 8.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
None