BUG: pd.concat dataframes with different datetime64 resolutions by Charlie-XIAO · Pull Request #53641 · pandas-dev/pandas (original) (raw)

I can confirm this issue still exists in pandas 2.1.1.

This bug together with the latest changes in pyarrow 13.0.0 apache/arrow#38171 makes it difficult to use pd.concat and pd.merge #55212.

t1 = '2023-09-01' t2 = '2023-09-01 01:00:00' t3 = '2023-09-01 01:30:00' ds1 = pd.date_range(t1, t1, freq='30T') ds2 = pd.date_range(t2, t3, freq='30T') df1 = pd.DataFrame({ 'ds': ds1.astype('datetime64[us]'), 'y': list(range(len(ds1))), }) df2 = pd.DataFrame({ 'ds': ds2.astype('datetime64[ns]'), 'y': list(range(len(ds2))), }) pd.concat([df1, df2], axis=0)

And the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 393, in concat
    return op.get_result()
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 680, in get_result
    new_data = concatenate_managers(
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 189, in concatenate_managers
    values = _concatenate_join_units(join_units, copy=copy)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
    concat_values = concat_compat(to_concat, axis=1)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/dtypes/concat.py", line 132, in concat_compat
    return cls._concat_same_type(to_concat_eas)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 2251, in _concat_same_type
    new_obj = super()._concat_same_type(to_concat, axis)
  File "/home/user/conda-envs/api/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 230, in _concat_same_type
    return super()._concat_same_type(to_concat, axis=axis)
  File "arrays.pyx", line 190, in pandas._libs.arrays.NDArrayBacked._concat_same_type
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 1 and the array at index 1 has size 2