BUG: Merging on two non-monotonic np.intc arrays fails · Issue #60091 · pandas-dev/pandas (original) (raw)

Pandas version checks

Reproducible Example

import pandas as pd import numpy as np

df = pd.DataFrame({ "join_key": pd.Series([0, 2, 1], dtype=np.intc) }) df_details = pd.DataFrame({ "join_key": pd.Series([0, 1, 2], dtype=np.intc), "value": ["a", "b", "c"] })

merged = pd.merge(df, df_details, on="join_key", how="left")

Issue Description

When merging on two np.intc arrays (where at least one is non-monotonic), pandas throws a "Buffer dtype mismatch" error. It only seems to happen when both arrays are np.intc type. I encountered this issue because that is the dtype returned from pd.DatetimeIndex.weekday.

Traceback (most recent call last):
  File "merge_bug_test.py", line 26, in <module>
    merged_1 = pd.merge(df, df_details, on="join_key", how="left")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 184, in merge
    return op.get_result(copy=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 886, in get_result
    join_index, left_indexer, right_indexer = self._get_join_info()
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1151, in _get_join_info
    (left_indexer, right_indexer) = self._get_join_indexers()
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1125, in _get_join_indexers
    return get_join_indexers(
           ^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1759, in get_join_indexers
    lidx, ridx = get_join_indexers_non_unique(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1793, in get_join_indexers_non_unique
    lkey, rkey, count = _factorize_keys(left, right, sort=sort)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 2561, in _factorize_keys
    llab = rizer.factorize(lk)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^
  File "pandas\\_libs\\hashtable_class_helper.pxi", line 3045, in pandas._libs.hashtable.Int64Factorizer.factorize
ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'int'

It seems this issue was raised in #52451, the solution that was implemented involves checking if np.intc is not np.int32. This check returns True, meaning that np.intc is mapped to Int64Factorizer. It appears that this check no longer works as intended, or it is not robust enough.

A similar issue is raised in #58713 for uintc. In the proposed solution (#58727), it was suggested to use .itemsize as a more robust check. I think that suggestion might get to the root cause, as my np.intc(0).itemsize is 4, so np.intc should be mapped to Int32Factorizer

Expected Behavior

Merging should succeed if dtypes for both keys are the same.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.5
python-bits : 64
OS : Windows
OS-release : 10
Version : ...
machine : AMD64
processor : Intel64 Family 6 ...
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : 1.4.0
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : None
numba : 0.60.0
numexpr : 2.10.1
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : None
pyarrow : 17.0.0
pyreadstat : 1.2.7
pytest : None
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2024.9.0
scipy : None
sqlalchemy : 2.0.35
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : None
tzdata : 2024.2
qtpy : 2.4.1
pyqt5 : None