BUG: pd.Series.factorize() provides wrong values with chunked dictionary array · Issue #54844 · pandas-dev/pandas (original) (raw)

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd import pyarrow as pa a=pd.Series(['a'], dtype=pd.ArrowDtype(pa.dictionary(pa.int32(), pa.string()))) b=pd.Series(['b'], dtype=pd.ArrowDtype(pa.dictionary(pa.int32(), pa.string()))) s = pd.concat([a,b]) s.factorize()

Returns:

(array([0, 1]),
 Index(['a'], dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

For information:

>>> s.values
<ArrowExtensionArray>
['a', 'b']
Length: 2, dtype: dictionary<values=string, indices=int32, ordered=0>[pyarrow]
>>> s.values._data
<pyarrow.lib.ChunkedArray object at 0x7fc52aa856c0>
[

  -- dictionary:
    [
      "a"
    ]
  -- indices:
    [
      0
    ],

  -- dictionary:
    [
      "b"
    ]
  -- indices:
    [
      0
    ]
]

Issue Description

When using an ArrowExtensionArray containing a pyarrow ChunkedArray with a dictionary type, pd.Series.factorize() provides wrong values for the uniques values. This is due to https://github.com/pandas-dev/pandas/blob/v2.0.3/pandas/core/arrays/arrow/array.py#L891:

uniques = type(self)(encoded.chunk(0).dictionary)

I suppose we should use this instead:

uniques = type(self)(encoded.combine_chunks().dictionary)

Expected Behavior

s.factorize() shoud return:

(array([0, 1]), Index(['a', 'b'], dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

('b' appears in the uniques values)

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.0-50-generic
Version : #50-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 10 18:24:29 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.3
numpy : 1.23.4
pytz : 2022.5
dateutil : 2.8.2
setuptools : 60.2.0
pip : 23.0
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : 1.0.3
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 8.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.3
snappy : None
sqlalchemy : 2.0.4
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None