BUG: Index and MultiIndex ops treating nan differently · Issue #38623 · pandas-dev/pandas (original) (raw)

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

Index.union treats nan differently than MultiIndex.union. Same goes for intersection.

idx1 = Index([1, np.nan]) idx2 = Index([1, np.nan, 3])

idx1.union(idx2) idx1.intersection(idx2)

mi1 = MultiIndex.from_arrays([[1, np.nan]]) mi2 = MultiIndex.from_arrays([[1, np.nan, 3]]) mi1.union(mi2) mi1.intersection(mi2)

Output:

Float64Index([1.0, 3.0, nan], dtype='float64')
Float64Index([1.0, nan], dtype='float64')
MultiIndex([(1.0,),
            (nan,),
            (nan,),
            (3.0,)],
           )
MultiIndex([(1.0,)],
           )

Problem description

Inconsistent behavior is a problem. Additionally if both MultiIndexes are equal nan is only returned once. When not matching nan is the desired output, we have to adjust for theses cases before checking for equality.

Expected Output

I am not quite sure what to expect, but taking the current behavior of merges into account, I think it would be more reasonable to return

MultiIndex([(1.0,),
            (nan,),
            (3.0,)],
           )
MultiIndex([(1.0,),
            (nan,)],
           )

for the MultiIndex operations. Thoughts @jbrockmendel ?

related to #38439

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit : c6b05c3
python : 3.8.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-58-generic
Version : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0.dev0+129.gc6b05c3ad7
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 49.6.0.post20201009
Cython : 0.29.21
pytest : 6.1.1
hypothesis : 5.37.1
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : 1.3.7
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.8.3
fastparquet : 0.4.1
gcsfs : 0.7.1
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : 2.0.0
pyxlsb : None
s3fs : 0.4.2
scipy : 1.5.2
sqlalchemy : 1.3.20
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.16.1
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.51.2
None