BUG: boolean series .isin([pd.NA])] inconsistent for series length · Issue #60678 · pandas-dev/pandas (original) (raw)

Pandas version checks

Reproducible Example

import pandas as pd ser = pd.Series(True, index=range(10_000_000), dtype='boolean') ser.iloc[0] = pd.NA ser.iloc[1] =False result_short = ser.iloc[:10].isin([False, pd.NA]) assert result_short.sum() == 2 result_long = ser.isin([False, pd.NA])

Issue Description

The behavior of the .isin() function on nullable boolean boolean series is inconsistent for pd.NA:
For short series, this returns a series with True for every null values in the original series

pd.Series([True, False, pd.NA], dtype='boolean').isin([pd.NA]) 0 False 1 False 2 True dtype: boolean

For long series, this raises a TypeError: boolean value of NA is ambiguous

pd.Series(True, index=range(1_000_000), dtype='boolean').isin([pd.NA]) 0 False 1 False 2 False 3 False 4 False ...
999995 False 999996 False 999997 False 999998 False 999999 False Length: 1000000, dtype: boolean pd.Series(True, index=range(1_000_001), dtype='boolean').isin([pd.NA]) Traceback (most recent call last): File "", line 1, in pd.Series(True, index=range(1_000_001), dtype='boolean').isin([pd.NA]) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/series.py", line 5559, in isin result = algorithms.isin(self._values, values) File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 505, in isin return comps_array.isin(values) ~~~~~~~~~~~~~~~~^^^^^^^^ File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/arrays/masked.py", line 971, in isin result = isin(self._data, values_arr) File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 545, in isin return f(comps_array, values) File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 534, in f return np.logical_or(np.isin(c, v).ravel(), np.isnan(c)) ~~~~~~~^^^^^^ File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/numpy/lib/_arraysetops_impl.py", line 1132, in isin return _in1d(element, test_elements, assume_unique=assume_unique, ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ invert=invert, kind=kind).reshape(element.shape) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/numpy/lib/_arraysetops_impl.py", line 982, in _in1d mask |= (ar1 == a) ^^^^^^^^ File "missing.pyx", line 392, in pandas._libs.missing.NAType.bool TypeError: boolean value of NA is ambiguous

Expected Behavior

.isin([pd.NA]) should either always work or it should always raise an error.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.13.1
python-bits : 64
OS : Linux
OS-release : 5.15.0-88-generic
Version : #98-Ubuntu SMP Mon Oct 2 15🔞56 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : en_US.UTF-8
LOCALE : C.UTF-8

pandas : 2.2.3
numpy : 2.2.1
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None