BUG: dataframe.any() method behaves differently for empty rows and columns · Issue #35450 · pandas-dev/pandas (original) (raw)

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd import numpy as np

df1 = pd.DataFrame(data = {"A" : [np.nan, 1], "B": [np.nan, 2], "C": [np.nan, np.nan]})

df2 = pd.DataFrame(data = [[np.nan, 1], [np.nan, 2], [np.nan, np.nan]], columns = ["A", "B"])

dataframe created with mixed values of bool and numeric with shape - 4,4

df3 = pd.DataFrame(data = [[1, np.nan, np.nan, True], [np.nan, 2, np.nan, True], [np.nan, np.nan, np.nan, False], [np.nan, np.nan, np.nan, np.nan]], columns = ["col1", "col2", "col3", "col4"])

Problem description

The documentation for any() method states that for empty row/column, np.nan is treated as True. So if the parameter "skipna = False", then the empty row should result in True for either, axis = 0/ index or 1/columns.

The dataframe.any(skipna = False, axis = {1 or 0}) method works as documented for df1 and df2. However, it behaves differently for df3.

(Similar output is obtained for df1 as well hence not included here) The obtained output for df2 is like this:

# df2.any(axis = 1, skipna = False) # df2.any(axis = "columns", skipna = False)
0    True
1    True
2    True
dtype: bool 

# df2.any(axis = 0, skipna = False) # df2.any(axis = "index", skipna = False)
A    True
B    True
dtype: bool

The obtained output for df3 is like this:

# df3.any(axis = 1, skipna = False) # df3.any(axis = "columns", skipna = False)
0      1
1    NaN
2    NaN
3    NaN
dtype: object 

# df3.any(axis = 0, skipna = False) # df3.any(axis = "index", skipna = False)
col1       1
col2     NaN
col3     NaN
col4    True
dtype: object

Expected Output

# The expected output for df3 should be, similar to df2's output, and like this: 

# df3.any(axis = 1, skipna = False) # df3.any(axis = "columns", skipna = False)
0    True
1    True
2    True
3    True
dtype: object 
    
# df3.any(axis = 0, skipna = False) # df3.any(axis = "index", skipna = False)
Col1    True
Col2    True
Col3    True
Col4    True
dtype: object

Output of pd.show_versions()

[INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.4
numpy : 1.16.5
pytz : 2019.3
dateutil : 2.8.0
pip : 19.2.3
setuptools : 41.4.0
Cython : 0.29.13
pytest : 5.2.1
hypothesis : None
sphinx : 2.2.0
blosc : None
feather : None
xlsxwriter : 1.2.1
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.8.4 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.8.0
pandas_datareader: None
bs4 : 4.8.0
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.2.1
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.2.1
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.3.9
tables : 3.5.2
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.1
numba : 0.45.1]

I am not sure where exactly I am going wrong with df3. I also thought of appending this issue with the open issue related to "bool_only" parameter here. But then before reporting this confusion/error I tried the dataframe.any()method on df1 and df2, without using parameter "bool_only" and it gives the correct result. However, it again behaves very differently for df3.

Since I might have already crossed, the threshold of a minimal bug report, my experiments with the bool_only parameter are in the attached "Any_Experiments.ipynb" file.
Any_Experiments.zip

This is my first issue/ bug/ query to Pandas. Apologies for typos and/or any violations of reporting guidelines.