REGR: fillna not filling NaNs after pivot without explicitly listing pivot values · Issue #36495 · pandas-dev/pandas (original) (raw)

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• (optional) I have confirmed this bug exists on the master branch of pandas.

Hi,

I have this (simplified) piece of code which seems to have stopped working correctly in Pandas 1.1:

import pandas as pd

data = [ [1, 1, 1, 1.], [2, 2, 2, 2.], [3, 3, 3, 3.], ]

df = pd.DataFrame(data, columns=['i1', 'i2', 'i3', 'f1']) df = df.pivot('i1', 'i2') df = df.fillna(0) print(df)

Problem description

This is the output in Pandas 1.1.2:

     i3             f1
i2    1    2    3    1    2    3
i1
1   1.0  NaN  NaN  1.0  0.0  0.0
2   NaN  2.0  NaN  0.0  2.0  0.0
3   NaN  NaN  3.0  0.0  0.0  3.0

It contains NaNs after using fillna. It can be fixed by explicitly providing remaining columns as values to pivot (which should be the default):
df = df.pivot('i1', 'i2', values=['i3', 'f1'])

It also works if all columns are of the same type, like when I change f1 to contain ints instead of floats. In that case I don't need to specify the values.

Expected Output

This is the output on Pandas 1.0.5 and 0.25.3:

     i3             f1
i2    1    2    3    1    2    3
i1
1   1.0  0.0  0.0  1.0  0.0  0.0
2   0.0  2.0  0.0  0.0  2.0  0.0
3   0.0  0.0  3.0  0.0  0.0  3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2a7d332
python : 3.7.8.final.0
python-bits : 32
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : None.None

pandas : 1.1.2
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 50.3.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : 1.3.18
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
None