REGR: unstack on 'int' dtype prevent fillna to work · Issue #37115 · pandas-dev/pandas (original) (raw)

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• (optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample, a copy-pastable example

import pandas as pd import numpy as np

df1 = pd.DataFrame({ 'a': ['A', 'A', 'B'], 'b': ['ca', 'cb', 'cb'], 'v': [10] * 3, })

df1 = df1.set_index(['a', 'b'])

int column

df1['is_'] = 1

df1 = df1.unstack('b')

Will not work, keeping NaN in the value

df1[('is_', 'ca')] = df1[('is_', 'ca')].fillna(0)

Will raise ValueError: Cannot convert non-finite values (NA or inf) to integer

df1[('is_', 'ca')] = df1[('is_', 'ca')].astype('uint8')

Problem description

df.unstack is creating columns from a int column, but adds some NaNs as expected.

Trying to get rid of the NaN is not possible as fillna will silently fail and leave the column values as is.

Potential Source of the Problem

It looks like with this configuration, the Block for this df1[('is_', 'ca')] Series in the BlockManager is actually IntBlock, but holds a float dtype np.ndarray (!?).

From what I understand, as soon as the BlockManager operates a block consolidation (with the _consolidate function), the Block is converted to a FloatBlock and everything is OK.
e.g. a call to cat2._is_mixed_type will trigger a _consolidate and thus clear the issue.

Potential Solution Suggestion

Should the unstack function fires a consolidation of blocks, especially for column types that do not accept NaN values (like 'int' and 'bool' from numpy)?

Some potential workarounds identified so far

• Trigger a Block consolidate:
  • Add a column: the bug appeared if I use fillna before adding a column, but not after
  • Call cat2._is_mixed_type
  • Call df1.infos(): this was the most surprising one (it is because if triggers df.count() and thus consolidation)
• Use a float value: df1['is_'] = 1.0

Expected Output

      v       is_
b    ca    cb  ca   cb
a
A  10.0  10.0   1  1.0
B   NaN  10.0   0  1.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : db08276
python : 3.8.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : fr_FR.UTF-8
LOCALE : fr_FR.cp1252

pandas : 1.1.3
numpy : 1.19.0
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : 0.4.1
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.2.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.4
pandas_gbq : None
pyarrow : 1.0.1
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
numba : None