BUG: groupby.shift return keys filled with fill_value when fill_value is specified · Issue #41556 · pandas-dev/pandas (original) (raw)

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

In [2]: df = pd.DataFrame({'a': [2, 1, 2, 1], 'b': ['x', 'x', 'y', 'y']})

In [3]: df.groupby('a').shift(1) Out[3]: b 0 NaN 1 NaN 2 x 3 x

In [4]: df.groupby('a').shift(1, fill_value='fill') Out[4]: a b 0 fill fill 1 fill fill 2 2 x 3 1 x

Problem description

When specifying fill_value in groupby.shift, the returned result includes the key column with keys filled with fill_value. When fill_value is unspecified (None), the key column is not included.

Expected Output

It seems pretty strange that keys are to be filled with fill_value. This makes more sense to me:

In [4]: df.groupby('a').shift(1, fill_value='fill') Out[4]: b 0 fill 1 fill 2 x 3 x

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.7.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-76-generic
Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 52.0.0.post20210125
Cython : 0.29.23
pytest : 6.2.4
hypothesis : 6.12.0
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.23.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.04.0
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.1
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.53.1