BUG: Unexpected behaviour of groupby + rank(pct=True, method="dense") · Issue #40518 · pandas-dev/pandas (original) (raw)

• I have checked that this issue has not already been reported. There was a previous issue about rank(pct=True, method="dense") but it was about applying it to pd.Series and has been resolved.
• I have confirmed this bug exists on the latest version of pandas.
• (optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample, a copy-pastable example

import pandas as pd import numpy as np data = [ [1, 3, 1], [1, 3, -5], [1, 3, 3], [1, 3, 3], [2, 3, np.nan], [1, 3, np.nan], ] df = pd.DataFrame(data, columns=["gr1", "gr2", "val"]) df["r"] = df.groupby(["gr1", "gr2"])["val"].rank( method="dense", pct=True ) print(df)

Output:

   gr1  gr2  val    r
0    1    3  1.0  1.0
1    1    3 -5.0  0.5
2    1    3  3.0  1.5
3    1    3  3.0  1.5
4    2    3  NaN  NaN
5    1    3  NaN  NaN

Problem description

The pct=True option should always return values in [0,1]. If we remove the row with gr1==2 there is no issue. Somehow the rank function miscounts the number of elements in each group which should be used to turn the ranked series to [0,1] ranked series. Note that pd.Series([1,-5,3,3,np.nan]).rank(pct=True, method='dense').values returns the desired output: array([0.66666667, 0.33333333, 1. , 1. , nan]) so the issue is related to groupby. Also, trying out all the options of na_option gives undesired result.

Expected Output

   gr1  gr2  val         r
0    1    3  1.0  0.666667
1    1    3 -5.0  0.333333
2    1    3  3.0  1.000000
3    1    3  3.0  1.000000
4    2    3  NaN       NaN
5    1    3  NaN       NaN

INSTALLED VERSIONS

commit : f2c8480
python : 3.9.2.final.0
python-bits : 64
OS : Darwin
OS-release : 20.3.0
Version : Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 1.2.3
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 54.1.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None