BUG: Rolling .apply() with method='table' ignores min_periods · Issue #58868 · pandas-dev/pandas (original) (raw)
Pandas version checks
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of pandas.
- I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]], columns=['A', 'B']) print(df)
prints this:
A B
0 1 2
1 3 4
2 5 6
3 7 8
4 9 10
df.rolling(3, min_periods=3).apply(lambda x: bool(print(x, '\n')))
prints this, so as expected, only batches of 3 are being evaluated
0 1.0
1 3.0
2 5.0
dtype: float64
1 3.0
2 5.0
3 7.0
dtype: float64
2 5.0
3 7.0
4 9.0
dtype: float64
0 2.0
1 4.0
2 6.0
dtype: float64
1 4.0
2 6.0
3 8.0
dtype: float64
2 6.0
3 8.0
4 10.0
dtype: float64
Out[33]:
A B
0 NaN NaN
1 NaN NaN
2 0.0 0.0
3 0.0 0.0
4 0.0 0.0
df.rolling(3, method='table', min_periods=3).apply(lambda x: bool(print(x, '\n')), raw=True, engine='numba')
prints this, where we see that the first two batches of sizes 1 and 2 are also being evaluated
[[1. 2.]] <-- batch of size 1, illegal
[[1. 2.] <-- batch of size 2, illegal
[3. 4.]]
[[1. 2.]
[3. 4.]
[5. 6.]]
[[3. 4.]
[5. 6.]
[7. 8.]]
[[ 5. 6.]
[ 7. 8.]
[ 9. 10.]]
Out[34]:
A B
0 NaN NaN
1 NaN NaN
2 0.0 0.0
3 0.0 0.0
4 0.0 0.0
Issue Description
See example above, which violates the contract imposed by the min_periods parameter.
Expected Behavior
The min_periods parameter should be respected.
Installed Versions
INSTALLED VERSIONS
commit : d9cdd2e
python : 3.12.3.final.0
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : German_Switzerland.utf8
pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
setuptools : 69.5.1
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.24.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None