Memory Leak with rolling quantile (original) (raw)
Code Sample
import os import psutil import pandas as pd import numpy as np
process = psutil.Process(os.getpid()) i = 0
def sink(): global i i += 1 if i % 100 == 0: mem = process.memory_info().rss / 1e6 print("mem %fMB" % mem)
while True: s = pd.Series(np.random.randn(5e6)) m = s.rolling(1e3).quantile(0.1) # leaks (unexpected) # m = s.rolling(1e3).quantile(0) # no leak (expected) sink()
Problem description
Rolling quantile leaks memory with quantiles within (0,1)
Related Code
| return _window.roll_quantile(arg, window, minp, indexi, |
|---|
| def roll_quantile(ndarray[float64_t, cast=True] input, int64_t win, |
|---|
The leak should be in roll_quantile cython code. I could not trace it with objgraph.
I believe a free for skiplist is missing.
Program Output
Expected
mem 66.822144MB
mem 66.801664MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
Actual
mem 80.261120MB
mem 91.615232MB
mem 103.239680MB
mem 114.864128MB
mem 126.218240MB
mem 137.842688MB
mem 149.467136MB
mem 161.091584MB
mem 172.445696MB
mem 184.070144MB
mem 195.694592MB
mem 207.048704MB
mem 218.673152MB
mem 230.297600MB
mem 241.651712MB