PERF: don't create the skiprows set if using the c-parser by jreback · Pull Request #13005 · pandas-dev/pandas (original) (raw)

In [4]: DataFrame(np.random.randn(1000000,1)).to_csv('test.csv',index=False)

branch

In [1]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.74 MiB, increment: 1.59 MiB

In [2]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.89 MiB, increment: 0.22 MiB

In [3]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.98 MiB, increment: 0.28 MiB

master

In [1]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 169.84 MiB, increment: 105.79 MiB

In [2]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 171.27 MiB, increment: 24.11 MiB

In [3]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 173.39 MiB, increment: 24.63 MiB