Issue 26418: multiprocessing.pool.ThreadPool eats up memories (original) (raw)

Created on 2016-02-23 09:47 by renlifeng, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
cvtest5.py renlifeng,2016-02-23 09:47 code to reproduce the symptom
Messages (4)
msg260716 - (view) Author: renlifeng (renlifeng) Date: 2016-02-23 09:47
If func creates lots objects and appends them to a list, and runs over and over, pool.map(func...) will eventually eat up all memories. Cleaning the list at the end of func does not help. One can reproduce by running the attached file. By contrast, after replacing ThreadPool with the theading module (see the commented out lines), memory usage will not grow continuously. By the way, I used what's in Debian stretch, i.e. python 2.7.11 and 3.5.1.
msg289893 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-20 18:11
Your script works fine here. I see: iter 30, 30040 iter 29, 269848 iter 28, 269848 iter 27, 271996 [...]
msg289909 - (view) Author: renlifeng (renlifeng) Date: 2017-03-21 03:22
I confirm that 3.5.3 and 2.7.13 have fixed this problem. Now the memory usage will stop growing after using 28% of physical memory. In other words, this problem can not be reproduced with the latest version of python, Thanks. It's OK for me to set it to any of out of date, fixed, or closed.
msg289926 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-21 07:50
Thanks for the feedback! I'm closing the issue then.
History
Date User Action Args
2022-04-11 14:58:27 admin set github: 70606
2017-03-21 07:50:57 pitrou set status: open -> closedresolution: fixedmessages: + stage: resolved
2017-03-21 03:22:29 renlifeng set messages: +
2017-03-20 18:11:05 pitrou set nosy: + pitrou, davinmessages: +
2016-02-23 09:47:20 renlifeng create