Issue 29861: multiprocessing Pool keeps objects (tasks, args, results) alive too long (original) (raw)

Created on 2017-03-20 18:14 by pitrou, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (14)

msg289894 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-20 18:14

The various workers in multiprocessing.Pool keep a reference to the last encountered task or task result. This means some data may be kept alive even after the caller is done with them, as long as some other task doesn't clobber the relevant variables.

Specifically, Pool._handle_tasks(), Pool._handle_results() and the toplevel worker() function fail to clear references at the end of each loop.

Originally reported at https://github.com/dask/distributed/issues/956

msg289895 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-20 18:17

Quick patch below. I'll make a PR once I have time to :-)

diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py index ffdf426..945afa2 100644 --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -128,6 +128,8 @@ def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None, util.debug("Possible encoding error while sending result: %s" % ( wrapped)) put((job, i, (False, wrapped))) +

   task = job = result = func = args = kwds = None
   completed += 1

util.debug('worker exiting after %d tasks' % completed)

@@ -402,6 +404,8 @@ class Pool(object): if set_length: util.debug('doing set_length()') set_length(i+1)

```
       finally:
```

           task = taskseq = job = None
   else:
       util.debug('task handler got sentinel')

@@ -445,6 +449,7 @@ class Pool(object): cache[job]._set(i, obj) except KeyError: pass

       task = job = obj = None

   while cache and thread._state != TERMINATE:
       try:

@@ -461,6 +466,7 @@ class Pool(object): cache[job]._set(i, obj) except KeyError: pass

       task = job = obj = None

   if hasattr(outqueue, '_reader'):
       util.debug('ensuring that outqueue is not full')

msg290086 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-24 12:52

New changeset 8988945cdc27ffa86ba8c624e095b51c459f5154 by Antoine Pitrou in branch 'master': bpo-29861: release references to multiprocessing Pool tasks (#743) https://github.com/python/cpython/commit/8988945cdc27ffa86ba8c624e095b51c459f5154

msg290088 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-24 13:45

New changeset cc3331fec8b7a61c3f06c097eac85bfa38490758 by Antoine Pitrou in branch '3.6': bpo-29861: release references to multiprocessing Pool tasks (#743) (#800) https://github.com/python/cpython/commit/cc3331fec8b7a61c3f06c097eac85bfa38490758

msg290089 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-24 14:19

New changeset 80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc by Antoine Pitrou in branch '3.5': bpo-29861: release references to multiprocessing Pool tasks (#743) (#801) https://github.com/python/cpython/commit/80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc

msg290092 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-03-24 15:03

New changeset 5084ff7ddfe68969d95af12075016f309718aea8 by Antoine Pitrou in branch '2.7': bpo-29861: release references to multiprocessing Pool tasks (#743) (#803) https://github.com/python/cpython/commit/5084ff7ddfe68969d95af12075016f309718aea8

msg291587 - (view)

Author: Xiang Zhang (xiang.zhang) * (Python committer)

Date: 2017-04-13 03:25

Hi, Antoine, after this change, I sometimes see tests fail for 3.5 branch, for example http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/194/steps/test/logs/stdio.

msg291628 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-04-13 19:03

I can't reproduce here, on Ubuntu 16.04, after running the test 500 times.

msg291629 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-04-13 19:12

Ok, I can reproduce now.

msg291646 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-04-14 11:10

New changeset 685cdb9acc3fca04a9897d88b89771ddfd50e772 by Antoine Pitrou in branch 'master': Relax test timing (bpo-29861) to avoid sporadic failures (#1120) https://github.com/python/cpython/commit/685cdb9acc3fca04a9897d88b89771ddfd50e772

msg291648 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-04-14 11:37

New changeset 413a8913aee255d0635e4ddbb9c343d9e5e76fea by Antoine Pitrou in branch '3.6': Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1132) https://github.com/python/cpython/commit/413a8913aee255d0635e4ddbb9c343d9e5e76fea

msg291649 - (view)

Author: Antoine Pitrou (pitrou) * (Python committer)

Date: 2017-04-14 11:45

New changeset 47f24a018f6a9c3ed996d2b83a7ebd0d1f905827 by Antoine Pitrou in branch '3.5': Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1133) https://github.com/python/cpython/commit/47f24a018f6a9c3ed996d2b83a7ebd0d1f905827

msg293061 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2017-05-05 07:41

New changeset 685cdb9acc3fca04a9897d88b89771ddfd50e772 by Antoine Pitrou in branch 'master': Relax test timing (bpo-29861) to avoid sporadic failures (#1120)

Oh, this change wasn't backported to 2.7 and caused the bpo-30269. I proposed a backport: https://github.com/python/cpython/pull/1472 I will merge it once tests pass ;-)

msg293064 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2017-05-05 07:47

New changeset fd6094cdebb5736745d164e0207de2d4cb0b50dc by Victor Stinner in branch '2.7': Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1472) https://github.com/python/cpython/commit/fd6094cdebb5736745d164e0207de2d4cb0b50dc

History

Date

User

Action

Args

2022-04-11 14:58:44

admin

set

github: 74047

2017-05-05 07:47:13

vstinner

set

messages: +

2017-05-05 07:41:54

vstinner

set

nosy: + vstinner
messages: +

2017-05-05 07:40:54

vstinner

link

issue30269 superseder

2017-05-05 07:40:16

vstinner

set

pull_requests: + <pull%5Frequest1571>

2017-04-14 11:45:29

pitrou

set

messages: +

2017-04-14 11:37:48

pitrou

set

messages: +

2017-04-14 11:21:01

pitrou

set

pull_requests: + <pull%5Frequest1267>

2017-04-14 11🔞21

pitrou

set

pull_requests: + <pull%5Frequest1266>

2017-04-14 11:10:02

pitrou

set

messages: +

2017-04-13 19:37:13

pitrou

set

pull_requests: + <pull%5Frequest1258>

2017-04-13 19:12:41

pitrou

set

messages: +

2017-04-13 19:03:31

pitrou

set

messages: +

2017-04-13 03:25:10

xiang.zhang

set

nosy: + xiang.zhang
messages: +

2017-03-24 15:04:38

pitrou

set

status: open -> closed
resolution: fixed
stage: needs patch -> resolved

2017-03-24 15:03:48

pitrou

set

messages: +

2017-03-24 14:53:25

pitrou

set

pull_requests: + <pull%5Frequest708>

2017-03-24 14:19:20

pitrou

set

messages: +

2017-03-24 13:45:36

pitrou

set

messages: +

2017-03-24 13:30:30

pitrou

set

pull_requests: + <pull%5Frequest706>

2017-03-24 13:15:28

pitrou

set

pull_requests: + <pull%5Frequest705>

2017-03-24 12:52:13

pitrou

set

messages: +

2017-03-20 18:43:52

pitrou

set

pull_requests: + <pull%5Frequest656>

2017-03-20 18:17:28

pitrou

set

messages: +

2017-03-20 18:14:14

pitrou

create