| msg311571 - (view) |
Author: OO O (OO O) |
Date: 2018-02-03 21:14 |
| OS: Win10 / 8.1 Python: 3.5 / 3.6 My program use mp.Array to share huge data. But suffer from out of memory after running for a while. But Windows task manager didn't show which process use that huge memory. And I use pympler to check my python memory usage. Still shows noting. So, I use RamMap to check, it shows a huge shared memory is used. I can reproduce the case by the simple test code: #------------------------------------------------------- import numpy as np import multiprocessing as mp import gc def F (): a = mp.Array ( 'I', 1800000000, lock = False ) # F () gc.collect () #------------------------------------------------------- No matter how hard I tried. the memory is not released. I put what I tried in the attachment picture. |
|
|
| msg311573 - (view) |
Author: OO O (OO O) |
Date: 2018-02-03 21:54 |
| mp.heap.BufferWrapper._heap = mp.heap.Heap () gc.collect () This is working!! The memory is cleared. Just delete the globe _heap and recreate a new one, but is the the correct way??? |
|
|
| msg312626 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2018-02-23 10:56 |
| Ok, this is because the multiprocessing Heap object never releases any unused arena objects, so the shared memory you allocate will probably stay allocated until the process tree ends. It is possible to change the strategy to delete unused arenas, though it's unsure whether doing so has adverse consequences (such as making later allocations costlier). It may make sense to only reclaim the larger arenas (larger than 1MB perhaps?). |
|
|
| msg312628 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2018-02-23 11:01 |
| It is also possible to uncommit the memory without deallocating it, making reuse potentially faster, but that requires low-level platform-specific code (e.g. madvise(MADV_DONTNEED) on Linux or DiscardVirtualMemory() on Windows). |
|
|
| msg315131 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2018-04-09 15:37 |
| New changeset e4679cd644aa19f9d9df9beb1326625cf2b02c15 by Antoine Pitrou in branch 'master': bpo-32759: Free unused arenas in multiprocessing.heap (GH-5827) https://github.com/python/cpython/commit/e4679cd644aa19f9d9df9beb1326625cf2b02c15 |
|
|
| msg315132 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2018-04-09 15:42 |
| I pushed a fix for this in 3.8. Since the fix is a bit delicate, I'd rather not backport it. Thank you for reporting this issue! |
|
|
| msg315133 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2018-04-09 16:01 |
| This change looks safe to me. I was just not sure that it is enough for practical cases. What if after allocating a large buffer the rest of the new area would be used for allocating small buffers? They can keep references to the large area after freeing the large buffer. Perhaps it is worth to block marking the remainder of a large area available. This will increase the memory consumption by small percent, but will reduce the risk of prolonging the life time of large blocks. |
|
|
| msg315137 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2018-04-09 16:34 |
| Yes, it's a minimal effort. More sophisticated behavior would require a more sophisticated allocator. |
|
|
| msg367153 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-04-23 22:20 |
| New changeset 857d573257ab150d6bea666354312a5fd284b171 by Victor Stinner in branch '3.7': bpo-39932: Fix multiprocessing test_heap() (GH-19690) https://github.com/python/cpython/commit/857d573257ab150d6bea666354312a5fd284b171 |
|
|
