| msg280566 - (view) |
Author: ProgVal (Valentin.Lorentz) |
Date: 2016-11-11 08:33 |
| Hi, I use `resource.setrlimit(resource.RLIMIT_DATA, ...)` to limit the memory usage of a child process. I noticed that on recent Linux versions, my processes are much more likely to raise MemoryError or behave inconsistently (which I believe is caused by a MemoryError being silently ignored somewhere). With Linux 4.7 under Python 3.4 (tested on a Debian 8.0 chroot) and 3.5 (tested on Debian Testing and Unstable), the attached script crashes on the `assert False`, which it should not encounter since all execution paths go through a `q.put`. With Linux 3.16 under Python 3.4 (tested on a Debian 8.0 virtual machine), the attached script terminates without an error. Even when restricting the memory to 1MB instead of 10MB, it still works. It may look like I just have to increase the limit to a larger value, eg. 20MB. However, when there is more imported modules in the parent process, the required value gets incredibly high (eg. 1GB). |
|
|
| msg281999 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2016-11-29 15:49 |
| Your script works fine here (Ubuntu 16.04). Besides, if you're seeing differences between two Linux versions with the same Python version, it is quite unlikely to be a Python issue. |
|
|
| msg282006 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-11-29 16:13 |
| It's very hard to estimate the water high-mark for memory because the memory includes different things: read-only memory, mmap() on files, read-only memory pages shared between multiple processes for libraries, etc. I suggest to use the tracemalloc module to have a better idea of the memory usage of the memory directly allocated by Python. |
|
|
| msg282008 - (view) |
Author: progval (progval) |
Date: 2016-11-29 16:23 |
| I dumped tracemalloc's statistics just before the assert False. Line numbers are unchanged (I added code on existing lines). Result as attachement. On 29/11/2016 17:13, STINNER Victor wrote: > STINNER Victor added the comment: > > It's very hard to estimate the water high-mark for memory because the memory includes different things: read-only memory, mmap() on files, read-only memory pages shared between multiple processes for libraries, etc. > > I suggest to use the tracemalloc module to have a better idea of the memory usage of the memory directly allocated by Python. > > ---------- > nosy: +haypo > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue28663> > _______________________________________ |
|
|
| msg282009 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-11-29 16:29 |
| rlimit_tracemalloc.txt: Oh, my idea was only to see the total, https://docs.python.org/dev/library/tracemalloc.html#tracemalloc.get_traced_memory Current and peak memory usage. |
|
|
| msg282013 - (view) |
Author: progval (progval) |
Date: 2016-11-29 16:42 |
| (4350362, 4376669) |
|
|
| msg282015 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-11-29 16:56 |
| ProgVal: "(4350362, 4376669)" is it the same result on Python 3.4 and Python 3.5? I don't understand your comment :-/ Can you please elaborate? |
|
|
| msg282027 - (view) |
Author: progval (progval) |
Date: 2016-11-29 18:39 |
| Sorry. That's the result of | |
|
get_traced_memory |
| msg285987 - (view) |
Author: Inada Naoki (methane) * |
Date: 2017-01-22 05:35 |
| @ProgVal, could you try following? I doubt that this difference just shows rlimit behavior, not memory usage. --- $ cat rlimit_difference_linux_versions.py import sys import queue import multiprocessing from urllib.request import Request, urlopen import resource import tracemalloc def fetch(): url = 'http://python.org/' request = Request(url) fd = urlopen(request) return 'ok' fetch() print(tracemalloc.get_traced_memory()) $ /usr/bin/time python3 rlimit_difference_linux_versions.py (0, 0) 0.12user 0.01system 0:01.47elapsed 9%CPU (0avgtext+0avgdata 18372maxresident)k 0inputs+0outputs (0major+2639minor)pagefaults 0swaps $ PYTHONTRACEMALLOC=1 /usr/bin/time python3 rlimit_difference_linux_versions.py (6072093, 6140554) 0.22user 0.01system 0:01.52elapsed 15%CPU (0avgtext+0avgdata 22640maxresident)k 0inputs+0outputs (0major+3942minor)pagefaults 0swaps |
|
|
| msg285997 - (view) |
Author: ProgVal (Valentin.Lorentz) |
Date: 2017-01-22 08:19 |
| first command: (0, 0) 0.14user 0.01system 0:00.94elapsed 16%CPU (0avgtext+0avgdata 18484maxresident)k 560inputs+0outputs (0major+2702minor)pagefaults 0swaps second command: (6663744, 6732519) 0.18user 0.01system 0:00.96elapsed 20%CPU (0avgtext+0avgdata 22572maxresident)k 480inputs+0outputs (2major+4081minor)pagefaults 0swaps |
|
|
| msg285999 - (view) |
Author: Inada Naoki (methane) * |
Date: 2017-01-22 08:53 |
| On Linux 3.16? 4.7? What is difference between 3.16 and 4.7? Again, I doubt Python's memory consumption increased by Linux version. Isn't rlimit more strict for now? Even if memory usage is really grow, I don't think it's a Python's issue. Maybe, environment issue or kernel issue. |
|
|
| msg286000 - (view) |
Author: ProgVal (Valentin.Lorentz) |
Date: 2017-01-22 09:16 |
| > On Linux 3.16? 4.7? What is difference between 3.16 and 4.7? On 4.8 (which has the same issue as 4.7). The difference is that on 4.7, the rlimit has to be set incredibly high for the child process not to hit the limit. > Again, I doubt Python's memory consumption increased by Linux version. > Isn't rlimit more strict for now? That's what I think too, because I would have noticed if the interpreter actually started using 20MB for such a simple operation. > Even if memory usage is really grow, I don't think it's a Python's issue. > Maybe, environment issue or kernel issue. I agree. But I don't know how to reproduce this issue without multiprocessing.Queue. |
|
|
| msg288575 - (view) |
Author: Inada Naoki (methane) * |
Date: 2017-02-25 18:47 |
| I close this issue, because there are no enough evidence it's Python's issue. When hit ulimit is just a OS's detail. Please ping or file a new issue when memory usage is really grown. |
|
|