[Python-Dev] FreeBSD 7 amd64 and large memory tests (original) (raw)

Alex Martelli aleaxit at gmail.com
Wed Sep 17 17:21:55 CEST 2008

Unbelievable as this may seem, this crazy over-committing malloc behavior is by now "a classic" -- I first fought against it in 1990, when IBM released AIX 3 for its then-new RS/6000 line of workstations; in a later minor release they did provide a way to optionally switch this off, but, like on Linux, it's a system-wide switch, NOT per-process:-(.

I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best explanation I know of the subject, and recommended reading) which, on this subject, says "Linux on the other hand is seriously broken." (just like AIX 3 was). Sad to learn that BSD is now also broken in the same way:-(.

Alex On Wed, Sep 17, 2008 at 8:00 AM, James Y Knight <foom at fuhm.net> wrote:

On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:

Martin v. Löwis wrote:

I haven't yet tried posting a query to a FreeBSD list, as it could simply be a bug on amd64, but I was wondering whether there was anything (other than deactivating tests and documenting use of ulimit -v on this platform) that could be done to work around this behaviour. I think it should be possible to debug this (for people with access to such a system, and appropriate skills). I find it hard to believe that a large malloc will simply crash the process, rather than returning NULL. More likely, there is a NULL returned somewhere, and Python (or libc) fails to check for it. A simple C program doing a repetitive malloc(), much as pymalloc would with continuous demand, does indeed not see any NULL from malloc() when swap is exhausted but the process gets KILLed (the allocated memory does have to be written to to force the situation...) I'll take this up with FreeBSD folk, but I'm open to ideas as to how best to deal with the problem in the context of the test suite pending resolution by FreeBSD. Linux does the same thing, unless the user has explicitly configured that behavior off. Search the web for linux overcommit. It's controlled by the vm.overcommitmemory sysctl. Although linux's default is some heuristic that might make Python's test case work right in most cases, depending on malloc returning NULL is not something you can actually depend on. James

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com

More information about the Python-Dev mailing list