[Python-Dev] Introducing memprof (was PyErr_NoMemory) (original) (raw)

Vladimir Marangozov Vladimir.Marangozov@inrialpes.fr
Fri, 18 Aug 2000 21:09:48 +0200 (CEST)

[Tim, on PyErr_NoMemory]

Looks good to me. And if it breaks something, it will be darned hard to tell .

It's easily demonstrated with the memprof.c module I'd like to introduce quickly here.

Note: I'll be out of town next week and if someone wants to play with this, tell me what to do quickly: upload a (postponed) patch which goes in pair with obmalloc.c, put it in a web page or remain quiet.

The object allocator is well tested, the memory profiler is not so thouroughly tested... The interface needs more work IMHO, but it's already quite useful and fun in it's current state .

Demo:

~/python/dev>python -p Python 2.0b1 (#9, Aug 18 2000, 20:11:29) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)

# Note the -p option -- it starts any available profilers through ... # a newly introduced Py_ProfileFlag. Otherwise you'll get funny results ... # if you start memprof in the middle of an execution ... import memprof memprof.doc 'This module provides access to the Python memory profiler.' dir(memprof) ['ALIGNMENT', 'ERROR_ABORT', 'ERROR_IGNORE', 'ERROR_RAISE', 'ERROR_REPORT', 'ERROR_STOP', 'MEM_CORE', 'MEM_OBCLASS', 'MEM_OBJECT', 'doc', 'name', 'geterrlevel', 'getpbo', 'getprofile', 'getthreshold', 'isprofiling', 'seterrlevel', 'setpbo', 'setproftype', 'setthreshold', 'start', 'stop'] memprof.isprofiling() 1 # It's running -- cool. We're now ready to get the current memory profile ... print memprof.getprofile.doc getprofile([type]) -> object

Return a snapshot of the current memory profile of the interpreter. An optional type argument may be provided to request the profile of a specific memory layer. It must be one of the following constants:

    MEM_CORE    - layer 1: Python core memory
    MEM_OBJECT  - layer 2: Python object memory
    MEM_OBCLASS - layer 3: Python object-specific memory

If a type argument is not specified, the default profile is returned. The default profile type can be set with the setproftype() function.

mp = memprof.getprofile() mp <global memory profile, layer 2, detailed in 33 block size classes> # now see how much mem we're using, it's a 3 tuple ... # (requested mem, minimum allocated mem, estimated mem) ... mp.memory (135038, 142448, 164792) mp.peakmemory (137221, 144640, 167032) # indeed, peak values are important. Now let's see what this gives in ... # terms of memory blocks ... mp.blocks (2793, 2793) mp.peakblocks (2799, 2799) # Again this is a 2-tuple (requested blocks, allocated blocks) ... # Now let's see the stats of the calls to the allocator. ... mp.malloc (4937, 0, 0) mp.calloc (0, 0, 0) mp.realloc (43, 0, 0) mp.free (2144, 0, 0) # A 3-tuple (nb of calls, nb of errors, nb of warnings by memprof) ... # ... # Good. Now let's see the memory profile detailed by size classes ... they're memory profile objects too, similar to the global profile: mp.sizeclass[0] <size class memory profile, layer 2, block size range [1..8]> mp.sizeclass[1] <size class memory profile, layer 2, block size range [9..16]> mp.sizeclass[2] <size class memory profile, layer 2, block size range [17..24]> len(mp.sizeclass) 33 mp.sizeclass[-1] <size class memory profile, layer 2, block size range [257..-1]> # The last one is for big blocks: 257 bytes and up. ... # Now let's see ithe detailed memory picture: for s in mp.sizeclass: ... print "%.2d - " % s.sizeclass, "%8d %8d %8d" % s.memory ... 00 - 0 0 0 01 - 3696 3776 5664 02 - 116 120 160 03 - 31670 34464 43080 04 - 30015 32480 38976 05 - 10736 11760 13720 06 - 10846 11200 12800 07 - 2664 2816 3168 08 - 1539 1584 1760 09 - 1000 1040 1144 10 - 2048 2112 2304 11 - 1206 1248 1352 12 - 598 624 672 13 - 109 112 120 14 - 575 600 640 15 - 751 768 816 16 - 407 408 432 17 - 144 144 152 18 - 298 304 320 19 - 466 480 504 20 - 656 672 704 21 - 349 352 368 22 - 542 552 576 23 - 188 192 200 24 - 392 400 416 25 - 404 416 432 26 - 640 648 672 27 - 441 448 464 28 - 0 0 0 29 - 236 240 248 30 - 491 496 512 31 - 501 512 528 32 - 31314 31480 31888 for s in mp.sizeclass: ... print "%.2d - " % s.sizeclass, "%8d %8d" % s.blocks ... 00 - 0 0 01 - 236 236 02 - 5 5 03 - 1077 1077 04 - 812 812 05 - 245 245 06 - 200 200 07 - 44 44 08 - 22 22 09 - 13 13 10 - 24 24 11 - 13 13 12 - 6 6 13 - 1 1 14 - 5 5 15 - 6 6 16 - 3 3 17 - 1 1 18 - 2 2 19 - 3 3 20 - 4 4 21 - 2 2 22 - 3 3 23 - 1 1 24 - 2 2 25 - 2 2 26 - 3 3 27 - 2 2 28 - 0 0 29 - 1 1 30 - 2 2 31 - 2 2 32 - 51 51 # Note that just started the interpreter and analysed it's initial ... # memory profile. You can repeat this game at any point of time, ... # look at the stats and enjoy a builtin memory profiler. ... # ... # Okay, now to the point on PyErr_NoMemory: but we need to restart ... # Python without "-p" ~/python/dev>python Python 2.0b1 (#9, Aug 18 2000, 20:11:29) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI) import memprof memprof.isprofiling() 0 memprof.start() memprof: freeing unknown block (0x40185e60) memprof: freeing unknown block (0x40175098) memprof: freeing unknown block (0x40179288) # See? We're freeing unknown blocks for memprof. ... # Okay, enough. See the docs for more: ... print memprof.seterrlevel.doc seterrlevel(flags) -> None

Set the error level of the profiler. The provided argument instructs the profiler on how tolerant it should be against any detected simple errors or memory corruption. The following non-exclusive values are recognized:

ERROR_IGNORE - ignore silently any detected errors
ERROR_REPORT - report all detected errors to stderr
ERROR_STOP   - stop the profiler on the first detected error
ERROR_RAISE  - raise a MemoryError exception for all detected errors
ERROR_ABORT  - report the first error as fatal and abort immediately

The default error level is ERROR_REPORT.

# So here's you're PyErrNoMemory effect: ... memprof.seterrlevel(memprof.ERRORREPORT | memprof.ERRORRAISE) import test.regrtest memprof: resizing unknown block (0x82111b0) memprof: raised MemoryError. Traceback (most recent call last): File "", line 1, in ? File "./Lib/test/regrtest.py", line 39, in ? import random File "./Lib/random.py", line 23, in ? import whrandom File "./Lib/whrandom.py", line 40, in ? class whrandom: MemoryError: memprof: resizing unknown block (0x82111b0) # Okay, gotta run. There are no docs for the moment. Just the source ... and function docs. (and to avoid another exception...) memprof.seterrlevel(memprof.ERRORIGNORE) for i in dir(memprof): ... x = memprof.dict[i] ... if hasattr(x, "doc"): ... print ">>>>>>>>>>>>>>>>>>>>>>>>>>>>> [%s]" % i ... print x.doc ... print '='*70 ...

[geterrlevel] geterrlevel() -> errflags

Get the current error level of the profiler.

[getpbo] getpbo() -> int

Return the fixed per block overhead (pbo) used for estimations.

[getprofile] getprofile([type]) -> object

Return a snapshot of the current memory profile of the interpreter. An optional type argument may be provided to request the profile of a specific memory layer. It must be one of the following constants:

    MEM_CORE    - layer 1: Python core memory
    MEM_OBJECT  - layer 2: Python object memory
    MEM_OBCLASS - layer 3: Python object-specific memory

If a type argument is not specified, the default profile is returned. The default profile type can be set with the setproftype() function.

[getthreshold] getthreshold() -> int

Return the size threshold (in bytes) between small and big blocks.

[isprofiling] isprofiling() -> 1 if profiling is currently in progress, 0 otherwise.

[seterrlevel] seterrlevel(flags) -> None

Set the error level of the profiler. The provided argument instructs the profiler on how tolerant it should be against any detected simple errors or memory corruption. The following non-exclusive values are recognized:

ERROR_IGNORE - ignore silently any detected errors
ERROR_REPORT - report all detected errors to stderr
ERROR_STOP   - stop the profiler on the first detected error
ERROR_RAISE  - raise a MemoryError exception for all detected errors
ERROR_ABORT  - report the first error as fatal and abort immediately

The default error level is ERROR_REPORT.

[setpbo] setpbo(int) -> None

Set the fixed per block overhead (pbo) used for estimations.

[setproftype] setproftype(type) -> None

Set the default profile type returned by getprofile() without arguments.

[setthreshold] setthreshold(int) -> None

Set the size threshold (in bytes) between small and big blocks. The maximum is 256. The argument is rounded up to the ALIGNMENT.

[start] start() -> None

Start the profiler. If it has been started, this function has no effect.

[stop] stop() -> None

Stop the profiler. If it has been stopped, this function has no effect.

-- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252