[Python-Dev] Add a new tracemalloc module to trace memory allocations (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Sep 5 23:37:52 CEST 2013

2013/8/31 Gregory P. Smith <greg at krypto.org>:

Think of the possibilities, you could even setup a test runner to enable/disable before and after each test, test suite or test module to gather narrow statistics as to what code actually caused the allocations rather than the ultimate individual file/line doing it.

It may be used with "-m test -R" to detect memory leaks, you need to repeat a test.

Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful.

I modified the PEP 454 and the implementation. By default, tracemalloc only stores 1 "frame" instance (filename and line number). But you can now use tracemalloc.set_number_frame(10) to store 10 frames per memory allocation. Don't store too much frames, remember that tracemalloc traces all Python memory allocations! So when Python allocates 1 byte, tracemalloc may need to allocate 1000 bytes to store the trace...

tracemalloc cannot get the C traceback. It was discussed in the design of the PEP 445 (Add new APIs to customize Python memory allocators) how to pass the C filename and line number. It was decided to not add them to have a simpler API.

In general, a C function is binded to a Python function. Knowing the Python function should be enough.

Victor

More information about the Python-Dev mailing list