[Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! (original) (raw)

Victor Stinner victor.stinner at gmail.com
Wed Oct 30 14:17:14 CET 2013

2013/10/30 Kristján Valur Jónsson <kristjan at ccpgames.com>:

The point of a PEP is getting something into standard python. The command line flag is also part of this. Piggybacking a lightweight client/server data-gathering version of this on top of the PEP could be beneficial in that respect.

In my opinion, your use case (log malloc/free and sent it to the network) is completly different to what tracemalloc does. Reusing tracemalloc for you would be inefficient (slow, use too much memory).

You can use tracemalloc if you want to send a snapshot of traces every N minutes. It should not be hard (less than 100 lines of Python) to implement that using a thread, pickle and a socket.

But I prefer to not include it to the PEP, Charles François wants a minimal module and prefers to develop tools on top of the module. I now agree with him (first I wanted to pack everything into the stdlib! even escape sequences to write text with colors!).

For example, the old code using "tasks" to take automatically a snapshot every N minutes or display the top 10 allocations every N minutes into the terminal with colors has been moved to a new project:

https://github.com/haypo/pytracemalloctext/blob/master/doc/index.rst

(the project is not usable yet, I will finish it after the PEP 454, and after updating the pytracemalloc module on PyPI)

Unless I am mistaken, the Pep 445 hooks must be setup before calling PyInitialize() and so using them is not trivial.

It depends on how you use the API. If you want to replace the memory allocators (use your own "malloc"), you have to call PyMem_SetAllocator() before the first memory allocation ! In Python, the first memory allocation occurs much earlier than Py_Initialize(), PyMem_RawMalloc() is the first instruction (!) executed by Python in its main() function... (see Modules/python.c).

If you want to install an hook calling the previous allocator, you can call PyMem_SetAllocator() anytime: that's why it's possible to call tracemalloc.enable() anytime.

Victor

More information about the Python-Dev mailing list