[Python-Dev] PEP 454 (tracemalloc): new minimalist version (original) (raw)

Charles-François Natali cf.natali at gmail.com
Sat Oct 19 15:58:32 CEST 2013

gettracemallocmemory() function:

Get the memory usage in bytes of the tracemalloc module as a tuple: (size: int, free: int). * size: total size of bytes allocated by the module, including free bytes * free: number of free bytes available to store data What's free exactly? I assume it's linked to the internal storage area used by tracemalloc itself, but that's not clear at all. Also, is the tracemalloc overhead included in the above stats (I'm mainly thinking about getstats() and gettracedmemory()? If yes, I find it somewhat confusing: for example, AFAICT, valgrind's memcheck doesn't report the memory overhead, although it can be quite large, simply because it's not interesting. My goal is to able to explain how every byte is allocated in Python. If you enable tracemalloc, your RSS memory will double, or something like that. You can use gettracemallocmemory() to add metrics to a snapshot. It helps to understand how the RSS memory evolves. Basically, gettracemallocsize() is the memory used to store traces. It's something internal to the C module (tracemalloc). This memory is not traced because it is the traces... and so is not counted in gettracedmemory(). The issue is probably the name (or maybe also the doc): would you prefer getpythonmemory() / gettracesmemory() names, instead of gettracedmemory() / gettracemallocmemory()?

No, the names are fine as-is.

FYI Objects allocated in tracemalloc.py (real objects, not traces) are not counted in gettracedmemory() because of a filter set up by default (it was not the case in previous versions of the PEP). You can remove the filter using tracemalloc.clearfilters() to see this memory. There are two exceptions: Python objects created for the result of gettraces() and getstats() are never traced for efficiency. It is possible to trace these objects, but it's really too slow. gettraces() and getstats() may be called outside tracemalloc.py, so another filter would be needed. Well, it's easier to never trace these objects. Anyway, they are not interesting to understand where your application leaks memory.

Perfect, that's all I wanted to know.

getobjecttrace(obj) is a shortcut for gettrace(getobjectaddress(obj)). I agree that the wrong size information can be surprising.

I can delete getobjecttrace(), or rename the function to getobjecttraceback() and modify it to only return the traceback. I prefer to keep the function (modified for getobjecttraceback). tracemalloc can be combined with other tools like Melia, Heapy or objgraph to combine information. When you find an interesting object with these tools, you may be interested to know where it was allocated.

If you mean modify it to return only the trace, then that's fine. As for the name, traceback does indeed sound less confusing than trace, but we should just make sure that the names are consistent across the API (i.e. always use "trace" or "always use "traceback", not both).

gettrace(address) function:

Get the trace of a memory block as a (size: int, traceback) tuple where traceback is a tuple of ``(filename: str, lineno: int)tuples, *filename* and *lineno* can beNone``. Return None if the tracemalloc module did not trace the allocation of the memory block. See also getobjecttrace(), getstats() and gettraces() functions. Do you have example use cases where you want to work with a raw addresses? An address is the unique key to identify a memory block. In Python, you don't manipulate directly memory blocks, that's why you have a getobjectaddress() function (link objects to traces). I added gettrace() because gettraces() is very slow. It would be stupid to call it if you only need one trace of a memory block. I'm not sure that this function is really useful. I added it to workaround the performance issue, and because I believe that someone will need it later :-) What do you suggest for this function?

Well, I can certainly find a use-case for get_object_trace(): even if it uses get_trace() internally, I'm not convinced that the later is useful. If we cannot come up with a use case for working with raw addresses, I'm tempted to just keep get_object_trace() public, and make get_object_address() and get_trace() private. In short, don't make any address-manipulating function public.

Are those match methods really necessary for the end user, i.e. are they worth being exposed as part of the public API? (Oh, I just realized that matchlineno() and may lead to bugs, I removed it.) Initially, I exposed the methods for unit tests. Later, I used them in Snapshot.applyfilters() to factorize the code (before I add 2 implementations to match a filter, one in C, another in Python). I see tracemalloc more as a library, I don't know yet how it will be used by new tools based on it. Snapshot is more an helper (convinient class) than a mandatory API to use tracemalloc. You might want to use directly filters to analyze raw datas. Users are supposed to use tracemalloc.addfilter() and Snapshot.applyfilters(). You prefer to keep them private and not document them? I don't have a strong opinion on this point.

IIUC, you only use those match methods for tests and internally for code factorization: IMO, that's a hint they shouldn't be made public.

I usually follow a simple rule of thumb for APIs: if you can't come up with a good use case for something, it shouldn't be made public: it leads to simpler, streamlined APIs, much more pleasant to learn and use. Also, one can always add a new method to an API, but it's impossible to remove from it without breaking backward compatibility. So I'd say leave them out for now, and we'll see with time if they're really necessary.

StatsDiff --------- Well, StatsDiff is useless :-) I just removed it.

Great, that's also what I thought.

I modified GroupedStats.compareto() to sort differences by default, but I added a sort parameter to get the list unsorted. sort=False can be used to sort differences differently (sorting the list twice would be inefficient).

OK, that's a good compromise.

Another option would be add sortkey and sortreverse parameters.

IMO that's overkill: the user can use sort() on the list if he wants a special-purpose order.

Snapshot --------

``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: dict=None)`` class: Snapshot of traces and statistics on memory blocks allocated by Python. I'm confused. Why are gettrace(), getobjecttrace(), getstats() etc not methods of a Snapshot object? getstats() returns the current stats. If you call it twice, you get different results. The principe of a snapshot is to be frozen: stats, traces and metrics are read once when the snapshot was created. To get stats of a snapshot, just read its stats attribute. To get a trace, it's snapshot.traces[address]. Is it because you don't store all the necessary information in a snapshot, or are they just some sort of shorthands, like: stats = getstats() vs snapshot = Snapshot.create() stats = snapshot.stats I already used other tools like Melia and Heapy, and it's convinient to get access to raw data to compute manually my own view. I don't want to force users to use the high-level API (Snapshot). Is it a problem to have two API (low-level like getstats() and high-level like Snapshot) for similar use cases? What do you suggest?

I didn't understand your above explanation: could get_stats() be implemented atop a snapshot, or not?

=> Done, I renamed Snapshot.write() to Snapshot.dump().

By the way, load() and dump() are limited to filenames (string). Should they accept file-like object? isinstance(filename, str) may be used to check if the parameter is a filename or a open file object.

IMO, they're fine as-is.

Metric ------

Metric(name: str, value: int, format: str) class: Value of a metric when a snapshot is created. Alright, what's a metric again ;-) ? I don't know if it's customary, but having short examples would IMO be nice. => done, I improved the doc

Humm... I didn't see any metric example at http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html Is it me?

cf

More information about the Python-Dev mailing list