[Python-Dev] cProfile and threads (original) (raw)

Kristján Valur Jónsson kristjan at ccpgames.com
Tue Aug 17 12:16:47 CEST 2010

-----Original Message----- From: Nick Coghlan [mailto:ncoghlan at gmail.com] Sent: 17. ágúst 2010 10:04

Both look like good ideas to me (multi-threaded profiling and debugging is fairly painful and it would be good to be able to do something to improve that situation). Indeed. I expect that profiling web server frameworks would be a good candidate, since a "profile page" can be easily set up.

> The first patch makes lsprof, the engine behind cProfile, multi-stack > aware.  This allows the same cProfiler.Profile() instance to be active > on multiple python threads and still meaningful information is gathered.

I'm curious as to the memory impact this has on the profiler (it obviously can't be too bad if you're able to run it against your live servers). The change is minimal. We have to have an extra rotatingtree to match tls to stack anchor points. There are also a few extra members in the profiling "Context" entries, but these are ephemeral.

One minor quibble here is that I would suggest using "global=False" in your docstring signatures. Okay. The docs also need to be in line with the docstrings (e.g. the docs say "setprofile(profilefunc)" while the docstring says "setprofile(function)" Both patches seem to be missing updates to the relevant documentation. Yes, this is intentional. I didn't want to waste effort on writing documentation before having exposed this to you. Sometimes my good ideas turn out to be not so good and end up being rejected.

I expect this would be difficult to unit test properly, but at least some basic tests to check that the new global configuration of settrace and setprofile don't blow, and that a profiler can be shared between two threads would be good. This is my intention too, but again, I wanted to give this some airing first. What't I probably end up doing is setting up a few threads, have them do some token work, and then do analysis on the Profile.stats member to make sure that all of them were accounted for and all and only callgraph relations show up.

Kristján

More information about the Python-Dev mailing list