[Python-Dev] Encoding of PyFrameObject members (original) (raw)

Maciej Fijalkowski fijall at gmail.com
Sun Feb 8 10:01:30 CET 2015

I'm working on vmprof (github.com/vmprof/vmprof-python) which works for both cpython and pypy (pypy has special support, cpython is patched on-the fly)

On Sun, Feb 8, 2015 at 6:39 AM, Gregory P. Smith <greg at krypto.org> wrote:

To get at the Python thread state from a signal handler (using 2.7 as a reference here; but i don't believe 3.4 has changed this part much) you need to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's "struct key" as well as "keyhead" and "keymutex".

From there, in your signal handler you must try to acquire the newly exposed keymutex and do nothing if you were unable to get it. If you did acquire it (rare not to), you can walk the keyhead list looking for autoTLSkey to find the current valid thread state. I had an intern (hi Elena!) write a signal sampling based low overhead Python CPU profiler based on that last summer. I believe there are still bugs to shaken out (if they are even possible to fix... Armin's comments are true: signal handler code is super limited). I am stating this here because I want someone to pester me at PyCon if I haven't released our work as a proof of concept by then. The important take away: From what I could figure out, you need to modify the CPython interpreter to be more amenable to such introspection. A downside of a signal based profiler: ALL of the EINTR mishandling bugs within the Python interpreter, stdlib, and your own code will show up in your application. So until those are fixed (hooray for Antoine's PEP!), it may not be practical for use on production processes which is sort of the entire point of a low overhead sampling profiler... I'd like to get a buildbot setup that runs the testsuite while a continual barrage of signals are being generated. We really don't stress test that stuff (as evidence by the EINTR mishandling issues that are rampant) as non-fatal signals are so rare for most things... until they aren't. As a side note and encouragement: I wonder what PyPy could do for dynamically enabled and disabled low overhead CPU profiling. (take that as a hint that I want someone else to get extremely creative!) -gps On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

Maciej Fijalkowski wrote: > However, you can't access thread > locals from signal handlers (since in some cases it mallocs, thread > locals are built lazily if you're inside the .so, e.g. if python is > built with --shared) You might be able to use PyAddPendingCall to schedule what you want done outside the context of the signal handler. The call will be made by the main thread, though, so if you need to access the frame of whatever thread was running when the signal occured, you will have to track down its PyThreadState somehow and get the frame from there. Not sure what would be involved in doing that. -- Greg

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

More information about the Python-Dev mailing list