[Python-Dev] DTRACE support (original) (raw)

Jesus Cea jcea at jcea.es
Sat Sep 7 05:40:04 CEST 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 20:33, Antoine Pitrou wrote:

On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:

It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :) As a concrete data point: - here are Dave's modifications to ceval.c for systemtap: http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here are your modifications to ceval.c for dtrace: http://bugs.python.org/review/13405/diff/6151/Python/ceval.c

Unfair, because that code is not doing the same thing.

Most of the extra complexity is there to deal with DTRACE ability to provide meaningful stackframes, with Python code instead of CPython evaluation loop. This is kind of magical.

Look at this:

"" [root at stargate-host /]# ps -lAf|grep -i correo 0 S correo 1371 1 0 40 20 ? 277063 ? Jul 29 ? 90:43 /usr/local/bin/python /export/home/ 0 S root 20397 20373 0 40 20 ? 1294 ? 05🔞16 pts/12 0:00 grep -i correo

[root at stargate-host /]# dtrace -n 'python1371:::function-entry {jstack();}'

5 3164 PyEval_EvalFrameEx:function-entry libpython2.7.so.1.0PyEval_EvalFrameEx+0x89a [ /usr/local/lib/python2.7/logging/__init__.py:1332 (isEnabledFor) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/logging/init.py:1202 (log) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0PyEval_EvalFrameEx+0x59ae [ /home/correo/durus/connection.py:483 (shrink) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:225 (shrink_cache) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:303 (commit2) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:261 (commit) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:191 (_monitor) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:125 (process_vacation2) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:138 (process_vacation) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0function_call+0x192 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py: ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0function_call+0xa4 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0instancemethod_call+0xa1 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0t_bootstrap+0x52 libc.so.1_thr_setup+0x4e libc.so.1`_lwp_start

^C """

Lets say I want to know about what codepaths are hitting the OS "sync" features, so I do this:

""" [root at stargate-host /]# dtrace -n 'syscall::fdsync:entry/pid==1371/{jstack();}' dtrace: description 'syscall::fdsync:entry' matched 1 probe

CPU ID FUNCTION:NAME 3 58344 fdsync:entry libc.so.1__fdsync+0x15 libdb-6.0.so__os_fsync+0x91 libdb-6.0.so__log_flush_int+0x685 libdb-6.0.so__log_flush+0x6a libdb-6.0.so__log_flush_pp+0x121 _pybsddb.soDBEnv_log_flush+0x39 libpython2.7.so.1.0PyEval_EvalFrameEx+0x6992 [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:918 (flush) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:1005 (lmtp2) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:763 (lmtp) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0function_call+0x192 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0PyEval_EvalFrameEx+0x2b94 [ /export/home/correo/lmtp.py:214 (_ignore_socket_errors) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0function_call+0x192 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py:504 (run) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:551 (__bootstrap_inner) ] libpython2.7.so.1.0PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:524 (__bootstrap) ] libpython2.7.so.1.0PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0function_call+0xa4 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0instancemethod_call+0xa1 libpython2.7.so.1.0PyObject_Call+0x5c libpython2.7.so.1.0PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0t_bootstrap+0x52 libc.so.1_thr_setup+0x4e libc.so.1_lwp_start """

I could even aggregate stacktraces and print them by frequency, cost, whatever.

Be aware that to provide this information I am not even using the probes I have integrated in Python. The program is in production and I don't need to stop it to make those measurements. They are 100% "live", not interrupting the service and without any kind of support in that Python program.

I can even plug to TCP Dtrace providers in kernel and know what exact line of Python code is sending those bytes that are being delayed because the Nagle algorithm <https://en.wikipedia.org/wiki/Nagle%27s_algorithm>. I actually digged this five years ago. what I hunt it was! :-)

PS: And it is supporting Unicode :-), another big complexity in the code.

Jesús Cea Avión // /// /// jcea at jcea.es - http://www.jcea.es/ // // // // // Twitter: @jcea // // ///// jabber / xmpp:jcea at jabber.org // // // // // "Things are not so easy" // // // // // // "My name is Dump, Core Dump" /// //_/ // // "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCUAwUBUiqgFJlgi5GaxT1NAQLezQP2PRkvn6+9Zev6GlGu14hfYTpwAMie9gcn tEieelaQniisLy22oj/MO2d2j78qlH4RoU3U/vzOTPru6uzt3OKy0UBivnBFqTcc fD/dPUCcfkigZO4nBzcqjAP4lz93wcDkx1Y8RcL3ygxeI6fSe8JKdyZKyXvyRf/9 f+3MfWJVDA== =29tQ -----END PGP SIGNATURE-----

More information about the Python-Dev mailing list