| msg329052 - (view) |
Author: Dylan Cali (Dylan Cali) |
Date: 2018-11-01 10:40 |
| Python version: 3.6.6-debug System: Kernel: 4.15.0-38-generic x86_64 Distro: Linux Mint 18.2 Sonya CPU: Quad core Intel Xeon E3-1505M Memory: 32018.6MB Expected: py-list and py-bt to print the current python frame and traceback when attaching to a hung python process with gdb, a debug build of python, and cpython/Tools/gdb/libpython.py loaded. Actual: py-list and py-bt fail with: Python Exception <class 'RuntimeError'> Type does not have a target.: Error occurred in Python command: Type does not have a target. Invoking 'set python print-stack full' in gdb produces the more useful: Traceback (most recent call last): File "", line 1, in File "~/git/cpython/Tools/gdb/libpython.py", line 916, in filename File "~/git/cpython/Tools/gdb/libpython.py", line 1158, in proxyval RuntimeError: Type does not have a target. so it is failing at: fields = gdb.lookup_type('PyUnicodeObject').target().fields() in libpython.py [1]. Reproduce: I haven't been able to create a simple standalone program that triggers the failure. I am working on a PR for Keras to try and fix the multiprocessing support in some of their utility classes. Currently the tests are sporadically hanging and my intention was to use python's gdb integration to identify exactly where and why the hangs are occuring... but I can't get that information at the moment because of the error above when invoking py-list and py-bt. So, unfortunately, the shortest path to reproduce is to checkout the PR branch, run the offending tests, connect with gdb, and invoke py-list/py-bt: * install a debug version of 3.6.6 if one isn't already available * git clone https://github.com/calid/keras.git -b fix-multiprocessing-hang * cd keras * pip install -e .[tests] * pip install tensorflow * py.test tests/keras/utils/data_utils_test.py * wait for hang * gdb -p <parent_pid_of_harness> * invoke py-list or py-bt I am happy to poke around in libpython.py and try to fix/submit a PR myself, but I'm not at all familiar with the python internals so I would need some guidance. And obviously let me know if this isn't actually a bug but rather environment related/user error. Thank you! [1] https://github.com/python/cpython/blob/v3.6.6/Tools/gdb/libpython.py#L1158 |
|
|
| msg334196 - (view) |
Author: Lisa Roach (lisroach) *  |
Date: 2019-01-22 00:40 |
| I experienced this bug as well and have tried to dig into it a little. I experimented with a service I have that uses shared libraries. If I compile with high level of optimizations for the C++ code (-O3) I don't experience the issue, but compiling without any optimizations I do. I also tried with a script that does not use any shared libraries and I do not see the issue. Digging into it a little, when running with the optimized version I found: gdb.lookup_type('PyUnicodeObject') returns a gdb.TYPE_CODE_TYPEDEF, which is the correct type. Running with the non-optimized version instead returns a gdb.TYPE_CODE_STRUCT. The struct type errors because it has no target(), but I actually think target() might be superfluous here. Both versions if I only call `fields = gdb.lookup_type('PyUnicodeObject').fields()` returns a list of fields, one of which is named `data` for the purposes of the pep check, so target() doesn't seem vital. |
|
|
| msg335518 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-02-14 11:39 |
| > Distro: Linux Mint 18.2 Sonya Hum, Dylan: what is your gdb version? On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless: $ gdb ./python GNU gdb (GDB) Fedora 8.2-6.fc29 ... (gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').target().fields()]) ['_base', 'data'] (gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').fields()]) ['_base', 'data'] I tested on a Python compiled manually from source, I ran my test in the directory of Python source code. I also tested on /usr/bin/python3 (system Python), same behavior. |
|
|
| msg335519 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-02-14 11:42 |
| Python 3.6 no longer accept bugfixes, only security fixes: http://devguide.python.org/#status-of-python-branches You can try faulthandler has a workaround. For example, install faulthandler signal handler using faulthandler.register(signal.SIGUSR1) and then send a SIGUSR1 signal to the blocked process. You might want to write the output into a file if you don't have access the program stdout: use the 'file' parameter of faulthandler.register(). See also faulthandler.dump_traceback_later() to use a timeout. https://docs.python.org/dev/library/faulthandler.html |
|
|
| msg335618 - (view) |
Author: Dylan Cali (Dylan Cali) |
Date: 2019-02-15 14:48 |
| > Hum, Dylan: what is your gdb version? $ gdb --version GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 > On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless I can confirm removing the .target() call in libpython.py resolved the issue for me with no ill effects > You can try faulthandler has a workaround. I'll give this a try, thank you |
|
|
| msg337713 - (view) |
Author: Lisa Roach (lisroach) * |
Date: 2019-03-12 03:21 |
| New changeset 1ceb3a3d172dcf0ddff38d5d6b559443ad065b84 by Lisa Roach in branch 'master': bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848) https://github.com/python/cpython/commit/1ceb3a3d172dcf0ddff38d5d6b559443ad065b84 |
|
|
| msg337729 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-03-12 09:17 |
| New changeset 047f8f25b93e2649d234fa565a59383fceb40e16 by Victor Stinner (Miss Islington (bot)) in branch '3.7': bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848) (GH-12284) https://github.com/python/cpython/commit/047f8f25b93e2649d234fa565a59383fceb40e16 |
|
|
| msg337730 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2019-03-12 09:22 |
| Thanks Dylan Cali for the bug report. Lisa Roach fixed the bug in 3.7 and master branches. Until a new release is published, you can copy manually the file from: https://github.com/python/cpython/blob/3.7/Tools/gdb/libpython.py |
|
|
| msg337824 - (view) |
Author: Dylan Cali (Dylan Cali) |
Date: 2019-03-13 00:16 |
| Thank you! And thank you Lisa Roach for the investigation and fix. |
|
|