| msg76454 - (view) |
Author: (rb) * |
Date: 2008-11-26 09:13 |
| Python cannot be embedded in shared library due to dependency problems with lib-dynload. I am implementing a shared library (in C) that implements a specific API as a dynamically loadable plugin to an application. To implement this library, I am calling out to Python via the C API. When my code tries to load a Python module, it fails. This is because lib-dynload/time.so (for example) cannot resolve symbols that are in libpython2.5.so because it never looks there. In this test case I'm trying to import "time", but it seems to fail on other modules too - presumably anything that is in lib-dynload. It also fails if I import a pure Python module that then tries to import time. The error produced is: ImportError: /usr/lib/python2.5/lib-dynload/time.so: undefined symbol: PyExc_ValueError From LD_DEBUG: 29682:file=/usr/lib/python2.5/lib-dynload/time.so [0]; needed by /usr/lib/libpython2.5.so.1.0 [0] ... 29682:relocation processing: /usr/lib/python2.5/lib-dynload/time.so 29682:symbol=PyExc_ValueError; lookup in file=./myprog [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib/libdl.so.2 [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib/libc.so.6 [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 29682:symbol=PyExc_ValueError; lookup in file=/usr/lib/python2.5/lib-dynload/time.so [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib/libm.so.6 [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib/libpthread.so.0 [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib/libc.so.6 [0] 29682:symbol=PyExc_ValueError; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 29682:/usr/lib/python2.5/lib-dynload/time.so: error: symbol lookup error: undefined symbol: PyExc_ValueError (fatal) $ nm -D /usr/lib/libpython2.5.so.1|grep PyExc_ValueError 000000000033a7e0 D PyExc_ValueError $ ldd /usr/lib/python2.5/lib-dynload/time.so libm.so.6 => /lib/libm.so.6 (0x00002afe014c9000) libpthread.so.0 => /lib/libpthread.so.0 (0x00002afe0174a000) libc.so.6 => /lib/libc.so.6 (0x00002afe01967000) /lib64/ld-linux-x86-64.so.2 (0x0000555555554000) I am told that the problem is that lib-dynload/*.so should depend on libpython2.5.so.1. Test case attached. mylib.c is the library that I'm implementing reduced to the problem case. myprog.c is a stub program that loads mylib.c to demonstrate the problem. The "real" myprog would be any third-party application that I have no control over that expects to be able to dlopen() mylib.so and call functions within it. I have been given the following workaround: in mylib.c, before PyInitialize() I can call dlopen("libpython2.5.so", RTLD_LAZY |
RTLD_GLOBAL); This works, but I believe that lib-dynload/*.so should depend on libpython2.5.so.1 so this hack should not be necessary. I am using Ubuntu 8.04 with Python version 2.5.2-2ubuntu4.1. |
|
| msg76479 - (view) |
Author: Ralf Schmitt (schmir) |
Date: 2008-11-26 18:05 |
| ubuntu comes with a static libpython.a which you can also link against. Linking the c modules against the shared python library would make that static library pretty much useless... |
|
|
| msg76506 - (view) |
Author: (rb) * |
Date: 2008-11-27 17:31 |
| What is the purpose of linking to the static library if you're still going to depend on shared objects in lib-dynload? |
|
|
| msg76510 - (view) |
Author: John Levon (movement) |
Date: 2008-11-27 17:48 |
| Besides, .so files should always declare their dependencies. |
|
|
| msg76518 - (view) |
Author: Ralf Schmitt (schmir) |
Date: 2008-11-27 21:53 |
| Yes, there is no purpose in linking with a static library when the dynamically loaded modules still depend on the shared library. And yes, the dynamically loaded libraries should declare their dependencies. My point is they do *not* depend on the shared library. That's why the static lib currently makes sense on ubuntu, but wouldn't make any sense when your proposal would be implemented. I recommend closing this issue as invalid. |
|
|
| msg76519 - (view) |
Author: Ralf Schmitt (schmir) |
Date: 2008-11-27 22:05 |
| Linking with the static library really works. You have to link with '-Xlinker -export-dynamic' however on linux in order to make the symbols from the static library visible. (just in case if I haven't been clear..) |
|
|
| msg76521 - (view) |
Author: (rb) * |
Date: 2008-11-28 09:10 |
| The problem, and the reason for the existence of this bug, is that I cannot build a shared object that links to libpython2.5.so.1 and works. Please don't mark this bug invalid until this problem is fixed. My proposal of adding dependencies to lib-dynload/*.so is just a suggestion on how to go about fixing this bug. If this is not a suitable solution, please suggest a solution that is. In response to your discussion on why the static library is currently useful, you haven't really answered the question. In what circumstance would someone actually want to build against the static library and then dynamically load other libraries? "Because it works" isn't an answer, as if this bug were fixed, then building against the shared library would validly have the same response. Why static over dynamic, if when static you have to link to dynamic anyway? |
|
|
| msg76536 - (view) |
Author: John Levon (movement) |
Date: 2008-11-28 15:17 |
| If Ubuntu wants to deliver a static libpython, it also needs to deliver static versions of the Python modules, or accept that they're not usable from a static libpython. It makes no sense at all to mix libpython.a with these .so files. Many of them declare dynamic dependencies on stuff like libkrb5.so or libsqlite3.so. I can't conceive of a scenario where you need a static libpython AND you're OK with both dlopen() of the Python modules and any of its dynamic dependencies. What is this case? |
|
|
| msg76539 - (view) |
Author: Ralf Schmitt (schmir) |
Date: 2008-11-28 16:25 |
| no john, linking with the static library really works. the resulting executable does not depend on the shared library and it is possible to import the e.g. the time module. |
|
|
| msg76540 - (view) |
Author: (rb) * |
Date: 2008-11-28 16:58 |
| Ralf, I'm sure it does work, but what is the point of linking statically to libpython.a but then having other dependencies, for example on lib-dynload/time.so? Why not just link to libpython2.5.so in the first place? |
|
|
| msg84336 - (view) |
Author: Matthias Klose (doko) *  |
Date: 2009-03-28 22:06 |
| > I'm sure it does work, but what is the point of linking statically > to libpython.a but then having other dependencies, for example on > lib-dynload/time.so? Why not just link to libpython2.5.so in the > first place? speed. Using a python executable with a statically linked libpython is about 10% faster on ix86, plus with some compilers on linux profile based optimization can be used on static objects, which doesn't work with shared libraries (at least on linux using gcc). |
|
|
| msg88152 - (view) |
Author: Yonas (yonas) |
Date: 2009-05-21 13:01 |
| Ralf, Linking against the static library (with '-Xlinker -export-dynamic') did _not_ solve the problem for me. gcc `python2.6-config --cflags` -c -fPIC mylib.c -o mylib.o -Xlinker -export-dynamic gcc -Xlinker -export-dynamic -lpthread -ldl -lutil -lm -lz -shared -Wl,-soname,mylib.so -o mylib.so mylib.o /usr/lib/python2.6/config/libpython2.6.a gcc `python2.6-config --cflags` -Xlinker -export-dynamic -ldl -lutil -lm -lz -o main main.c /usr/lib/python2.6/config/libpython2.6.a ---------------------- `python2.6-config --cflags` returns -I/usr/include/python2.6 -I/usr/include/python2.6 -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -------- Ubuntu Jaunty Linux yonas-laptop 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux |
|
|
| msg88153 - (view) |
Author: Yonas (yonas) |
Date: 2009-05-21 13:03 |
| Same error message: import dl: ImportError: /usr/lib/python2.6/lib-dynload/dl.so: undefined symbol: PyExc_ValueError |
|
|
| msg88154 - (view) |
Author: Yonas (yonas) |
Date: 2009-05-21 13:15 |
| Here's my log file (`LD_DEBUG=all ./main`) |
|
|
| msg88155 - (view) |
Author: Yonas (yonas) |
Date: 2009-05-21 14:12 |
| Updated log file. |
|
|
| msg88160 - (view) |
Author: Yonas (yonas) |
Date: 2009-05-21 15:13 |
| rb's workaround works, though. |
|
|
| msg91096 - (view) |
Author: Michael Gorven (mgorven) |
Date: 2009-07-30 14:01 |
| Note that on Ubuntu Hardy at least, the libpython2.5.so symlink only exists in the python2.5-dev package. This means that in order for the library to work without the dev package installed, it should dlopen libpython2.5.so.1 instead. |
|
|
| msg93102 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2009-09-25 08:39 |
| Does it help if you add the --whole-archive option when linking with the static library? As explained there: http://www.lysium.de/blog/index.php?/archives/222-Lost-static-objects-in-static-libraries-with-GNU-linker-ld.html This way your main program will contain the whole python interpreter, and all symbols used by time.so &co will be resolved. |
|
|
| msg112907 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2010-08-04 22:03 |
| It is not obvious that this issue is about a Python bug, rather than how to use on a specific system, or that it applies to any current version. If this really is a Python bug and applies to a current version, respond to re-open. |
|
|
| msg112932 - (view) |
Author: John Levon (movement) |
Date: 2010-08-05 00:29 |
| Terry, what in the above discussion leads you to believe there is no bug? The original complaint "Python cannot be embedded in shared library due to dependency problems with lib-dynload" is still valid. |
|
|
| msg112939 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2010-08-05 01:37 |
| I carefully and specifically questioned the existence of current 'Python bug', which is what this tracker is for. For Linux, that would mean a bug in any of the source files in a current 2.7, 3.1.2, or 3.2.1a1 tarball release or the newer source repository versions thereof. But I do not see any claim that there is any problem with any of the files. There certainly is no bug report for any specific stdlib module, which is what is meant by the Library (Lib) component. This issue *is* about lib-dynload that I gather is supplied by Ubuntu. There may or may not be a bug in how Ubuntu packages, compiles, and loads Python 2.5.2. John, your , beginning "If *Ubuntu*..., it [Ubuntu] also needs..." suggests that. But unless you *know* that that results from a bug in the files downloaded by Ubuntu, you should take that up with them. So I will follow Ralf's suggestion and close this, and suggest in turn that this issue be taken to an Ubuntu support list where help is more likely. There might even be knowledgeable Ubuntu users on python-list, I don't know. Another possible suggestion to the OP is compile everything yourself with whatever compile flags you need to get all your stuff to work together. In particular, can you make your own lib-dynload that acts the way you want it to work? (But here is not the place to ask about how.) |
|
|
| msg112943 - (view) |
Author: (rb) * |
Date: 2010-08-05 02:18 |
| Original reporter here. lib-dynload is part of Python's dynamic loading mechanism. Perhaps somewhere like Python/dynload_dl.c is the relevant code? Lib seemed like the right place to put the bug at the time, since it was the stdlib module files that were missing dependency declarations. Although perhaps Extension Modules would have been better? In any case, I've just tried to reproduce the bug with 2.7 (on Debian Squeeze pulling version 2.7-2 of python2.7 and python2.7-dev from experimental), and failed (both 32- and 64-bit). So it seems that this bug has been fixed at some point since 2.5. |
|
|
| msg112944 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2010-08-05 02:47 |
| > perhaps Extension Modules would have been better? For future reference (like if some 3.x is not ok), yes > So it seems that this bug has been fixed at some point since 2.5. Great! |
|
|
| msg208377 - (view) |
Author: Björn Lindqvist (Björn.Lindqvist) |
Date: 2014-01-18 00:34 |
| The bug is still present on the default build which many distros uses which produces an invalid build. That means ./configure && make && make install is broken! --enable-shared should be turned on by default because many (most?) distros build with the default settings. |
|
|