[Python-Dev] Integrating Expat (original) (raw)

Martin v. Loewis martin@loewis.home.cs.tu-berlin.de
Tue, 9 Oct 2001 03:55:15 +0200

What? I've never seen a .a installed on a system that isn't usable. What makes Python modules so special that they could not use a library? I find your statement rather surprising, and I'm quite doubtful of the issue.

I've linked .a files into .so modules often. Never a problem.

It happens on Solaris with gcc as the compiler and /usr/ccs/bin/ld as the linker. Watch this

pandora loewis 39 ( ~/tmp ) > cat a.c int i; int f(){return i;} pandora loewis 40 ( ~/tmp ) > gcc -c a.c pandora loewis 41 ( ~/tmp ) > gcc -o a.so -shared a.o Text relocation remains referenced against symbol offset in file i 0x4 a.o i 0x8 a.o ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status

The problem here is that gcc passes -z text to the linker, unless -mimpure-text is specified. ld(1) then complains if the code was not compiled with -fPIC/-KPIC, i.e. that relocations remain.

Python is not special at all in this respect; that happens every time when you integrate a non-PIC static library (or object file) into a shared library, and

If the linker is binutils ld, it won't happen since the linker ignores -z text. If the compiler is SUNWspro, it won't happen because cc does not pass -z text by default. On a Linux system, it won't happen because relocations against the text segment are not considered a problem in the first place - COW will take care of that.

Of course, relocations against the text segment are a problem: they cause the shared library not to be shared at run-time.

I don't know whether the problem is specific to Python; perhaps we deal with more varying configurations than anybody else. Please see the bug reports #232783, #404322, and #405329 for prior reports of this issue.

> - if the installed version of expat is older than the one we ship, > I think Python should still use the included one.

Every time you use the included one, you run into the possibility of symbol conflict problems. If you refuse to use the one installed in the system, then you increase the chance of problems.

How would such a conflict occur using the current mechanisms for dynamic loading in Python?

> I think the problem you had with Apache does not apply to Python: > > > Apache 1.3 would get loaded and export the Expat symbols to the rest > > of the process. > > In Python, symbols of one extension module are not exported to > anything else.

Eh? Again: what makes Python so special? How can you guarantee this rule? On all platforms?

Unlike everybody else, Python does not use RTLD_GLOBAL in the flags passed to dlopen. This alone is a guarantee that shared libraries won't interfere with each other:

Optionally, RTLD_GLOBAL may be or'ed with flag, in which case the external symbols defined in the library will be made available to subsequently loaded libraries. [from the Linux dlopen man page]

It will work on all systems using dynload_shlib, plus it will work on Windows (where you have to explicitly specify the symbols to export and the libraries you import symbols from). Please also note the Misc/HISTORY item

The specific problem was that somebody was using a library that exposed an initsocket function, which conflicted with the init function of the socket module. So we cannot do much for systems that cannot protect independent shared libraries from each other.

This argument doesn't hold. dlopen() is not used on every platform. Thus, you cannot specify RTLDLOCAL on all platforms, thus you cannot guarantee that symbols in a library will remain private.

Systems that do not support local symbols in shared libraries have bigger problems than competing libexpat implementations. BTW, which of the systems in today's use do not have dlopen, and aren't Windows? [How does MacOS X operate in this area?]

Incorporating Expat directly into Python will definitely be "nice", but I've been through this already. It isn't pretty. If that shared library exists, then use the darned thing, or you're gonna get burned.

OTOH, I'm not looking forward to bug reports which turn out problems with obsolete libexpat installations. You have to judge the risks. Any user who is concerned about using the installed libexpat can still modify Modules/Setup to use that instead of the one coming with Python.

Regards, Martin