Issue 9046: Python 2.7rc2 doesn't build on Mac OS X 10.4 (original) (raw)

Issue9046

Created on 2010-06-21 18:35 by lemburg, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (19)
msg108294 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 18:35
The RC2 builds fine on Mac OS X 10.6 (Snow Leopard), but fails to build any of the required extension modules on 10.3: Python build finished, but the necessary bits to build these modules were not found: _bsddb gdbm linuxaudiodev ossaudiodev readline spwd sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules: _AE _AH _App _bisect _CarbonEvt _CF _CG _Cm _codecs_cn _codecs_hk _codecs_iso2022 _codecs_jp _codecs_kr _codecs_tw _collections _csv _Ctl _ctypes _ctypes_test _curses _curses_panel _Dlg _Drag _elementtree _Evt _File _Fm _Folder _functools _hashlib _heapq _Help _hotshot _IBCarbon _Icn _io _json _Launch _List _locale _lsprof _Menu _Mlte _multibytecodec _multiprocessing _OSA _Qd _Qdoffs _Qt _random _Res _Scrap _sha256 _sha512 _Snd _socket _sqlite3 _ssl _struct _TE _testcapi _tkinter _weakref _Win array audioop autoGIL binascii bsddb185 bz2 cmath ColorPicker cPickle crypt cStringIO datetime dbm dl fcntl future_builtins gestalt grp icglue imageop itertools MacOS math mmap Nav nis operator OSATerminology parser pyexpat resource select strop syslog termios time unicodedata zlib Example: building '_struct' extension gcc-4.0 -fno-strict-aliasing -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Mac/Include -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Include -I/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2 -c _struct.c -o build/temp.macosx-10.4-fat-2.7/_struct.o powerpc-apple-darwin8-gcc-4.0.1: _struct.c: No such file or directory powerpc-apple-darwin8-gcc-4.0.1: no input files i686-apple-darwin8-gcc-4.0.1: _struct.c: No such file or directory i686-apple-darwin8-gcc-4.0.1: no input files lipo: can't figure out the architecture type of: /var/tmp//cckP7ZEq.out ... These were the used configure options: ./configure --prefix=/usr/local/python-2.7-ucs2 --enable-unicode=ucs2 --enable-universalsdk MACOSX_DEPLOYMENT_TARGET=10.4
msg108298 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 19:08
Some debugging shows that the ext.sources list in setup.py does not include the "Modules/" prefix for the source files: *** moddirlist=['/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Modules', '/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Mac/Modules'] but: *** Extension _struct: ext.sources=['_struct.c'] *** Extension _ctypes_test: ext.sources=['_ctypes/_ctypes_test.c'] *** Extension _weakref: ext.sources=['_weakref.c'] *** Extension array: ext.sources=['arraymodule.c'] ...
msg108301 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 19:21
Turns out that find_file() always returns None for the shared mods: *** module _struct.c in ['/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Modules', '/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Mac/Modules']: find_file returned None *** Extension _struct: ext.sources=['_struct.c']
msg108303 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 19:30
Instrumenting find_file() a bit: if sys.platform == 'darwin': # Honor the MacOSX SDK setting when one was specified. # An SDK is a directory with the same structure as a real # system, but with only header files and libraries. sysroot = macosx_sdk_root() # Check the standard locations for dir in std_dirs: f = os.path.join(dir, filename) if sys.platform == 'darwin' and is_macosx_sdk_path(dir): f = os.path.join(sysroot, dir[1:], filename) print '*** checking standard location %s' % f if os.path.exists(f): return [] this gives: *** checking additional location /Developer/SDKs/MacOSX10.4u.sdk/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Modules/_struct.c *** checking additional location /Developer/SDKs/MacOSX10.4u.sdk/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Mac/Modules/_struct.c *** module _struct.c in ['/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Modules', '/usr/local/src/egenix-build-environment/Python-2.7rc2-ucs2/Mac/Modules']: find_file returned None *** Extension _struct: ext.sources=['_struct.c'] So the reason for the failure is that is_macosx_sdk_path(dir) doesn't do what it's probably supposed to do, or the code assumes that it's getting relative paths in moddirslist rather the absolute paths it is getting.
msg108304 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 19:34
Note that I've not checked whether an SDK build works on Mac OS X 10.6. The regular build does work. The problem appears to be related to SDK builds only, e.g. if you plan to build Universal binary on Mac OS X 10.3.
msg108315 - (view)	Author: Ned Deily (ned.deily) *	Date: 2010-06-21 20:48
As far as I know, it is not supported to "upbuild" Python (or anything else) on OS X, i.e. you cannot use a higher level SDK (i.e. 10.4u) to build on an earlier system (i.e. 10.3). In particular, to build Python with a deployment target of 10.3, you need to build on 10.4 or higher with the 10.4u SDK.
msg108316 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-21 20:54
Sorry, my bad. The system in question is a 10.4 Tiger system.
msg108338 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-22 06:13
I'll fire up my 10.4 system to further investigate this.
msg108372 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-22 13:22
Marc-Andre: what version of Xcode do you use? (the version in the About menu of Xcode.app). I'm getting clean builds with '--enable-universalsdk' on OSX 10.4.11 (PPC) with Xcode 2.5 (the last Xcode for OSX 10.4). That is, all extension build, except for the ones grouped with '_bsddb' in your initial message (and that's expected). Furthermore all tests passed. I had one oddity in the tests: make test prints "Warning -- sys.path was modified by test_site". I don't get this warning on OSX 10.6. BTW. universal builds work fine on OSX 10.6, that's how I do most of my development.
msg108383 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-22 14:17
Ronald Oussoren wrote: > > Ronald Oussoren <ronaldoussoren@mac.com> added the comment: > > Marc-Andre: what version of Xcode do you use? (the version in the About menu of Xcode.app). We have Xcode 2.5 and all updates on the machine. Python 2.6 and older versions compile just fine. The changes you added for the SDK builds in Python 2.7 made the problem appear. What I don't understand is why you are redirecting files under /usr to the SDK virtual root dir. We install all the local builds under /usr/local/ and as result, the build itself also happens under a /usr path. The function definition appears to be a bit coarse in this respect: def is_macosx_sdk_path(path): """ Returns True if 'path' can be located in an OSX SDK """ return path.startswith('/usr/') or path.startswith('/System/') I believe that this function should really only return True if the path in question does exists in the SDK virtual root.
msg108386 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-22 14:32
What I don't quite understand is why the build fails for you but passes for me. What configure flags did you use? This version should fare better: def is_macosx_sdk_path(path): """ Returns True if 'path' can be located in an OSX SDK """ return (path.startswith('/usr/') and not path.startswith('/usr/local')) or path.startswith('/System/') This explicitly tests for paths that must be in the SDK: * Anything in /System is owned by the system, and should be fetched through the SDK * Likewise for anything in /usr that isn't in /usr/local IMHO anyone that installs additional libraries in /usr/lib, or /System/Libraries/Frameworks is confused at best, and we shouldn't even try to support that. The repository contains an simpler (but in hindsight too simple) version because ${SDKROOT}/usr/local/lib is a symlink to the real /usr/local/lib. That works fine when looking for libraries, but not when looking for other files (such as headers).
msg108399 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-22 17:46
Ronald Oussoren wrote: > > Ronald Oussoren <ronaldoussoren@mac.com> added the comment: > > What I don't quite understand is why the build fails for you but passes for me. What configure flags did you use? I posted the configure options in the first message on this ticket. The failure appears to be due to the fact that we run and install our software in /usr/local (which is the standard we use on all Unix systems). > This version should fare better: > > def is_macosx_sdk_path(path): > """ > Returns True if 'path' can be located in an OSX SDK > """ > return (path.startswith('/usr/') and not path.startswith('/usr/local')) or path.startswith('/System/') With this variant, the build works correctly, however, see below... > This explicitly tests for paths that must be in the SDK: > > * Anything in /System is owned by the system, and should be fetched > through the SDK > * Likewise for anything in /usr that isn't in /usr/local > > IMHO anyone that installs additional libraries in /usr/lib, or /System/Libraries/Frameworks is confused at best, and we shouldn't even try to support that. ...wouldn't it be better to check whether the SDK does indeed provide the path in question and only then redirect to the SDK path instead of the normal system one ? I'm not sure whether these will ever be needed by Python, but it would certainly make the function more robust: There are quite a few things in /usr/lib that you don't find in the SDK usr/lib/ dir. For /System the situation is even more obvious: the SDK version only has these entries: # ls -l /Developer/SDKs/MacOSX10.4u.sdk/System/Library/ total 0 drwxr-xr-x 3 root wheel 102 Oct 27 2007 CFMSupport drwxr-xr-x 92 root wheel 3128 Oct 27 2007 Frameworks drwxr-xr-x 3 root wheel 102 Oct 27 2007 Printers drwxr-xr-x 123 root wheel 4182 Oct 27 2007 PrivateFrameworks whereas the OS directory has dozens of entries. > The repository contains an simpler (but in hindsight too simple) version because ${SDKROOT}/usr/local/lib is a symlink to the real /usr/local/lib. That works fine when looking for libraries, but not when looking for other files (such as headers). Right, but really only that link. SDK also doesn't contain any links for the missing /System/Library directories.
msg108417 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-22 21:08
The search code must look in the SDK and not fall back on looking into the system to avoid finding new headers and libraries when trying to build using an older SDK on a newer system. I don't think this is a problem at the moment, but it could be in the future. The reason I added the SDK support code was that Snow Leopard contained a newer version of the OpenSSL libraries and the Python build picked up version information from the system version of OpenSSL instead of the SDK, and that resulted in a disfunctional build (IIRC hashlib was incomplete and that caused numerous test failures). Falling back to looking in the current system would be needed when looking for a file that isn't a header of library, but AFAIK we don't do that in setup.py.
msg108423 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-22 21:38
Ronald Oussoren wrote: > > Ronald Oussoren <ronaldoussoren@mac.com> added the comment: > > The search code must look in the SDK and not fall back on looking into the system to avoid finding new headers and libraries when trying to build using an older SDK on a newer system. Right, but at least there should be an option to fall back to the system provided libs. This is currently not the case. The code would need some refactoring to make this possible, though, e.g. a combined version of the SDK functions you added to return the corrected path instead of just True/False. > I don't think this is a problem at the moment, but it could be in the future. > > The reason I added the SDK support code was that Snow Leopard contained a newer version of the OpenSSL libraries and the Python build picked up version information from the system version of OpenSSL instead of the SDK, and that resulted in a disfunctional build (IIRC hashlib was incomplete and that caused numerous test failures). > > Falling back to looking in the current system would be needed when looking for a file that isn't a header of library, but AFAIK we don't do that in setup.py. I guess you should at least add a note pointing to this potential future problem to the code and perhaps also reference this ticket.
msg108434 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-23 05:48
I don't agree that there must be an option to fall back to system provided libs. The point of using an SDK is to avoid doing that because you might end up with a binary that won't work on an earlier version of the OS (the OpenSSL one is an example of that). I agree that the documentation/comments should be extended to not that additional work would be needed when we start looking for files that aren't headers or libraries. BTW. I still don't quite understand why the build did fail for you in the first place. Is your source tree in /usr/local as well?
msg108436 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-23 07:29
Ronald Oussoren wrote: > > Ronald Oussoren <ronaldoussoren@mac.com> added the comment: > > I don't agree that there must be an option to fall back to system provided libs. The point of using an SDK is to avoid doing that because you might end up with a binary that won't work on an earlier version of the OS (the OpenSSL one is an example of that). If you don't and the SDK doesn't include the files Python is looking for, then the build will fail, so I don't see an improvement in not doing so ;-) Also note that a user may not have a require to be able to use the built Python on some previous version of the OS. Note that if you use MacPorts it's fairly common to use e.g. use their Tcl version for Python which resides in /Library as well. The SDK logic should not redirect such paths to the SDK were they don't exist. > I agree that the documentation/comments should be extended to not that additional work would be needed when we start looking for files that aren't headers or libraries. > > BTW. I still don't quite understand why the build did fail for you in the first place. Is your source tree in /usr/local as well? Yes, we build and install from /usr/local/src - as is standard for Unix platforms. I know that Mac OS X is different in some way, but at least the Mac ports collection uses the same approach.
msg108439 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-23 08:24
That (/usr/local/src) explains why I haven't been able to reproduce the problem, that worried me a little. W.r.t. to the SDK: 1) You don't have to use an SDK: use configure --enable-universalsdk=/ MACOSX_DEPLOYMENT_TARGET=10.5 (or whatever target you wish to support) 2) The SDK should only affect system files, that is anything in /System and /usr (except for /usr/local). /Library is not part of the SDK and is not affected by SDK settings.
msg108440 - (view)	Author: Marc-Andre Lemburg (lemburg) *	Date: 2010-06-23 08:35
Ronald Oussoren wrote: > > Ronald Oussoren <ronaldoussoren@mac.com> added the comment: > > That (/usr/local/src) explains why I haven't been able to reproduce the problem, that worried me a little. > > W.r.t. to the SDK: > > 1) You don't have to use an SDK: use > > configure --enable-universalsdk=/ MACOSX_DEPLOYMENT_TARGET=10.5 > > (or whatever target you wish to support) Well, we want to build universal binaries, so we do need the SDK. > 2) The SDK should only affect system files, that is anything in /System > and /usr (except for /usr/local). /Library is not part of the SDK > and is not affected by SDK settings. Sorry, I should have written /System/Library/. You find Tcl in /System/Library, but not in /Developer/SDKs/MacOSX10.4u.sdk/System/Library/ Anyway, this doesn't appear to matter, since setup.py picks up the files from a different dir: /System/Library/Frameworks/Tk.framework and that is included in the SDK. As I said: The fix makes the build work, so it's good enough for now. In the future all this may will have to be revisited.
msg108782 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-06-27 12:40
I've committed a fix in r82272 (2.7), r82273 (3.2), r82274 (2.6), r82273 (3.1)