| msg97397 - (view) |
Author: Ned Deily (ned.deily) *  |
Date: 2010-01-08 09:06 |
| r77031 (trunk) and r77032 (py3k) for Issue6834 enhanced the the OS X python interpreter launcher, python/pythonw, to allow user-selection of the interpreter execution architecture for multiple architecture builds, i.e. 32-bit vs 64-bit, by using the enhanced capability of the posix_spawn system call instead of execv. However, posix_spawn does not exist prior to OS X 10.5 and it is still important for the OS X installer to support builds and execution on 10.4 and earlier systems. The solution here is to modify Mac/Tools/pythonw.c to conditionally compile calls to execv for builds with a deployment target of 10.4 or earlier and use posix_spawn for deployments of 10.5 or above. Also, correct 32-bit arch names added in configure/configure.in: for 10.4, they must be the more general "ppc", not "ppc7400". APPLIES py3k, trunk |
|
|
| msg97399 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2010-01-08 09:18 |
| Would it be possible to make this a runtime choice? I.e. use posix_spawn when available, and exec otherwise? |
|
|
| msg97400 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-08 09:23 |
| Not cleanly, I think. The problem is there's no spawn.h in the 10.4u SDK so it can't be built on 10.4 without supplying a copy of a system header file. |
|
|
| msg97401 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-08 09:25 |
| Also, the new functionality is really only needed for 32-bit/64-bit multi-arch builds which aren't support with the 10.4u SDK anyway. |
|
|
| msg97402 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2010-01-08 09:31 |
| To elaborate, it should be possible to declare the posix_spawnp functions as __attribute__((weak)), and then test at run-time whether the function (pointers) are null. See http://tinyurl.com/y8myawg for an example. |
|
|
| msg97403 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-01-08 09:42 |
| Ned: the new functionality is also needed for 2-way univeral binaries, it makes pythonw behave much more as if you execute the real interpreter instead of a stub executable. That posix_spawn doesn't exist sucks, and I'm a bit annoyed with myself for not noticing that before the commit. It is pretty easy to create executables on 10.5 that use posix_spawnv when available using weak linking (we already do that in posixmodule.c for other functionality). That won't work for builds on 10.4, or linking to the 10.4u SDK, if the posix_spawn symbols aren't present there: weak linking still has to resolve the symbol at link-time and only ensures that the executable won't fail to start when the symbol isn't present at runtime. I don't have the 10.4u SKD on my machine at the moment and cannot investigate further at the moment, but will do ASAP. Another thing I'd like to do before 2.7 is released is to see if it is possible to perform the build using the default SDK and patch distutils to use the 10.4u SDK when building extensions for a universal build on a n OSX 10.4 system (because the default SDK on those systems cannot build universal binaries). |
|
|
| msg97404 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-01-08 09:46 |
| BTW. The patch is incorrect in it current form: * The change to LIPO_32BIT_FLAGS is unconditional, the current values are needed to build on modern system, I guess the proposed new value would be needed for building on 10.4? * The patch changes pythonw to use execv unconditionally when building a binary that should work on 10.4, as noted in the discussion it is easy enough to build an executable that uses posix_spawnv when it is available and falls back to execv when it is not. A configure check can be used to detect if posix_spawnv is available at link-time. |
|
|
| msg97405 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-08 10:03 |
| As far as I can see, the only possible shortcoming of the patch is that it restores the current behavior with a 2-way fat ppc/i386 build with 10.4u (i.e. the way python.org installers are currently built); that is, you would still not be able to use "arch" to select PPC-emulation on an Intel machine when a 10.4 SDK build is installed on a 10.5 or later Intel system. That's no different than what we have today and I don't know of any demand for that feature. There are other fairly easy ways around it, if really necessary. But perhaps I'm overlooking something. On the other hand, I don't see anything wrong with going the weak linking route (thanks for the pointer, Martin), either, other than the additional effort and complexity. |
|
|
| msg97409 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-08 10:58 |
| "* The change to LIPO_32BIT_FLAGS is unconditional, the current values are needed to build on modern system, I guess the proposed new value would be needed for building on 10.4?" Ah, yes, sorry. I built and tested with and on 10.6 with 10.5 SDK/gcc-4.0, on 10.5 with gcc-4.0, and on 10.4u/gcc-4.0 but I see I did overlook trying 3-way on 10.6 with 10.6 SDK/gcc-4.2, where ppc7400 is indeed needed. Arch ppc is definitely needed for building on 10.6 and 10.5 with the 10.4u SDK and gcc-4.0: lipo: -extract ppc7400 specified but fat file: pythonw does not contain that architecture If you don't get to it first, I'll fix and resubmit the patch later. |
|
|
| msg97437 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2010-01-08 20:10 |
| > As far as I can see, the only possible shortcoming of the patch is > that it restores the current behavior with a 2-way fat ppc/i386 build > with 10.4u (i.e. the way python.org installers are currently built); > that is, you would still not be able to use "arch" to select > PPC-emulation on an Intel machine when a 10.4 SDK build is installed > on a 10.5 or later Intel system. That's no different than what we > have today and I don't know of any demand for that feature. There > are other fairly easy ways around it, if really necessary. For users, what matters are really the binaries provided by python.org (and those provided by Apple). So if you restore the previous behavior, that's basically "for good" - that you still have the compile option to support arch won't matter for users. Now, ISTM that users do indeed request the ability to chose between a 32-bit interpreter and a 64-bit interpreter at launch time. I know that they do on Windows (just because you can), and I would be puzzled if there weren't OSX users that wanted it one way or the other (depending on the application you intend to run). |
|
|
| msg97448 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-09 01:02 |
| Martin, just to be clear: the purpose of the new feature *is* to allow the choice between 32-bit/64-bit and that is important. My comment was that the downside of the submitted fix, as it stands, would be to not allow choosing archs only for pythons built with the 10.4u SDKs. And with the 10.4u SDK, those pythons are and can only be 32-bit so the only arch choice there would be between 32-bit Intel and 32-bit PPC and the only environment where that could mean anything is on a 10.5 or 10.6 Intel machine where, in theory, it should be possible to run 32-bit PPC pythons in emulation mode. For pythons built with 10.5 or 10.6 SDKs (in as yet to be supplied python.org installers), then the arch feature allows choice among the up-to 4 archs in the build: 32/64, Intel/PPC. The proposed fix doesn't alter that aspect of the new feature at all. But the patch needs to be redone anyway so, if the weak linking is added, there wouldn't even be that very small drawback. OS X multi-architecture executables: complicated! |
|
|
| msg97616 - (view) |
Author: Sridhar Ratnakumar (srid) |
Date: 2010-01-12 01:03 |
| Our nightly ActivePython builds broke because of this issue. Note that we still build on 10.4 w/ 10.4u SDK. I should be able to help verify the new fix - once it is uploaded - in our build machine. |
|
|
| msg97626 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-01-12 09:14 |
| I've attached a patch that should fix the build issues with the 10.4 SDK. The patch touches configure.in, run autoconf and autoheader after applying the patch. I haven't tested the patch yet beyond compilation on 10.6 system without the 10.4 SDK. This patch does not attempt to fix the issues Ned has with LIPO_32BIT_FLAGS, I will look into that later today. |
|
|
| msg97687 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-13 05:26 |
| Thanks, Ronald. The patch looks good. I've got a patch in progress for the LIPO flags part: looks like the key is 'ppc' for DEPLOYMENT_TARGET <= 10.4 and 'ppc7400' for DEPLOYMENT_TARGET >= 10.5 for either gcc-4.0 or -4.2. I'll test the two together on the various platforms tomorrow. |
|
|
| msg97689 - (view) |
Author: Sridhar Ratnakumar (srid) |
Date: 2010-01-13 05:46 |
| Ronald, with your patch I see this on 10.4 (Xcode 2.5): [...] lipo -extract ppc7400 -extract i386 -output /Users/apy/rrun/build/activepython-DEV/build/pyhg_trunk-macosx-hgtip27-rrun/image/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-32 pythonw lipo: -extract ppc7400 specified but fat file: pythonw does not contain that architecture [...] |
|
|
| msg97700 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2010-01-13 08:33 |
| @Sridhar: that's the part that's not fixed yet. We'll have a patch for that shortly. |
|
|
| msg97701 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-01-13 09:30 |
| Ned and Sridhar: IMO we need a configure test to detect which argument should be used to extract ppc code (ppc or ppc7400). |
|
|
| msg97955 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-01-17 16:28 |
| I've committed a fix for this in r77585, please test. I can now compile python-trunk on OSX 10.6 while targetting the 10.4 SDK, and have compiled on OSX 10.4 as well. I will forward port this to 3.2. |
|
|
| msg98127 - (view) |
Author: Sridhar Ratnakumar (srid) |
Date: 2010-01-22 02:24 |
| > RONALD: I've committed a fix for this in r77585, please test. Verified on Tiger with 10.4u SDK. |
|
|
| msg99013 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-02-07 20:07 |
| This is now fixed in the 3.2 branch as well. |
|
|