[Python-Dev] Bug in build system for cross-platform builds (original) (raw)

Martin Panter vadmium+py at gmail.com
Fri Mar 11 19:48:07 EST 2016

On 11 March 2016 at 23:16, Russell Keith-Magee <russell at keith-magee.com> wrote:

On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium+py at gmail.com> wrote: make clean # Work around confusion with existing in-source build mkdir native (cd native/ && ../configure) make -C native/ Parser/pgen mkdir mingw (cd mingw/ && ../configure --host=i486-mingw32 --build=x86) make -C mingw/ PGEN=../native/Parser/pgen

Actually it was not as smooth as the above commands, because pgen tends to get overwritten with a cross-compiled version. Perhaps we could add a PGENFORBUILD override, like HOSTPGEN in the patch used at <https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html>. That might fix the pgen problem, but freezeimportlib still remains. I suppose the same thing might be possible for freezeimportlib as well…

Yes. I never got up to it failing in my experiments, but I think I would propose a FREEZE_IMPORTLIB override variable for that too.

Would it be more correct to say instead that in 3.4 you did a separate native build step, precompiling pgen and freezeimportlib for the native build host? Then you hoped that the child Make was not invoked in the cross-compilation stage and your precompiled executables would not be rebuilt?

Yes - as far as I can make out (with my admittedly hazy understanding), that appears to be what is going on. Although it’s not that I “hoped” the build wouldn’t happen on the second pass - it was the behavior that was previously relied, and on was altered.

Do you have a copy/patch/link/etc to the actual commands that you relied on? It’s hard to guess exactly what you were doing that broke without this information.

> As best as I can work out, the solution is to: > > (1) Include the parser generator and freezeimportlib as part of the > artefacts of local platform. That way, you could use the version of pgen > and > freezeimportlib that was compiled as part of the local platform build. > At > present, pgen and freezeimportlib are used during the build process, > but > aren’t preserved at the end of the build; or

I don’t understand. After I run Make, it looks like I get working executables leftover at Programs/freezeimportlib and Parser/pgen. Do you mean to install these programs with “make install” or something? Making them part of the installable artefacts would be one option, but they don’t have to be installed, just preserved.

What commands are you running that cause them to not be preserved at the end of the build?

For example, as a nasty hack, I’ve been able to use this approach to get the build working for 3.5. After the native build, I copy freezeimportlib to a “safe” location. I then copy it back into place prior to the target build. It works, but it’s in no way suitable for a final build.

> (2) Include some concept of the “local compiler” in the build process, > which > can be used to compile pgen and freezeimportlib; or On the surface solution (2) sounds like the ideal fix. But I guess the local native compiler might also require a separate set of CPPFLAGS, pyconfig.h settings etc. In other words it is sounding like a whole separate “configure” run. I am thinking it might be simplest to just require this native “configure” run to be done manually. That run is going to happen anyway, since you have to compile and build for the native platform.

More information about the Python-Dev mailing list