Issue 15102: Fix 64-bit building for buildbot scripts (3.3) (original) (raw)
Created on 2012-06-18 14:48 by jkloth, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (47)
Author: Jeremy Kloth (jkloth) *
Date: 2012-06-18 14:48
The buildbot scripts do not work for the 64-bit targets.
Firstly, the "/p:UseEnv=True" parameter to msbuild causes the 32-bit only projects (make_buildinfo and make_versioninfo) to fail due to architecture mismatch. The scripts now unconditionally build those projects using the 32-bit command-line tools.
The make_versioninfo project wasn't completely converted to be 32-bit only (); it would work correctly only from the IDE. This fixes that and removes the Debug configuration as that was already #ifdef'd in the generated .h file.
Author: Stefan Krah (skrah) * 
Date: 2012-06-18 20:42
There's some unrelated stuff in the patch that reverts commits in mpdecimal.c and multiprocessing. Maybe a Rietveld issue.
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-18 21:20
The unrelated changes do seem to be an issue with the diff generated by Rietveld
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-18 21:27
If I understand, the proposed changes are the following: https://bitbucket.org/jkloth/cpython-buildbot-amd64/changeset/2a20cee18add ?
I'm not a Windows specialist, but they do seem a bit weird:
- the pcbuild.sln changes remove "Debug|x64.Build" and replaces "Debug" with "Release"
- build-common.bat doesn't allow to distinguish between x86 and x64
- build.bat does not call build-common.bat
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-18 22:26
On Mon, Jun 18, 2012 at 3:27 PM, Antoine Pitrou <report@bugs.python.org> wrote:
If I understand, the proposed changes are the following: https://bitbucket.org/jkloth/cpython-buildbot-amd64/changeset/2a20cee18add
That seems to be correct, yes.
I'm not a Windows specialist, but they do seem a bit weird:
- the pcbuild.sln changes remove "Debug|x64.Build" and replaces "Debug" with "Release"
That change is related to the make_versioninfo project which doesn't use the Debug build (like make_buildinfo)
- build-common.bat doesn't allow to distinguish between x86 and x64
Correct as make_buildinfo and make_versioninfo are only ever built as 32-bit (see )
- build.bat does not call build-common.bat
Oops, an oversight on my part. It didn't show up in testing as the solution already is built in 32-bit mode. The call should in the same location as in build-amd64.bat
Author: Jeremy Kloth (jkloth) *
Date: 2012-06-20 14:56
Now that my buildbot is up and building (and failing at that) could these changes be committed? It would get the buildbot to at least start testing changes to Python proper.
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-22 20:03
Now that my buildbot is up and building (and failing at that) could these changes be committed?
Well, it would be nice if you could post an updated patch after the comments above. (I'm not saying I'm gonna commit, since I'm not a Windows expert, but it will certainly help other core developers review it)
Author: Georg Brandl (georg.brandl) *
Date: 2012-06-23 15:01
I'd like Martin to have a look at this.
Author: Martin v. Löwis (loewis) *
Date: 2012-06-23 15:09
I think this really needs to be broken further down than "the AMD64 buildbot doesn't work". For each presumed failure, there should be one issue, and perhaps a proposed patch.
The patch, as it stands, is clearly incorrect (as has been reviewed and acknowledged).
So unless Jeremy can provide an updated patch quickly, I rather suggest to close this issue, and ask for new issues to be created for the individual problems.
As for the changes to project files and the solution file: none of these changes should be necessary at all, unless it's actually the case that the AMD64-Debug build doesn't work (i.e. neither in the buildbot, nor from the solution file).
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-23 15:27
The patch, as it stands, is clearly incorrect (as has been reviewed and acknowledged).
The most recent patch is correct.
As for the changes to project files and the solution file: none of these changes should be necessary at all, unless it's actually the case that the AMD64-Debug build doesn't work (i.e. neither in the buildbot, nor from the solution file).
The debug build doesn't even build! That's the issue.
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-23 15:32
As for the changes to project files and the solution file: none of these changes should be necessary at all, unless it's actually the case that the AMD64-Debug build doesn't work (i.e. neither in the buildbot, nor from the solution file).
As to the debug build not working, has anyone even looked at the 64-bit Windows buildbot? As the following link demonstrates, building is clearly failing:
Author: Martin v. Löwis (loewis) *
Date: 2012-06-23 15:56
The most recent patch is correct.
See my review then; I think it has issues.
The debug build doesn't even build! That's the issue.
It builds fine for me - I just tried.
Author: Stefan Krah (skrah) *
Date: 2012-06-23 16:05
Martin v. L??wis <report@bugs.python.org> wrote:
The debug build doesn't even build! That's the issue.
It builds fine for me - I just tried.
What command line do you use? I tried:
C:\Users\stefan\pydev\cpython>.\Tools\buildbot\build-amd64.bat
Error: project file 'pcbuild.sln' was not found or not a valid project file. Microsoft (R) Visual C++ Project Builder - Command Line Version 9.00.21022 Copyright (C) Microsoft Corporation. All rights reserved.
Author: Martin v. Löwis (loewis) *
Date: 2012-06-23 16:10
The debug build doesn't even build! That's the issue.
It builds fine for me - I just tried.
What command line do you use?
Not the command line - I built from the IDE. My claim is if that works, no changes to the solution and project files should be necessary.
C:\Users\stefan\pydev\cpython>.\Tools\buildbot\build-amd64.bat
Error: project file 'pcbuild.sln' was not found or not a valid project file.
I can believe that there are issues with the batch files. That's why I propose to look at one issue at a time (but it's not my priority at the moment).
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-23 16:15
I can believe that there are issues with the batch files. That's why I propose to look at one issue at a time (but it's not my priority at the moment).
That would mean that you do not care that the buildbots are failing. That is exactly the reason that this bug report has been opened.
Author: Georg Brandl (georg.brandl) *
Date: 2012-06-23 16:28
Of course these failures have to be fixed. But what Martin is saying is that for him (and probably other devs working on Windows) the build works, so the change must be done in the scripts only running on the buildbots.
As for priority, we're two days away from 3.3b1 and feature freeze, so you can imagine that there are quite a lot of things on Martin's table.
Author: Stefan Krah (skrah) *
Date: 2012-06-23 16:31
I seem to be getting a functioning amd64 debug build by applying the small useenv.diff and then running:
Tools\buildbot\build-amd64.bat
Author: Martin v. Löwis (loewis) *
Date: 2012-06-23 16:35
This change eliminates the need for HOST_PYTHON. Why is it required in the first place? It is not documented as being required to build Python.
It helps to build 64-bit binaries on a 32-bit Windows installation. Normally, Python uses the python.exe it just built, but this fails if the binary is a 64-bit binary, but the host is 32-bits.
It would be best if HOST_PYTHON was used if set, but it otherwise falls back to using the Python just built.
I'm fine with dropping HOST_PYTHON for 3.4, but not for 3.3.
When building in the VS IDE, the proper compilers are chosen. However, when building via the buildbot scripts, the compilers are forced to the 64-bit only versions and thus cause link errors. See the buildbot status for the exact error messages.
I see the errors, but I believe the solution is incorrect. It must be possible to build from the command line exactly in the same way as from the IDE, and it was possible to do so with VS 2008. We just need to find out how to do it with VS 2010.
This file is a redundant with python_nt.h. It was removed for simplicity.
That is out of scope for this issue, then. If the file annoys you much, create a separate issue.
I do not know what you reviewed, but the diff I generated does not have those changes.
Interesting. Now Rietveld doesn't show them anymore, either, in the place where I left the comment.
Author: Martin v. Löwis (loewis) *
Date: 2012-06-23 16:37
Stefan: your patch looks good, please apply.
Author: Roundup Robot (python-dev) 
Date: 2012-06-23 16:59
New changeset 979567d33376 by Stefan Krah in branch 'default': Issue #15102: Allow platform-specific settings for the current project http://hg.python.org/cpython/rev/979567d33376
Author: Roundup Robot (python-dev)
Date: 2012-06-23 19:08
New changeset a9d4cf7d15b9 by Martin v. Löwis in branch 'default': Issue #15102: Use HOST_PYTHON only if it is set. http://hg.python.org/cpython/rev/a9d4cf7d15b9
Author: Stefan Krah (skrah) *
Date: 2012-06-23 19:18
I've fixed some paths in external-amd64.bat. Uploading a diff instead of committing right away so as not to interfere with the steadily incoming commits.
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-23 19:37
Stefan, don't fear interfering. Merging is easy.
Author: Roundup Robot (python-dev)
Date: 2012-06-23 20:19
New changeset c56783fe2e3b by Stefan Krah in branch 'default': Issue #15102: Fix paths in external-amd64.bat. http://hg.python.org/cpython/rev/c56783fe2e3b
Author: Stefan Krah (skrah) *
Date: 2012-06-23 20:22
Antoine Pitrou <report@bugs.python.org> wrote:
Stefan, don't fear interfering. Merging is easy.
Good to know, I loathe merging. :)
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-23 20:33
There's still a problem when building ssl:
Project "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pcbuild.sln" (1) is building "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild_ssl.vcxproj" (18) on node 1 (default targets). Project "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild_ssl.vcxproj" (18) is building "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\ssl.vcxproj" (19) on node 1 (default targets). Build: cd "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild" "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\python_d.exe" build_ssl.py Release x64 -a
'"C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\python_d.exe"' is not recognized as an internal or external command, operable program or batch file.
Author: Roundup Robot (python-dev)
Date: 2012-06-23 22:22
New changeset 9c71d7f057b3 by Martin v. Löwis in branch 'default': Issue #15102: find python.exe in OutDir, not SolutionDir. http://hg.python.org/cpython/rev/9c71d7f057b3
Author: Stefan Krah (skrah) *
Date: 2012-06-24 10:45
All Windows buildbots are failing now:
C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Microsoft.MakeFile.Targets(38,5): error MSB3073: The command "cd "d:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\PCbuild" [d:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\PCbuild\ssl.vcxproj] C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Microsoft.MakeFile.Targets(38,5): error MSB3073: "d:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\PCbuild\Debug\python_d.exe" build_ssl.py Release Win32 -a [d:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\PCbuild\ssl.vcxproj] C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Microsoft.MakeFile.Targets(38,5): error MSB3073: " exited with code 3. [d:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\PCbuild\ssl.vcxproj]
Perhaps it would be possible to set HOST_PYTHON in build-amd64.bat as a temporary measure until a more elegant solution is found?
Author: Roundup Robot (python-dev)
Date: 2012-06-24 11:04
New changeset fc84ac3e8dfc by Stefan Krah in branch 'default': Issue #15102: Try setting HOST_PYTHON in build-amd64.bat as a temporary http://hg.python.org/cpython/rev/fc84ac3e8dfc
Author: Stefan Krah (skrah) *
Date: 2012-06-24 11:06
I'm trying out my suggestion on the 64-bit bot. Will revert if it doesn't work out.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 11:22
Well, the path to python_d.exe looks right now, but Perl and nasm aren't found:
"C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\amd64\python_d.exe" build_ssl.py Release x64 -a
Found a working perl at 'C:\perl\bin\perl.exe' 'nasm' is not recognized as an internal or external command, operable program or batch file. nasm assembler has failed.
Jeremy, did you install these on your machine?
Author: Martin v. Löwis (loewis) *
Date: 2012-06-24 11:23
I'm trying out my suggestion on the 64-bit bot. Will revert if it doesn't work out.
Even if it does work out it should be reverted. As HOST_PYTHON is not set, PythonExe should be computed correctly. So we really need to find out why it isn't.
Author: Martin v. Löwis (loewis) *
Date: 2012-06-24 11:31
On 24.06.2012 13:22, Stefan Krah wrote:
Stefan Krah <stefan-usenet@bytereef.org> added the comment:
Well, the path to python_d.exe looks right now, but Perl and nasm aren't found:
perl shouldn't be necessary. Not sure why nasm is.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 11:36
Martin v. L??wis <report@bugs.python.org> wrote:
Even if it does work out it should be reverted. As HOST_PYTHON is not set, PythonExe should be computed correctly. So we really need to find out why it isn't.
Yes, of course. It's meant as a temporary measure to work out things like the missing nasm and hopefully get a functioning bot before the beta.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 11:45
Well, the path to python_d.exe looks right now, but Perl and nasm aren't found:
"C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\amd64\python_d.exe" build_ssl.py Release x64 -a
Found a working perl at 'C:\perl\bin\perl.exe'
Hmm, obviously Perl is found. That leaves nasm.
Author: Jeremy Kloth (jeremy.kloth)
Date: 2012-06-24 14:00
Hmm, obviously Perl is found. That leaves nasm.
nasm.exe was installed for myself (hence testing worked). I've now made it available to the buildbot account as well.
Note that I have Perl installed but not on my PATH (on purpose). The patch supplied in will fix the issue brought out by that circumstance.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 14:17
Jeremy Kloth <report@bugs.python.org> wrote:
nasm.exe was installed for myself (hence testing worked). I've now made it available to the buildbot account as well.
Note that I have Perl installed but not on my PATH (on purpose). The patch supplied in will fix the issue brought out by that circumstance.
Thanks. The Perl situation looks OK since the buildbot has proceeded to the testing stage in the last build.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 14:57
Jeremy, your python-dev suggestion to move x64.props before pyproject.props does not work for me. For instance, in python3dll.vcxproj:
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.MakeFile.Targets(38,5): error MSB3073: The comma nd "cd C:\Users\stefan\pydev\cpython\PCbuild\..\PC [C:\Users\stefan\pydev\cpython\PCbuild\python3dll.vcxproj] C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.MakeFile.Targets(38,5): error MSB3073: nmake /f py thon3.mak MACHINE=x64 OutDir=C:\Users\stefan\pydev\cpython\PCbuild\x64-pgo" exited with code 9009. [C:\Users\ stefan\pydev\cpython\PCbuild\python3dll.vcxproj]
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-24 15:50
Ouch. The test failures on the buildbot now seem to point to problems with OpenSSL's SHA1 implementation.
Jeremy, perhaps you can try to run the OpenSSL self-tests?
Author: Stefan Krah (skrah) *
Date: 2012-06-24 16:43
outdir.diff contains a solution that doesn't rely on HOST_PYTHON. Works for both x64 and Win32.
Note that I've edited the files using Vim. -- I'm not sure if one can even select Condition="'$(Platform)'=='Win32'" using Visual Studio.
Author: Martin v. Löwis (loewis) *
Date: 2012-06-24 16:51
Stefan: I think that's acceptable. Perhaps someone will come up with a better solution; if this works, it's good enough for me.
Author: Roundup Robot (python-dev)
Date: 2012-06-24 17:05
New changeset f12a2f2c93e5 by Stefan Krah in branch 'default': Issue #15102: Fix paths of python executables. http://hg.python.org/cpython/rev/f12a2f2c93e5
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-24 19:29
Note that there also appears to be a compile error on the 3.2 and 2.7 branches:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.2 http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%202.7
Build started: Project: ssl, Configuration: Debug|x64 Performing Makefile project actions '""' is not recognized as an internal or external command, operable program or batch file. Project : error PRJ0019: A tool returned an error code from "Performing Makefile project actions" Project : warning PRJ0018 : The following environment variables were not found: $(HOST_PYTHON) Build log was saved at "file://C:\Users\Buildbot\buildbot.python.org\3.2.kloth-win64\build\PCbuild\x64-temp-Debug\ssl\BuildLog.htm" ssl - 1 error(s), 0 warning(s)
Author: Martin v. Löwis (loewis) *
Date: 2012-06-24 19:47
Note that there also appears to be a compile error on the 3.2 and 2.7 branches:
Can we please, pretty pretty please, deal with one issue at a time?
If you want to keep this issue open until everything works, you will probably see a dozen more commits, and wait several more months.
Author: Stefan Krah (skrah) *
Date: 2012-06-24 20:35
As far as I can see, the 3.3 buildbot scripts are fixed: After af1d98b7ca80 I can run Tools/buildbot/build.bat and Tools/buildbot/build-amd64.bat successfully in any order.
I've run the tests, too, and I'm not getting the SHA1 failures.
As requested, I've opened separate issues for the 2.7/3.2 scripts and one for the SHA1 failures on the bot.
Author: Antoine Pitrou (pitrou) *
Date: 2012-06-24 21:14
Hmm, the buildbot fails compiling again after af1d98b7ca801049803d83a85bb8558f6e59f2a1:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/139
Author: Stefan Krah (skrah) *
Date: 2012-06-24 21:20
Antoine Pitrou <report@bugs.python.org> wrote:
Hmm, the buildbot fails compiling again after af1d98b7ca801049803d83a85bb8558f6e59f2a1:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/139
I think it's unrelated (probably ssl maintenance going on):
cd "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild" "C:\Users\Buildbot\buildbot.python.org\3.x.kloth-win64\build\PCbuild\amd64\python_d.exe" build_ssl.py Release x64 -a
Found a working perl at 'C:\perl\bin\perl.exe' Traceback (most recent call last): File "build_ssl.py", line 253, in main() File "build_ssl.py", line 187, in main os.chdir(ssl_dir) PermissionError: [Error 5] Access is denied: '..\..\openssl-1.0.1c' [55050 refs]
History
Date
User
Action
Args
2022-04-11 14:57:31
admin
set
github: 59307
2012-06-24 21:20:01
skrah
set
messages: +
2012-06-24 21:14:56
pitrou
set
messages: +
2012-06-24 20:35:40
skrah
set
status: open -> closed
title: Fix 64-bit building for buildbot scripts (issue 15102) -> Fix 64-bit building for buildbot scripts (3.3)
messages: +
resolution: fixed
stage: resolved
2012-06-24 19:47:07
loewis
set
messages: +
2012-06-24 19:29:26
pitrou
set
messages: +
2012-06-24 17:05:52
python-dev
set
messages: +
2012-06-24 16:51:00
loewis
set
messages: +
2012-06-24 16:43:30
skrah
set
files: + outdir.diff
messages: +
2012-06-24 15:50:41
pitrou
set
messages: +
2012-06-24 14:57:45
skrah
set
messages: +
2012-06-24 14:17:08
skrah
set
messages: +
2012-06-24 14:00:09
jeremy.kloth
set
messages: +
2012-06-24 11:45:40
skrah
set
messages: +
2012-06-24 11:36:25
skrah
set
messages: +
2012-06-24 11:32:00
loewis
set
messages: +
2012-06-24 11:23:17
loewis
set
messages: +
2012-06-24 11:22:57
skrah
set
messages: +
2012-06-24 11:06:07
skrah
set
messages: +
2012-06-24 11:04:54
python-dev
set
messages: +
2012-06-24 10:45:49
skrah
set
messages: +
2012-06-23 22:22:37
python-dev
set
messages: +
2012-06-23 20:33:05
pitrou
set
messages: +
2012-06-23 20:22:09
skrah
set
messages: +
2012-06-23 20:19:54
python-dev
set
messages: +
2012-06-23 19:37:21
pitrou
set
messages: +
2012-06-23 19🔞46
skrah
set
files: + external-amd64.diff
messages: +
2012-06-23 19:08:13
python-dev
set
messages: +
2012-06-23 16:59:18
python-dev
set
nosy: + python-dev
messages: +
2012-06-23 16:37:55
loewis
set
messages: +
2012-06-23 16:35:53
loewis
set
messages: +
title: Fix 64-bit building for buildbot scripts -> Fix 64-bit building for buildbot scripts (issue 15102)
2012-06-23 16:31:17
skrah
set
files: + useenv.diff
messages: +
2012-06-23 16:28:17
georg.brandl
set
messages: +
2012-06-23 16:15:48
jeremy.kloth
set
messages: +
2012-06-23 16:10:10
loewis
set
messages: +
2012-06-23 16:05:41
skrah
set
messages: +
2012-06-23 15:56:16
loewis
set
messages: +
2012-06-23 15:32:00
jeremy.kloth
set
messages: +
2012-06-23 15:27:57
jeremy.kloth
set
messages: +
2012-06-23 15:09:40
loewis
set
messages: +
2012-06-23 15:01:20
georg.brandl
set
messages: +
2012-06-23 12:32:20
pitrou
set
nosy: + georg.brandl
2012-06-22 21:25:43
jkloth
set
files: + 5924b376d15f.diff
2012-06-22 20:03:19
pitrou
set
messages: +
2012-06-20 14:58:38
skrah
set
nosy: + loewis, brian.curtin
2012-06-20 14:56:16
jkloth
set
messages: +
2012-06-19 17:01:20
zach.ware
set
nosy: + zach.ware
2012-06-18 22:26:11
jeremy.kloth
set
messages: +
2012-06-18 21:27:45
pitrou
set
nosy: + pitrou
messages: +
2012-06-18 21:20:00
jeremy.kloth
set
nosy: + jeremy.kloth
messages: +
2012-06-18 20:42:45
skrah
set
nosy: + skrah
messages: +
2012-06-18 19:45:53
pitrou
set
files: + 2a20cee18add.diff
keywords: + patch
2012-06-18 14:48:15
jkloth
create