Issue 16895: Batch file to mimic 'make' on Windows (original) (raw)
Created on 2013-01-08 20:43 by zach.ware, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (21)
Author: Zachary Ware (zach.ware) * 
Date: 2013-01-08 20:43
I find myself frustrated with some of the hoops I have to jump through and extra things I have to remember when I want to play around with Python on Windows. To make things a little easier, I've created a 'make.bat' file that can sit in the root of the repository and do a few of the more common tasks:
make [build] [-64] -- Runs Tools\buildbot\build[-amd64].bat make clean [-64] -- Runs Tools\buildbot\clean[-amd64].bat make doc [args] -- Runs Doc\make.bat [args] with %PYTHON% set to 'py.exe -2' make patchcheck -- Runs Tools\scripts\patchcheck.py make test [test args] -- Runs the test suite with passed arguments
It seems to work pretty well for me and I think it could be very useful to others, particularly new contributors. I'm sure there's plenty of room for improvement, though :)
I don't know if this script might interfere or conflict with Cygwin or other tools; if so and it were decided to include this file, it could easily be renamed to 'wmake.bat' or some such.
Hope this is useful enough to include, or at least useful to someone else :)
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-03-09 20:08
+1
Add make external[-64] to runs Tools/buildbot/external[-amd64].bat and explain this in the devguide (if not already) rather than buried in the opaque PCBuild/readme.
Even after reading build.bat, I do not understand what is the different net effect of that versus just double clicking pcbuild.sln and building. So some explanation in the devguide would be needed.
Author: Zachary Ware (zach.ware) *
Date: 2013-03-10 06:31
Here's an updated version in the form of a patch.
Features include:
Ensure things are done in the right directory with pushd/popd
Allow setting environment variables (for the life of the script) with: make test "PYTHON=..\PCbuild\python.exe"
Emulate UNIX's
make -C <dir>(cd dir; call make.bat %*, essentially)Choose x86 or x64 with "-64" command line switch
Intelligently choose an interpreter for
make patchcheckandmake testAvailable targets are: build, clean, patchcheck, ready, test. "ready" is implemented by Tools/buildbot/external[-amd64].bat; "make ready" sounds better and possibly more accurate than "make external" to me.
Create a convenience "python.bat" script, which calls the newly built interpreter with all supplied arguments. Created by :build, deleted by :clean (and added to .hgignore)
Everything seems to work ok for me, but my testing platforms are limited to a single Win7 laptop, currently. This issue is also somewhat constrained by issue 17202; it is supposed to work without any change to .hgeol, but that can't be guaranteed.
Author: Richard Oudkerk (sbt) *
Date: 2013-03-10 14:59
+1
To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the "-C" argument: just set the correct directory automatically.
I think make.bat should also support creation of non-debug builds. (Maybe have targets "release" and "debug"?)
Tools/buildbot/build.bat already calls external.bat and clean.bat. This currently makes the "ready" target unnecessary. However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build.bat.
Author: Zachary Ware (zach.ware) *
Date: 2013-03-10 17:29
Richard Oudkerk added the comment:
+1 Thank you :)
To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the "-C" argument: just set the correct directory automatically.
That is true, and it does already automatically set the correct directory. The point of the -C option is for Doc/make.bat--from the root dir, call "make -C Doc html", and make cd's into Doc, and there calls "make.bat html", and the html docs are made. I didn't just hardwire the option to "-C Doc" even though that's the only place that can use -C, just in case there is someday another make.bat floating around the repository. The original version I posted just had a "doc" target that called Doc/make.bat with supplied arguments, but forcing the use of the UNIXy command makes documenting how to do things easier: just one command on all platforms.
I think make.bat should also support creation of non-debug builds. (Maybe have targets "release" and "debug"?)
I agree. I've had a thought for this; what about adding a "configure.bat" that takes a few of the applicable options from the UNIX "configure" script, and sets some environment variables (and/or writes out a "win-config.dat") for make.bat's use?
Tools/buildbot/build*.bat already calls external.bat and clean.bat. This currently makes the "ready" target unnecessary.
I thought the same initially, but I realized that it is useful for building using the VS GUI--clone the repo, call "make ready", then open pcbuild.sln in Visual Studio and you're ready to build everything there.
However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build*.bat.
I've thought about this too, and am thinking that it would make a certain amount of sense to inline the entirety of Tools/buildbot. Make everything possible from just make.bat, and eventually migrate the buildbots to use it, and then the Tools/buildbot directory can go away alltogether. This would also make doing release vs debug easier.
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-03-10 18:25
One must run external.bat if one is to subsequently build from the vs gui with most of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular.
Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date?
What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.
Author: Zachary Ware (zach.ware) *
Date: 2013-03-10 23:12
Terry J. Reedy added the comment:
One must run external.bat if one is to subsequently build from the vs gui with most of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular.
Fair point. Thinking about it again after some sleep, I agree. Next version will revert to 'external' rather than 'ready'.
Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date?
I think the idea is to make sure everything is fresh, and avoid phantom problems from things that shouldn't cause problems, but do.
What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.
I believe it's what actually performs the cleaning process. Tools/buildbot/clean.bat builds kill_python.exe and then runs it.
Author: Richard Oudkerk (sbt) *
Date: 2013-03-11 00:13
What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.
It kills any currently running python(_d).exe processes. This is because Windows does not allow program or library files to be removed or overwritten while they are being used, potentially causing compilation failures.
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-03-11 01:38
Thanks, clean and kill make sense for unattended build-bots. For interactive use, when 2 minutes rebuilding is a big deal, clean is a last resort. Let's just make sure we do not somehow suggest that it needs to be done routinely. Or to put it another way, do document that 'build' is intended for fail-safer unattended buildbots and not necessarily for interactive human use. And also that kill does what a human would normally do by closing windows on the task bar, or, as a last resort, with task manager.
Author: Zachary Ware (zach.ware) *
Date: 2013-04-03 16:42
I was rather off about kill_python, wasn't I?
Anyway, I've finally gotten another version put together that I'd like to get comments on. Major differences between this patch and the previous one, in no particular order:
Add a 'configure.bat' script to increase the parallel with Unix building. Currently only has 2 options, --with-pydebug and --with(out)-64-bit, which affect how PY_PLATFORM and PY_CONFIGURATION are set in make.bat, which allows for building any combination of Debug, Release, 32 bit, and 64 bit with make.bat.
Add 'win-config.dat' to .hgignore; this file stores configure.bat's options for make.bat's use.
Update documentation: PCbuild/readme.txt gets significant updates, including some that apply without make.bat and which I can split out into a separate patch if someone wants to commit those separately. Doc/using/windows.rst also gets a small update which actually just goes with one of the PCbuild/readme.txt updates and isn't affected by make.bat.
make.bat is now stand-alone; it does not rely on any other batch scripts except configure.bat (namely Tools/buildbot/*).
Renamed "ready" target to "external", as promised.
Reimplemented "test" target to match the Unix implementation. Also added "buildbottest" target, which matches Unix implementation.
Added separate "tcltk" target to build tcltk and copy the built .dlls to the appropriate place for the built Python to find them.
Added "all" alias to "build" target (to match Unix). "build" calls :external on its own, and calls :tcltk if it can't find the appropriate tcl/tk .dlls.
Added "clobber" target, which does its best to obliterate everything make.bat can create. The only way I've thought of to be more thorough is to actually delete the entire contents of the PCbuild directory and do an
hg revert -C PCbuildto get it back, but that does have the possibility to be more destructive than desired.Added variables to the top of the file to set the versions of the external projects. This means there's only one place to update when an external is updated, and for easy testing one could do something akin to
make "OPENSSLVERSION=openssl-1.0.1e"Added "msi" target to perform the task of Tools/buildbot/buildmsi.bat. Note that this one is hard for me to test, as buildmsi.bat does not work properly for me. However, the msi target works as well or better than buildmsi.bat for me, and should work assuming msi.py can do what it is meant to. I wonder, though; does anybody use buildmsi.bat? I noticed that it has information in it hard-coded for Python 2.6a3...
Several things have gone through an iteration or two since the last patch; I'll try to go through and add some comments on Rietveld myself.
Since make.bat is now stand-alone, if it is accepted (and it is deemed desirable to do so), the Windows buildbots could be converted to use make.bat instead of Tools/buildbot/*, and that directory could actually be removed. This would also make it easy to set up a buildbot that tests a Release configuration of Python.
All comments welcome :)
Author: Richard Oudkerk (sbt) *
Date: 2013-04-07 12:45
You seem to end your subroutines (or whatever they are called) using "goto end" rather than "exit /b". Since popd follows the "end" label, does this mean that you get a popd after calling each subroutine? Is this intended and can it cause unmatched pushd/popd-s?
(I am not familiar with writing batch files.)
Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support.
Author: Zachary Ware (zach.ware) *
Date: 2013-04-08 17:08
You seem to end your subroutines (or whatever they are called) using "goto end" rather than "exit /b". Since popd follows the "end" label, does this mean that you get a popd after calling each subroutine?
Yes.
Is this intended and can it cause unmatched pushd/popd-s?
It is intended and it can, but so far I haven't run into any trouble with it. However, I have realized that I haven't done any testing with some pushd's already on the stack... I'll do some more looking into that, and may have to implement either some form of callback labels or counting pushd's to properly popd.
(I am not familiar with writing batch files.)
Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support.
Fair point. I suppose I had been assuming that the error message from not finding the proper vcvars*.bat file would be an indication to pass '--without-64-bit' to configure.bat, but it is much nicer to just fall back to 32 bit. Also, it would be nice to just implode from the start if there is no compiler available. Next version of the patch will do both of these things in configure.bat; quit with an error message if %VS100COMNTOOLS% is not defined, and check for the existence of %VS100COMNTOOLS%....\VC\bin\x86_amd64\vcvarsx86_amd64.bat before setting %PY_PLATFORM% to x64.
Thanks for the comments.
Author: Zachary Ware (zach.ware) *
Date: 2013-05-23 19:58
Here's a new version of the patch, which is a major rewrite. Among the changes:
Switch from using 'goto' to execute the right subroutine to 'call' and end every subroutine with 'exit /B' (thank you Richard for making me aware of that possibility). This makes the :end label useless, so it's removed. Every subroutine now expects to start in the root of the source tree, and is expected to return there before exiting.
Changed the argument parsing magic to a different flavor of magic. Setting variables on the command line is now much more like it is using Unix make, no more required double quotes around the whole assignment, and you can now specify multiple targets in a single command. This method can be quite a bit slower if you give a huge command, as it steps through the command line a character at a time (on the order of several seconds if you're appending something to PATH, for instance. The delay is not noticeable if you're just giving targets), but at least all the magic is self-contained.
Add '--no-externals' and '--interactive' options to configure.bat to ignore the presence or absence of the external libs and to allow questions to be asked of the user, respectively. I've waffled back and forth on whether it is better for the '--interactive' option to be on or off by default, currently it is off.
On 64 bit machines, default to Win32 platform if vcvarsx86_amd64.bat can't be found.
Give some nicer output just about everywhere. This includes making configure.bat die early in case %VS100COMNTOOLS% is not defined with a message to install MSVC++ 2010 Express, a warning if NASM is not available, and tagging all output with the name of the script. Also, add a message at the end of building Python giving the best estimate of the outcome.
Another part of giving a nice message at the end of building Python: if the expected executable file exists at the end, exit code 0 is returned. This is for the buildbots, so that even if the build had errors, the tests are still run so we can see what is actually broken. This ties in with the new '--no-externals' option on configure.bat, which would allow a buildbot to build with '--no-externals' and still be able to test everything else.
List and use specific exit codes. If anyone has advice on better numbers to use (more standard numbers, perhaps), please let me know :).
Use "absolute relative" paths (paths that always start with %~dp0, but may include '..') to avoid some directory changing magic just to make a relative path point to the right place.
'clobber' target builds the clean target of Win32 configuration, then x64 config if the environment can be set up to do so.
'importlib' will clean up after itself if there wasn't a build already done.
Some refactoring and cleanup to put things in places that make more sense and look better. This includes major changes to :external and :tcltk to use a couple of loops to be more DRY.
Change Tools/scripts/run_tests.py (used by the test targets in make.bat and Makefile) to use subprocess.call instead of os.execv on win32. Since execv creates a new process on Windows, control was handed back to the console as soon as the execv call was made, but the test output still came to the terminal so the prompt was lost in the test output and did not automatically come up at the end of the test. Using subprocess.call keeps control in the parent python(_d) process and doesn't release back to the console until the test is done.
Added better documentation of make.bat usage to PCbuild\readme.txt, and took out unrelated changes to that file (which will be posted to another issue).
Everything works as expected for me, but I do only have a 32bit machine available to me currently. I hope to be able to test on a 64bit machine soon, but have no guarantees. For anyone testing this, I would suggest to run the command "prompt $+%PROMPT%" before testing; this will add a "+" to the beginning of your prompt for every dir on the pushd/popd stack. Neither configure.bat nor make.bat should ever remove dirs from that stack, and should not leave any extras when they're done (unless forcibly killed in the middle of running).
Author: Richard Oudkerk (sbt) *
Date: 2013-05-28 20:06
I can't say I know enough about batch files to understand much of the code, but a few notes:
Windows XP does not have the command "where" which you use -- Python 3.4 will still support XP.
Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in "exit /b ..." make it very confusing as to where "exit /b" will return control.
The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like
...
pushd "%~dp0"
call :main ...
popd
exit /b
:main
...
exit /bIt would also be helpful if the end of the subroutines were marked with a comment like
rem end :fooAuthor: Brian Curtin (brian.curtin) *
Date: 2013-05-28 20:09
Can't this just be a Python script?
Author: Richard Oudkerk (sbt) *
Date: 2013-05-28 20:20
Can't this just be a Python script?
That would cause bootstrap issues for people who do not already have python installed.
Author: Zachary Ware (zach.ware) *
Date: 2013-05-28 20:26
Richard Oudkerk added the comment:
I can't say I know enough about batch files to understand much of the code, but a few notes:
Windows XP does not have the command "where" which you use -- Python 3.4 will still support XP.
Oh, that is an issue. I don't have an XP machine to test on anymore, thank you for that catch. I found a workaround on StackOverflow that looks short enough to be usable instead.
Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in "exit /b ..." make it very confusing as to where "exit /b" will return control.
The only goto's that are not part of loops are now one near the beginning for the -C option, in the target validation routine to show the usage message and die, and in a couple of routines which use "goto no-configure" to show a common message and die. I'd rather not have to copy that message every place it is used, but that is an option. Would just adding comments explaining where execution is going and whether it is coming back be sufficient?
The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like
... pushd "%~dp0" call :main ... popd exit /b :main ... exit /b
Fair enough, I can change that. I tried to keep the matches to the initial pushd to a minimum, but perhaps there are a couple more I can eliminate.
It would also be helpful if the end of the subroutines were marked with a comment like
rem end :foo
Easy enough, consider it done :)
Author: Brian Curtin (brian.curtin) *
Date: 2013-05-28 20:27
Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL?
I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python.
Author: Zachary Ware (zach.ware) *
Date: 2013-05-28 20:35
Brian Curtin added the comment:
Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL?
Only for building a 64-bit Python on 32-bit Windows. Otherwise, OpenSSL is built using the just-built interpreter.
I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python.
Fair enough, but even when Python is installed, there's still the issue of whether it will be findable on PATH, whether .py is in PATHEXT, what version of Python is installed (and which one is on PATH), etc. etc...
However, you've made me think; perhaps configure.bat could build a minimal Python that can then be used for a make.py. I'll do some experimenting and see what I can come up with.
Author: Mark Lawrence (BreamoreBoy) *
Date: 2014-07-12 22:12
Is this still relevant or has it been overtaken by other work from Zach or Steve?
Author: Zachary Ware (zach.ware) *
Date: 2014-07-14 21:08
Still relevant, but the current patch is overkill (Brian was right, it's just way more batch than is healthy to check in). Things are changing rapidly, though; I'll close it as "postponed" for now, with the expectation of reopening someday.
History
Date
User
Action
Args
2022-04-11 14:57:40
admin
set
github: 61099
2014-07-14 21:08:47
zach.ware
set
status: open -> closed
messages: +
assignee: zach.ware
components: + Build
resolution: postponed
2014-07-12 22:13:30
brian.curtin
set
nosy: - brian.curtin
2014-07-12 22:12:10
BreamoreBoy
set
nosy: + BreamoreBoy, steve.dower
messages: +
versions: + Python 3.5, - Python 3.4
2014-02-03 17:42:03
jkloth
set
nosy: + jkloth
2014-02-03 17:09:19
BreamoreBoy
set
nosy: - BreamoreBoy
2013-05-28 20:35:18
zach.ware
set
messages: +
2013-05-28 20:27:27
brian.curtin
set
messages: +
2013-05-28 20:26:42
zach.ware
set
messages: +
2013-05-28 20:20:57
sbt
set
messages: +
2013-05-28 20:09:45
brian.curtin
set
messages: +
2013-05-28 20:06:19
sbt
set
messages: +
2013-05-28 14:44:45
BreamoreBoy
set
nosy: + BreamoreBoy
2013-05-23 19:58:30
zach.ware
set
files: + win_configure-make.v2.diff
messages: +
2013-04-08 17:08:53
zach.ware
set
messages: +
2013-04-07 12:45:52
sbt
set
messages: +
2013-04-03 16:42:36
zach.ware
set
files: + win_configure-make.diff
messages: +
2013-03-11 01:38:19
terry.reedy
set
messages: +
2013-03-11 00:13:57
sbt
set
messages: +
2013-03-10 23:12:33
zach.ware
set
messages: +
2013-03-10 18:25:03
terry.reedy
set
messages: +
2013-03-10 17:29:47
zach.ware
set
messages: +
2013-03-10 14:59:10
sbt
set
nosy: + sbt
messages: +
2013-03-10 06:31:38
zach.ware
set
files: + win_make.diff
keywords: + patch
messages: +
2013-03-09 20:08:49
terry.reedy
set
nosy: + terry.reedy, tim.golden, brian.curtin
messages: +
2013-01-08 20:43:41
zach.ware
create