[Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X)) (original) (raw)

David Bolen db3l.net at gmail.com
Thu Apr 15 09:20:20 CEST 2010

Ronald Oussoren <ronaldoussoren at mac.com> writes:

Speaking of which... I have a mac-mini that could be used for a buildbot. How much work is needed to kickstart a buildbot, taking into account that I'd prefer to have a buildbot with different configure-flags that the default unix build (that is, I want to test a framework build that is a universal binary).

Sort of picking this message to enter the thread ...

Interestingly enough, I had made an offer to Martin to host an OSX build slave last week (before this thread), since I had a Mini whose disk crashed and would be reasonably free after I rebuilt it, and I had noticed there were no longer any OSX build slaves around.

It's online now, though it certainly need not be the only one. There's not much to setting up a build slave - just easy_install buildbot on top of a basic Python install (you'll need Twisted), use the "build-slave" command to create a tree, edit the files in "info", and "buildbot start " to get it started. Of course, there's some overhead to monitoring things over time.

My new slave is a Tiger box, since that's what I still need for my own application builds. So it won't help with building/testing x64 builds.

Per the "startup on reboot" part of the thread, I typically use the LaunchAgents setup on my Macs for that purpose, although to be honest, not many of my build slaves start automatically anyway, as the boxes don't tend to get restarted other than manually.

In the first builds, I have noticed that the build master seems to execute the builds as a Unix system, so isn't building with the universalsdk option or as a framework, though the latter would probably need a bit of glue to make the framework available just locally to the buildbot process.

On Windows, the buildbot tasks use scripts that keep the third party stuff in the buildbot branch tree itself. I wonder if perhaps with a little TLC, we could come up with some Mac-specific buildbot scripts to better have an OSX build slave mimic the final build process. It might be as simple as using build-installer with a specified directory within the build tree, though utilitizing the built framework for the tests probably needs some support. The build master could then be set to execute those scripts (much as it does on Windows) rather than the stock Unix approach.

I anticipate some tuning to do with respect to external libraries. Rather than use system libraries or my own (or finks or MacPorts) version of libraries, I suspect it would be better to download the same releases of the third party stuff that the installer script downloads and install them locally for normal buildbot builds, to best match what would be packaged up with a final binary. Of course, if so, then there's the question of keeping the buildbot environment up to date with the installer script.

As I believe Martin mentioned elsewhere in the thread, he used to have my Windows slave generate nightly MSI builds and upload them, but the process became fragile over time, and didn't seem to be missed when we discontinued them. But something similar could probably be worked out for building nightly DMGs for OSX if it were deemed useful.

Creating the Mac installer is easy: just run Mac/BuildScript/build-installer.py on an OSX 10.5 system where a local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The system should also not have fink or darwinports and a clean /usr/local tree to avoid contaminating the build.

Yes, I've successfully used the script to create a DMG of the latest 2.7 out of trunk, and at least the basic sniff test (install it locally, run a few things) seems fine. Did run into one quirk where if you have already built Python within the tree you run the script from, it doesn't rebuild everything beneath /tmp/_py/_bld - such as the pgen stuff. So you want a clean source tree too.

To the extent that the output (DMG) of the build-installer script is what is currently needed for OSX, I can generate one if it is still needed, though I can't promise much beyond just executing the script. The new build slave can also be made available for RMs (or whomever) to generate the DMG when needed if that might be helpful. Though there should probably be some basic installation test on other systems prior to publishing any such generated files.

-- David

More information about the Python-Dev mailing list