[Python-Dev] Packaging and binary distributions (original) (raw)

Paul Moore p.f.moore at gmail.com
Sun Oct 30 15:09:58 CET 2011

I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions.

First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here.

Second question - is there a problem at all? For the majority of Windows users, I suspect not. The existing bdist_wininst and bdist_msi formats have worked fine for a long time, offer Windows integration and a GUI installer, and in the case of MSI offer options for integrating with corporate distribution policies that some users consider significant, if not essential. (Binary eggs are a third, and somewhat odd, case - a number of projects have started distributing binary eggs, but I don't know what benefits they have over bdist_wininst in particular, as easy_install will read bdist_wininst installers. Perhaps a setuptools/distribute user could comment. For now I'll assume that binary eggs will slowly go away as packaging gets more widely adopted).

So that leaves a minority who (1) prefer integration with packaging, (2) need to work with virtual environments or custom local builds, (3) need binary extensions in some or all of their environments and (4) don't want to have to build all the binaries they need from scratch.

Given the scale of the issue, it seems likely that putting significant effort into addressing it is unwise. In particular, it seems unlikely that developers are going to move en masse to a new distribution format just to cater for this minority. On the other hand, for people who care, the fact that packaging (currently) offers no direct support for consuming binary distributions is a fairly obvious hole. And having to build from source just to install into a virtual environment could be a showstopper.

The bdist_wininst format is relatively amenable to manipulation - it's little more than a zip file, after all. So writing 3rd party code to install the contents via packaging shouldn't be hard (I've done some proof of concept work, and it isn't :-)) Vinay's proposal to use the resource mechanism and some custom hooks would work, but I'd like to see a small amount of extra direct support added to packaging to make things cleaner. Also, if packaging supported plugins to recognise new distribution formats, this would make it possible to integrate the extra code seamlessly.

The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable.

So, my proposal is as follows:

  1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging
  2. Where packaging changes are useful to make installing binaries easier, I'll request them (by supplying patches)
  3. I'll look at creating a format-handling plugin mechanism for packaging. If it's viable, I'll post patches
  4. If it seems useful, my module could be integrated into the core packaging module

I don't intend to do anything about a GUI, or modify the existing formats at all. These don't interest me, particularly, so I'll leave them to someone who has a clear picture of what they want in those areas, and the time to develop it.

For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs.

Unix/MacOS users who care will need to propose something themselves.

Does anyone have any comments?

Paul.

More information about the Python-Dev mailing list