[Python-Dev] Packaging and binary distributions for Python 3.3 (original) (raw)

Paul Moore p.f.moore at gmail.com
Fri Oct 14 17:26:34 CEST 2011

On 14 October 2011 15:13, "Martin v. Löwis" <martin at v.loewis.de> wrote:

Thanks for the clarification. I can see why this would be important. But maintaining 3 different interfaces to do essentially the same thing (collect some data from the user, then based on that data put the same set of files in the same places) seems a waste of effort, and a recipe for discrepancies in capabilities.

Maybe the wininst and MSI installers should ultimately become simple UIs around a zipfile and an invocation of the packaging APIs? Not that I'm offering to do that work, I'm afraid... I think you are mixing issues: even if they were simple wrappers, they would still be 3 different interfaces (presented to the user). I.e. the user doesn't really care whether there is a zip file inside (as in bdistwininst) or a cab file (as in bdistmsi), and whether or not the packaging APIs are invoked during installation. So if you want to get rid of interfaces, you really have to drop one of the formats. Making maintenance of the interfaces simpler and more homogenous by having them call into installed Python code is certainly worthwhile. The challenge here is that the installers also work on older Python versions (unless there are extension modules in there), so they could only use the packaging API when installing into 3.3 or newer.

You are right that I'm mixing issues somewhat. I think the two issues are multiple interfaces and multiple formats.

The problem is that the packager has two roles:

  1. He does the build, which is an essential service as end users don't necessarily have the means to compile.
  2. He chooses the distribution format (and hence the user experience), which is not essential per se.

The second role is the problem, as the packager's preferences may not match those of the end user.

I am proposing decoupling those 2 roles. In the purest sense, I'd like to see a distribution format that contains purely the binaries, with no UI intelligence. That's what the proposed bdist_simple format is about. This format can be consumed by external tools that just read it as data (that's what pysetup, and packaging, does).

For a standard user experience on Windows, at least, there needs to be a means of wrapping that data into an installer. MSI is the obvious choice, because it's the MS standard, but there's a problem here in that there's no obvious way to retrieve the raw bdist_simple file from the MSI once you've bundled it. The wininst format is better here, as you can retrieve the original format (just by removing, or ignoring, the EXE header).

I have no opinions on how the MSI/wininst installers should present options to the user, nor do I have any strong views on how they should do the installation (my instinct says that they should reuse the packaging core code, but as you say that causes compatibility problems, so may not be feasible).

My only strong views are:

The second point is because pysetup offers capabilities that MSI/wininst do not - at least right now, and probably always, as I expect the core packaging code to support new features like venvs sooner than Windows-specfic formats. If only because Unix developers can hack on packaging.

Conversion between bdist_wininst and the proposed bdist_simple format is easy as they are both fundamentally zipfiles. The MSI format is the odd one out here, as I don't know how to build an MSI from binary data (without using bdist_msi - or to put it another way, the layout of the files that bdist_msi expects to build the installer from isn't documented) and I don't know how to get binaries out of an MSI (without installing). If these two things were possible, we'd have the means to convert between formats at will (and writing a tool that non-packagers could use to convert formats would be possible).

I'm focusing on writing code to allow packaging to install from a binary zipfile for a very simple reason - it's easy! The code is in Python, most of the support routines are there, and it's just a matter of putting the bits together. I had a mostly-working prototype done in a couple of hours last night. In contrast, I wouldn't even know where to start to modify bdist_wininst or bdist_msi to conform to the new package metadata standards, let alone to cope with non-system Pythons (which pysetup does out of the box). So by developing a new format, I can at least offer code to help, rather than just flooding the mailing lists with suggestions (although I seem to be doing that as well :-))

Paul.

[1] The one sticking point is that users generally want all the features that exist, so if bdist_msi doesn't support installing into "non-system" Python builds, and pysetup does, that will drive people to either switch, or more likely complain about the "limitations" of MSI :-)

More information about the Python-Dev mailing list