[Python-Dev] Packaging and binary distributions for Python 3.3 (original) (raw)

Éric Araujo merwok at netwok.org
Sun Oct 9 09:15:21 CEST 2011

Hi Paul,

Thanks for raising this during the development phase.

I see that the Packaging documentation is now more complete (at least at docs.python.org) - I don't know if it's deemed fully complete yet, but I scanned the documentation and "Installing Python Projects" looks pretty much converted (and very good!!), but "Distributing Python Projects" still has quite a lot of distutils-related text in, and I need to read more deeply to understand if that's because it remains unchanged, or if it is still to be updated. The basic structure is in place (distributing/installing/library reference), but the docs are far from up-to-date. I have nearly finished a first patch that fixes a ton of markup issues and updates distutils idioms (setup.py snippets e.g.) to packaging ones (setup.cfg snippets, using pysetup, etc.), it’s already a thousand changed lines. Then I will work on another patch to move things around, consolidate, expand and rephrase. See http://bugs.python.org/issue12779 if you want to follow along and review patches.

But one thing struck me - the "Installing Python Projects" document talks about source distributions, but not much about binary distributions. This is inherited from distutils docs, not a deliberate choice. We just haven’t thought much, if at all, about binary distributions.

On Windows, binary distributions are significantly more important than on Unix, because not all users have easy access to a compiler, and more importantly, C library dependencies can be difficult to build, hard to set up, and generally a pain to deal with. Are there that many distributions with extension modules? sdists should work well even on Windows for pure Python projects.

I don't know what format packaging favours. As a direct distutils descendant, can create bdist_wininst and bdist_msi. For installing, I was not aware of the problem you reported (“does not interact well with pysetup”); can you give more info? I’m guessing it boils down to the fact that Windows binary installers are meant to be clicked by users, not managed with command-line tools.

IIRC the current behavior in pysetup is to favor source distributions, but bdists should probably be favored for non-pure distributions on Windows.

So there will be a need for a pysetup-friendly binary format. I assume that the egg format will fill this role - or is that not the case? What is the current thinking on binary distribution formats for Python 3.3? First, we don’t want to include wholesale support for setuptools eggs in packaging. We have options to support egg/egg-info metadata in the PEP 376 implementation (packaging.database), because we need that to provide a useful tool for users and help them switch, but eggs are another matter. After all, if setuptools and then pkg_resources were turned down for inclusion in Python 2.5, it’s not now that we have packaging that we’ll change our mind and just bless eggs. What we can do however is to see what bdist_egg does and define a new bdist command inspired by it, but without zipping, pkg_resource calls, etc.

The main reason I am asking is that I would like to write an article (or maybe a series of articles) for Python Insider, introducing the new packaging facilities from the point of view of an end user with straightforward needs (whether a package user just looking to manage a set of installed packages, or a module author who just wants to publish his code in a form that satisfies as many people as possible). That’s excellent. I too thought about writing something about packaging for that blog, but an outside end-user viewpoint like yours would best match the readership. I can write a shorter piece just for packaging tool developers (i.e. how to use packaging as a library), or you can write that one too and act as a tester for our doc and API.

What I'd hope to do is, as well as showing people all the nice things they can expect to see in Python 3.3, to also start package authors thinking about what they need to do to support their users under the new system. Yes! We need feedback to provide a much better tool than distutils, before the API is locked by backward compatibility rules.

I actually wanted to talk about that, so let me take the opportunity. What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI) as a not-quite-final release? (Something like Python 3.0, which was not considered a real version and not supported as much as the other ones.) The goal would be to expose it to a large range of users to get bug reports and feature requests, but without locking us forever into one API or design, which was the death of distutils a year ago. The idea is not to scare people with warnings that we’ll break APIs on a whim, but that we keep the option to change parts of packaging and release a 2.0 with Python 3.4, with documented changes from 3.3. Opinions?

Regards

More information about the Python-Dev mailing list