[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) (original) (raw)

Ned Deily nad at acm.org
Thu Oct 8 23:28:47 CEST 2009

In article <94bdd2610910080131j323b98d9i871bce43465f237a at mail.gmail.com>, Tarek Ziadé <ziade.tarek at gmail.com> wrote:

Here's a quick summary of the main things that are going to happen in Distutils, and Distribute, and a few words on virtualenv and pip. (there is much much more work going on, but I don't want to drown people with details)

Thanks for the summary! Unfortunately, as the saying goes, the devil is in the details. Here are a few comments.

= Distribute =

I won't explain here again why we have forked, I think it's obvious to anyone here now. I'll rather explain what we are planning in Distribute and how it will interact with Distutils. Distribute has two branches: - 0.6.x : provides a Setuptools-0.6c9 compatible version - 0.7.x : will provide a refactoring == 0.6.x == Not "much" is going to happen here, we want this branch to be helpful to the community today by addressing the 40-or-so bugs that were found in Setuptools and never fixed. This is eventually happen soon because its development is fast : there are up to 5 commiters that are working on it very often (and the number grows weekly.) The biggest issue with this branch is that it is providing the same packages and modules setuptools does, and this requires some bootstrapping work where we make sure once Distribute is installed, all Distribution that requires Setuptools will continue to work. This is done by faking the metadata of Setuptools 0.6c9. That's the only way we found to do this. There's one major thing though: thanks to the work of Lennart, Alex, Martin, this branch supports Python 3, which is great to have to speed up Py3 adoption. The goal of the 0.6.x is to remove as much bugs as we can, and try if possible to remove the patches done on Distutils. We will support 0.6.x maintenance for years and we will promote its usage everywhere instead of Setuptools.

This is a key and, I think, difficult issue that's going to need a lot of thought and work to be successful. setuptools / easy_install filled several unmet needs of both package developers and end users and became a de-facto standard because there was essentially no competition. So, in a sense, there was never a migration to setuptools from something.
But now, especially given GvR's blessing of Distribute, there needs to be a migration from setuptools to Distribute, although neither are part of the standard library. There are no references to setuptools or easy_install in any of the standard Python documentation.

So how do end users know about easy_install? Their first encounter is/was probably through some recipe on a web site, most likely the installation instructions for some package they want to use. And I suspect pretty much all of those installation instructions direct the user to one of two places, either the PyPi page for setuptools or the peakcommunity wiki page:

http://pypi.python.org/pypi/setuptools http://peak.telecommunity.com/DevCenter/EasyInstall

Right now those are essentially the ultimate contact points for users.
A general Google seach for easy_install gets 357,000 hits; one restricted to pypi.python.org gets 2,500. I don't know how many of the latter are active packages on PyPi but certainly there must be hundreds that currently use setuptools and document the use of easy_install for their users. Those packages' documentation often directs users to download the ez_install script to bootstrap setuptools; I believe there are some packages that download it automatically, if necessary.

So, without some other intervention, users will continue to follow the instructions with those hundreds of packages (or on those thousands of websites) and manually or automatically try to install setuptools.
Which may or may not work - as witnessed with the release of 2.6.3.

Brett and others suggested including some sort of warnings in the python release documentation. Is there consensus that should be done?
Starting with 2.6.4? Seems like that would be up to Barry and Guido to pronounce.

Ultimately, to successfully migrate to Distribute is going to require a good understanding of the use cases for easy_install (primarily those of end users) and for the setuptools API (those of package developers).
Those use cases should be documented somehow, either in the roadmap or a reference elsewhere. This is especially true if your intentions are to deprecate the easy_install command in Distribute 0.7 and if APIs are going to change for package developers. As Antoine points out, pip is not equivalent to easy_install. BTW, keep in mind that today people also use easy_install to manage the installation of packages that do not use setuptools.

It will also be important to understand and address the major distribution channels for Python installations, many of which already include setuptools in their distributions. There are the multi-platform distributors, like python.org (which does not currently distribute setuptools), ActiveState, and Enthought. You've already mentioned some of the Unix-y O/S ones: Debian, Fedora, et al. On the OS X side, there's Apple and the major third-party providers of open-source packages, MacPorts and Fink - they all distribute setuptools. I don't have personal experience with the Windows world to know what all the channels are there. Are there other operating systems who package Python and possibly setuptools? They, one way or another, need to know and have some sort of plan going forward for their distributions regarding setuptools and Distribute.

And, of course, the stickiest issues in all this are not technical, but rather social. For most of this, the most important part will likely be giving a clear direction to all of the major groups involved: end users, package developers, distributors. How to decide on and then effectively communicate that direction is not at all trivial, I think.

Thanks again for your hard work on this, Tarek!

-- Ned Deily, nad at acm.org

More information about the Python-Dev mailing list