[Python-Dev] Packaging and binary distributions for Python 3.3 (original) (raw)

Éric Araujo merwok at netwok.org
Thu Oct 13 18🔞56 CEST 2011

Le 09/10/2011 13:54, Paul Moore a écrit :

On 9 October 2011 08:15, Éric Araujo <merwok at netwok.org> wrote:

Are there that many distributions with extension modules? sdists should work well even on Windows for pure Python projects. [...] Looking at my installations, I see: - database drivers (cxOracle, in my case) - lxml - pywin32 - pyQT - pyzmq (that's just for playing a bit with IPython, so doesn't really count...) - I've also used in the past PIL modpython (modwsgi more recently) and wxPython,

These are good examples. Even if the number is not high, they are widely used, so support for binary distributions really seems needed.

(When pip switches to distutils2 as underlying lib, they’ll get bdist support for free! IOW, proper bdist support would be another argument to make the world switch to the new standards.)

The pysetup features for uninstalling packages aren't going to work with bdistwininst/bdistmsi (that's an assumption, I haven't tried them but I can't see how they would, and it'd certainly be a lot of marginally-useful effort to do even if it were possible).

The virtual environment stuff also wouldn't work that well with the installers, because they wouldn't have any way of finding which environments existed to ask where to install to. The same problem exists with virtualenv. (Again this is speculation backed by a small amount of playing with virtualenv, so I may be wrong here).

wininst and msi bdists can continue to be used as previously, for people who want clicky installation to their system Python. With built-in package management and virtual environments in 3.3+, we can just recommend that people publish bdist_simple instead of wininst or eggs.

It may be that the bdistdumb format would be OK. I haven't checked it out (to be honest, I don't think it's ever been used much). It may or may not be. bdist_dumb just makes a tarball or zipfile of Python modules and built extension modules and is supposed to be unpacked under sys.prefix or sys.exec_prefix. However, that won’t play nice with install options (--home, --user, --install-lib or redefined --prefix) or sysconfig categories (i.e. I may want config files under /usr/local/etc, scripts in /usr/local/bin, etc.). I think we could revamp bdist_dumb so that it’s really “sdist with compiled files” and then we let pysetup install things to the right places.

Yes! We need feedback to provide a much better tool than distutils, before the API is locked by backward compatibility rules. Always the chicken and egg problem :-) I’d rather say it’s “code in the stdlib has one foot in the grave” + “stdlib code without active maintainer is effectively frozen” (hi asyncore changes in 2.6!).

I actually wanted to talk about that, so let me take the opportunity. What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI) as a not-quite-final release? [...] My immediate thought is that it would actually put people off using packaging for 3.3, they'd wait until "it is stable". OK. Too bad.

I’ll probably post that question again in its own message to get more feedback.

What is the status of distutils2? Is that (still?) intended to be effectively a backport of packaging to earlier Python versions? Yes. It works with 2.4-3.3. I maintain it synchronized with packaging in 3.3. There are a small number of test failures which needs fixing before I release distutils2 1.0a4 on PyPI.

I'd suggest getting a distutils2 release available, and promoted, as the "early adopter" version of packaging. We may do that, but I fear we’re going to lack time for that. As part of the stdlib, the packaging module API will be frozen in June, for the first 3.3 beta. We still have a lot to do: defining all in all public modules, changing some important internals (Tarek wants to kill the subcommands system for example), fixing a number of bugs which may imply incompatible API changes, etc. The pace of development has slowed much these last months, so I’m not sure we’ll reach 1.0 status months before June.

Maybe even with an option to install it as "packaging" so that people can use it in 3.2 and earlier and expect to need no changes when 3.3 is released. Not gonna happen! d2 changed name on purpose when entering the stdlib, so that in the future code can choose to use packaging (in the stdlib, for example 1.0) or distutils2 (external backport, possibly of 2.0). Code wanting to use “packaging if available otherwise distutils2” will use the same import try/except as what’s done with unittest/unittest2, json/simplejson and similar.

A python-announce article "Python 3.3 new packaging features - early adopter release" publicising it, would be what I'm thinking of... Here’s a plan:

  1. make the docs usable (I’m on it)
  2. fix the three test failures we currently have in d2
  3. test on Windows and Mac
  4. release 1.0a4 (I’ll do it)
  5. announce and request feedback
  6. work frantically before Python 3.3b1 to improve stuff and limit the public API so as not to lock ourselves

Regards

More information about the Python-Dev mailing list