[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) (original) (raw)

Paul Moore p.f.moore at gmail.com
Thu Oct 8 14:56:05 CEST 2009

2009/10/8 Tarek Ziadé <ziade.tarek at gmail.com>:

The egg format

One thing missing from your roadmap (unless I missed it) is the fate of the egg (zipfile) format. If it's to remain a valid option (bdistegg and the like) then I'd assume that Distribute needs to be the place to develop it. One thing it will need is an equivalent to PEP 376 introspection and uninstall features. And that is where I believe a standard protocol belongs - developed based on the real use requirements for egg files, with a suitable file-based layer which can be added into distutils (somewhat like importlib does for PEP 302) once the protocol is agreed and standardised. Distribute is planning to continue supporting the egg format, but will probably drop the zipped version. This is still under discussions though so don't take this as the final word. In any case, distribute.resources will probably add the missing bits on the top of a PEP 376 -compliant Distutils to provide support for any file querying Distutils will not provide (for installation and uninstallation)

[Note: For the purposes of below I use "egg format" to only refer to zipped eggs. I see no value AT ALL in having an unzipped "egg format" which is almost the same as, but subtly different from, the distutils-supported standard filesystem format. If the distutils standard doesn't provide for all the requirements, fix that, don't add another variation!]

I'm not sure I follow this. I think it's important to provide for non-filesystem based packages (that's the whole point of things like PEP 302 and importlib!). The fact that core distutils doesn't support installing, querying or removing such packages is fine. They can be supplied by 3rd party systems like Distribute.

However, it is not fine (in my view) for each independent format to have to implement its own set of interfaces. PEP 302 is precisely the solution for this in terms of import - every importer has a standardised set of protocols to follow, and if it does so, then it will (should :-)) work seamlessly with any Python code. PEP 302 is not perfect, in part because it aims to be minimal rather than comprehensive, hence the fact that not all packages are "zip safe". But that's a failing in PEP 302 rather than in the concept.

Multiple package formats should be supported in the same way - with a set of well-defined (via a PEP) interfaces that all package formats must provide, which enable core and user code to be written in a format-independent way. Once such a PEP is written, distutils' PEP 376 support should be rewritten to use those interfaces, and to implement the necessary filesystem-format support layer. (In other words, whereas I support reducing PEP 376's scope to filesystem only for now, I still strongly advocate that the longer-term solution should be based on a user extensible format-independent approach).

In this context, eggs are "merely" the first (and most important) example of a format extension, and so should drive the development of a standard.

To summarise:

I believe that we need a statement of direction on the (zipped) egg format. I see a number of options:

  1. Retain it as-is in Distribute, but deprecated and no further development.
  2. Deprecate egg support in Distribute, and ultimately drop it.
  3. Develop into a reference example of an extensible architecture for adding to PEP 376 support.
  4. Extend PEP 376 support to eggs by means of egg-specific approaches hooking into distutils.

For the record, I dislike (1), I see (2) as a lost opportunity, I see (3) as the right way to go, but appreciate that it's the most work, and I strongly oppose (4) (not least because it effectively makes a latrge part of PEP 302 and importlib wasted effort).

It looks like you're proposing (2), but aren't entirely clear because you're still allowing for non-zipped "egg" formats (which I see no value in, as I noted above).

I'm willing to help work on option (3). Once it's decoupled from questions around OS package managers (which are more the province of PEP 376) then I'd hope that my lack of knowledge about package managers is less of a problem.

Paul.

More information about the Python-Dev mailing list