[Python-Dev] Status of packaging in 3.3 (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Wed Jun 20 14:53:07 CEST 2012

On Wed, Jun 20, 2012 at 9:31 PM, Tarek Ziadé <tarek at ziade.org> wrote:

Yeah maybe this subset could be left in 3.3

and we'd remove packaging-the-installer part (pysetup, commands, compilers) I think it's a good idea !

OK, to turn this into a concrete suggestion based on the packaging docs.

Declare stable, include in 3.3

packaging.version — Version number classes
packaging.metadata — Metadata handling
packaging.markers — Environment markers
packaging.database — Database of installed distributions

Maybe needed as dependencies for above?

packaging.errors — Packaging exceptions
packaging.util — Miscellaneous utility functions

It seems to me that stripping the library, docs and tests back to just these 4 modules and their dependencies shouldn't be much harder than stripping packaging in its entirety, but my question is what benefit would we gain from having these (and just these) in the 3.3 stdlib over having them available on PyPI in distutils2? Third party projects over the next couple of years are going to want to support more than just 3.3, so simply depending on distutils2 for this functionality seems like a far more sensible option.

OTOH, it does send a clear message that progress is being made, we just tried to take too big a jump from implementing these lower level standards up to "wholesale replacement of distutils" without first clearly documenting exactly what was wrong with the status quo and what we wanted to change about it as a sequence of PEPs.

I've broken up the rest of packaging's functionality below into a series of candidate PEPs that may be more manageable than a single monolothic "fix packaging" PEP. If we can get multiple interested parties focusing on different aspects, that may also help with reducing the burnout factor. Python's current packaging and distribution story is held together with duct tape and baling wire due to decisions that were made years ago - unwinding some of those decisions and replacing them with an alternative that is built on a solid architectural foundation backed by some level of community consensus is not an easy task, and not one that will be completed quickly (undue haste will fail the "some level of community consensus" part, thus missing much of the point of the exercise). That said, I don't think it's unsolvable either, and there's a reasonable chance to get something cleaner in place for 3.4.

3.4 PEP: Distutils replacement: Packaging, Distribution & Installation

# This is one of the big balls of mud w.r.t distutils where third

party projects dig deep into the implementation details because that is the only way to get things done # It may even be the case that this can be broken up even further packaging.install — Installation tools packaging.dist — The Distribution class packaging.manifest — The Manifest class packaging.command — Standard Packaging commands packaging.command.cmd — Abstract base class for Packaging commands

3.4 PEP: Distutils replacement: Compiling Extension Modules

# Another big ball of mud. It sounds like the Gentoo folks may

have some feedback in this space. packaging.compiler — Compiler classes packaging.compiler.ccompiler — CCompiler base class packaging.compiler.extension — The Extension class

3.4 PEP: Standard library package downloader (pysetup)

# Amongst other things, this needs to have a really good security

story (refusing to install unsigned packages by default, etc) packaging.depgraph — Dependency graph builder packaging.pypi — Interface to projects indexes packaging.pypi.client — High-level query API packaging.pypi.base — Base class for index crawlers packaging.pypi.dist — Classes representing query results packaging.pypi.simple — Crawler using the PyPI “simple” interface packaging.pypi.xmlrpc — Crawler using the PyPI XML-RPC interface packaging.tests.pypi_server — PyPI mock server packaging.fancy_getopt — Wrapper around the getopt module # Why does this exist?

3.4 PEP: Simple binary package distribution format

bdist_simple has been discussed enough times, finally seeing a PEP

for it would be nice :)

I think the main lesson to be learned here is that "fix packaging" is simply too big a task to be managed sensibly. Smaller goals like "Standardise versioning", "Fix package metadata", "Support uninstall" are hard enough to achieve, but also provide concrete milestones along the way to the larger goal.

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list