[Python-Dev] Status of packaging in 3.3 (original) (raw)

Chris McDonough chrism at plope.com
Thu Jun 21 16:12:06 CEST 2012

On 06/21/2012 09:29 AM, Nick Coghlan wrote:

My only comment on that is this: Since this is a problem related to the installation of Python distributions, it should deal with the problems that Python developers have more forcefully than non-Python developers and non-programmers. Thanks to venv, there's an alternative available that may be able to keep both of us happy: split the defaults. For system installs, adopt a vendor-centric, multi-language, easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding *.pth files by unpacking eggs to the file system). For venv installs, do whatever is most convenient for pure Python developers (e.g. leaving eggs packed and using *.pth files to extend sys.path within the venv).

I'd like to agree with this, but I think there's a distinction that needs to be made here that's maybe not obvious to everyone.

A tool to generate an OS-specific system package from a Python library project should be unrelated to a Python distribution installer. Instead, you'd use related tools that understood how to unpack the distribution packaging format to build one or more package structures. The resulting structures will be processed and then eventually installed by native OS install tools. But the Python distribution installer (e.g easy_install, pip, or some future similar tool) would just never come into play to create those structures. The Python distribution installer and the OS-specific build tool might share code to introspect and unpack files from the packaging format, but they'd otherwise have nothing to do with one another.

This seems like the most reasonable separation of concerns to me anyway, and I'd be willing to work on the code that would be shared by both the Python-level installer and by OS-level packaging tools.

One of Python's great virtues is its role as a glue language, and part of being an effective glue language is playing well with others. That should apply to packaging& distribution as well, not just to runtime bindings to tools written in other languages.

When we add the scientific users into the mix, we're actually getting to a third audience: multi-language developers that want to use Python's packaging utilities for their source and binary distribution formats. The Python community covers a broad spectrum of use cases, and I suspect that's one of the big reasons packaging can get so contentious - the goals end up being in direct conflict. Currently, I've identified at least half a dozen significant communities with very different needs (the names aren't meant to be all encompassing, just good representatives of each category, and many individuals will span multiple categories depending on which hat they're wearing at the time): Library authors: just want to quickly and easily publish their work on the Python package index in a way that is discoverable by others and allows feedback to reach them at their development site Web developers: creators of Python applications, relying primarily on other Python software and underlying OS provided functionality, potentially with some native extensions, that may need to run on multiple platforms, but can require installation using a language specific mechanism by technical staff Rich client developers: creators of Python applications relying primarily on other Python software and underlying OS provided functionality, potentially with native extensions, that need to run on multiple platforms, but must be installed using standard system utilities for the benefit of non-technical end users Enterprise developers: creators of Python or mixed language applications that need to integrate with corporate system administration policies (including packaging, auditing and configuration management) Scientists: creators of Python data analysis and modelling applications, with complex dependencies on software written in a variety of other languages and using various build systems Python embedders: developers that embed a Python runtime inside a larger application

I think we'll also need to put some limits on the goal independent of the union of everything all the audiences require.

Here's some scope suggestions that I believe could be shared by all of the audiences you list above except for embedders; I think that use case is pretty much separate. It might also leave "rich client developers" wanting, but no more than they're already wanting.

IOW, an installer should be about installing Python libraries and supporting files to a well-known location defined by the interpreter or venv that runs it, not full applications-that-require-persistent-state which just happen to be written in Python and which require deployment to arbitrary locations.

You shouldn't expect the Python packaging tools to install an instance of an application on a system, you should expect them to install enough code that would allow you to generate an instance of such an application. Most tools make that possible by installing a console script which can generate a sandbox that can be used to keep application state.

Hopefully this is preaching to the choir.

setuptools (or, perhaps, easyinstall, although I've seen enough posts about eggs being uploaded to PyPI to suspect otherwise), encourages the deployment of system configuration changes that alter the runtime environment of every single Python application executed on the system. That's simply not cool.

Again, it would help if you tried it in anger. What's the worst that could happen? You might like it! ;-) Oh, believe me, if I ever had distribution needs that required the power and flexibility of setuptools, I would reach for it in a heartbeat (in fact, I already use it today, albeit for tasks that ordinary distutils could probably handle). That said, I do get to cheat though - since I don't need to worry about cross-platform deployment, I can just use the relevant RPM hooks directly :)

Ideally this is all you'd ever need to care deeply about in an ideal world, too, given the separation of installer vs. system-packaging-support-tools outlined above.

You're right that most of my ire should be directed at the default behaviour of easyinstall rather than at setuptools itself, though. I shall moderate my expressed opinions accordingly.

Woot!

More information about the Python-Dev mailing list