[Python-Dev] Status of packaging in 3.3 (original) (raw)

Chris McDonough chrism at plope.com
Thu Jun 21 13:48:34 CEST 2012

On 06/21/2012 04:45 AM, Nick Coghlan wrote:

On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough<chrism at plope.com> wrote:

All of these are really pretty minor issues compared with the main benefit of not needing to ship everything with everything else. The killer feature is that developers can specify dependencies and users can have those dependencies installed automatically in a cross-platform way. Everything else is complete noise if this use case is not served. Cool. This is the kind of thing we need recorded in a PEP - there's a lot of domain knowledge floating around in the heads of packaging folks that needs to be captured so we can know *what the addition of packaging to the standard library is intended to fix*. And, like it or not, setuptools has a serious PR problem due to the fact it monkeypatches the standard library, uses *.pth files to alter sys.path for every installed application by default, actually uses the ability to run code in *.pth files and has hard to follow documentation to boot. I don't trust that I fully understand the import system on any machine with setuptools installed, because it is demonstrably happy to install state to the file system that will affect all Python programs running on the machine.

I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff on the default sys.path of the system Python of the box that's related to their software's concerns only. I don't understand why people accept this but get crazy about the fact that installing a setuptools distribution using easy_install changes the default sys.path.

Installing a distribution will change behavior whether or not sys.path is changed as a result. That's its purpose. The code that runs in the .pth file (there's only one that matters: easy_install.pth) just mutates sys.path. The end result is this: if you understand how sys.path works, you understand how eggs work. Each egg is addded to sys.path. That's all there is to it. It's the same as manually mutating a global PYTHONPATH, except you don't need to do it.

And note that this is not "setuptools" in general. It's easy_install in particular. Everything you've brought up so far I think is limited to easy_install. It doesn't happen when you use pip. I think it's a mistake that pip doesn't do it, but I think you have to make more accurate distinctions.

A packaging PEP needs to explain: - what needs to be done to eliminate any need for monkeypatching - what's involved in making sure that *.pth are not needed by default - making sure that executable code in implicitly loaded *.pth files isn't used at all

I'll note that these goals are completely sideways to any actual functional goal. It'd be a shame to have monkeypatching going on, but the other stuff I don't think are reasonable goals. Instead they represent fears, and those fears just need to be managed.

I think trying to achieve this is actually the genesis of the original distribute fork, that subsequently became distutils2 as Tarek discovered how much of the complexity in setuptools was actually due to the desire to not officially fork distutils (and instead monkeypatch it, effectively creating a runtime fork).

However, for those of us that weren't directly involved, this is all still a strange mystery dealt with by other people. I've cribbed together bits and pieces just from following the fragments of the discussions that have happened on python-dev and at PyCon US, but if we want the madness to ever stop, then *the problems with the status quo* need to be written down so that other core developers can understand them.

It'd also be useful if other core developers actually tried to use setuptools in anger. That'd be a good start towards understanding some of its tradeoffs. People can write this stuff down til they're blue in the face, but if core devs don't try the stuff, they'll always fear it.

In fact, I just remembered that Tarek has written a lot of this down, just not in PEP form: http://www.aosabook.org/en/packaging.html

Cool.

More information about the Python-Dev mailing list