[Python-Dev] Status of packaging in 3.3 (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Jun 21 14:21:55 CEST 2012

On Thu, Jun 21, 2012 at 9:48 PM, Chris McDonough <chrism at plope.com> wrote:

On 06/21/2012 04:45 AM, Nick Coghlan wrote:

And, like it or not, setuptools has a serious PR problem due to the fact it monkeypatches the standard library, uses *.pth files to alter sys.path for every installed application by default, actually uses the ability to run code in *.pth files and has hard to follow documentation to boot. I don't trust that I fully understand the import system on any machine with setuptools installed, because it is demonstrably happy to install state to the file system that will affect all Python programs running on the machine.

I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff on the default sys.path of the system Python of the box that's related to their software's concerns only.  I don't understand why people accept this but get crazy about the fact that installing a setuptools distribution using easyinstall changes the default sys.path.

Because the vendor gets to decide what goes into the base install of the OS. If I'm using the system Python, then I expect sys.path to contain the system paths, just as I expect gcc to be able to see the system include paths. If I don't want that, I'll use virtualenv or a completely separate Python installation.

However, when I install a new Python package into site-packages it should just sit there and have zero impact on other Python applications that don't import that package. As soon as someone installs a *.pth file, however, that's no longer the case - every Python application on that machine will now be scanning additional paths for modules whether it wants to or not. It's unnecessary coupling between components that should be completely independent of each other.

Now, *.pth support in the interpreter certainly cannot be blamed on setuptools, but encouraging use of a packaging format that effectively requires them certainly can be.

It's similar to the reason why monkeypatching and global environment variable modifications (including PYTHONPATH) are a problem: as soon as you start doing that kind of thing, you're introducing coupling that shouldn't exist. If there is no better solution, then sure, do it as a near term workaround, but that isn't the same as accepting it as the long term answer.

Installing a distribution will change behavior whether or not sys.path is changed as a result.  That's its purpose.

No it won't. An ordinary package will only change the behaviour of Python applications that import a package by that name. Other Python applications will be completely unaffected (as it should be).

 The code that runs in the .pth file (there's only one that matters: easyinstall.pth) just mutates sys.path.  The end result is this: if you understand how sys.path works, you understand how eggs work.  Each egg is addded to sys.path.  That's all there is to it.  It's the same as manually mutating a global PYTHONPATH, except you don't need to do it.

Yes, it's the same as mutating PYTHONPATH. That's a similarly bad system global change. Individual libraries do not have the right to change the sys.path seen on initialisation by every other Python application on that system.

And note that this is not "setuptools" in general.  It's easyinstall in particular.  Everything you've brought up so far I think is limited to easyinstall.  It doesn't happen when you use pip.  I think it's a mistake that pip doesn't do it, but I think you have to make more accurate distinctions.

What part of "PR problem" was unclear? setuptools and easy_install are inextricably linked in everyone's minds, just like pip and distribute.

A packaging PEP needs to explain: - what needs to be done to eliminate any need for monkeypatching - what's involved in making sure that *.pth are not needed by default - making sure that executable code in implicitly loaded *.pth files isn't used at all I'll note that these goals are completely sideways to any actual functional goal.  It'd be a shame to have monkeypatching going on, but the other stuff I don't think are reasonable goals.  Instead they represent fears, and those fears just need to be managed.

No, they reflect the mindset of someone with configuration management and auditing responsibilities for shared systems with multiple applications installed which may be written in a variety of languages, not just Python. You may not care about those people, but I do.

It'd also be useful if other core developers actually tried to use setuptools in anger.  That'd be a good start towards understanding some of its tradeoffs.  People can write this stuff down til they're blue in the face, but if core devs  don't try the stuff, they'll always fear it.

setuptools (or, perhaps, easy_install, although I've seen enough posts about eggs being uploaded to PyPI to suspect otherwise), encourages the deployment of system configuration changes that alter the runtime environment of every single Python application executed on the system. That's simply not cool.

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list