[Python-Dev] Status of packaging in 3.3 (original) (raw)

Chris McDonough chrism at plope.com
Thu Jun 21 14:51:04 CEST 2012

On 06/21/2012 08:21 AM, Nick Coghlan wrote:

Installing a distribution will change behavior whether or not sys.path is changed as a result. That's its purpose. No it won't. An ordinary package will only change the behaviour of Python applications that import a package by that name. Other Python applications will be completely unaffected (as it should be).

If a Python application is effected by a change to sys.path which doesn't impact modules it uses, then that Python application is plain broken, because the developer of that application cannot make assumptions about what a user does to sys.path unrelated to the modules it requires. This is completely independent of easy_install.

Any Python application is going to be effected by the installation of a distribution that does impact modules it imports, whether sys.path is used to change the working set of modules or not.

So what concrete situation are we actually talking about here?

The code that runs in the .pth file (there's only one that matters: easyinstall.pth) just mutates sys.path. The end result is this: if you understand how sys.path works, you understand how eggs work. Each egg is addded to sys.path. That's all there is to it. It's the same as manually mutating a global PYTHONPATH, except you don't need to do it. Yes, it's the same as mutating PYTHONPATH. That's a similarly bad system global change. Individual libraries do not have the right to change the sys.path seen on initialisation by every other Python application on that system.

Is it reasonable to even assume there is only one-sys.path-to-rule-them-all? And that users install "the set of libraries they need" into a common place? This quickly turns into failure, because Python is used for many, many tasks, and those tasks sometimes require conflicting versions of libraries. This is the root cause of why virtualenv exists and is popular.

The reason it's disappointing to see OS vendors mutating the default sys.path is because they put very old versions of very common non-stdlib packages (e.g. zope.interface, lxml) on sys.path by default. The path is tainted out of the box for anyone who wants to use the system Python for development of newer software. So at some point they invariably punt to virtualenv or a virtualenv-like system where the OS-vendor-provided path is not present.

If Python supported the installation of multiple versions of the same module and versioned imports, both PYTHONPATH and virtualenv would be much less important. But given lack of enthusiasm for that, I don't think it's reasonable to assume there is only one sys.path on every system.

I sympathize, however, with Oscar's report that PYTHONPATH can't the setuptools-derived path. That's indeed a mistake that a future tool should not make.

And note that this is not "setuptools" in general. It's easyinstall in particular. Everything you've brought up so far I think is limited to easyinstall. It doesn't happen when you use pip. I think it's a mistake that pip doesn't do it, but I think you have to make more accurate distinctions. What part of "PR problem" was unclear? setuptools and easyinstall are inextricably linked in everyone's minds, just like pip and distribute.

Hopefully for the purposes of the discussion, folks here can make the mental separation between setuptools and easy_install. We can't help what other folks think in the meantime, certainly not solely by making technological compromises anyway.

A packaging PEP needs to explain: - what needs to be done to eliminate any need for monkeypatching - what's involved in making sure that *.pth are not needed by default - making sure that executable code in implicitly loaded *.pth files isn't used at all

I'll note that these goals are completely sideways to any actual functional goal. It'd be a shame to have monkeypatching going on, but the other stuff I don't think are reasonable goals. Instead they represent fears, and those fears just need to be managed. No, they reflect the mindset of someone with configuration management and auditing responsibilities for shared systems with multiple applications installed which may be written in a variety of languages, not just Python. You may not care about those people, but I do.

I care about deploying Python-based applications to many platforms. You care about deploying multilanguage-based applications to a single platform. There's going to be conflict there.

My only comment on that is this: Since this is a problem related to the installation of Python distributions, it should deal with the problems that Python developers have more forcefully than non-Python developers and non-programmers.

It'd also be useful if other core developers actually tried to use setuptools in anger. That'd be a good start towards understanding some of its tradeoffs. People can write this stuff down til they're blue in the face, but if core devs don't try the stuff, they'll always fear it. setuptools (or, perhaps, easyinstall, although I've seen enough posts about eggs being uploaded to PyPI to suspect otherwise), encourages the deployment of system configuration changes that alter the runtime environment of every single Python application executed on the system. That's simply not cool.

Again, it would help if you tried it in anger. What's the worst that could happen? You might like it! ;-)

More information about the Python-Dev mailing list