[Python-Dev] Internal documentation for egg formats now available (original) (raw)

M.-A. Lemburg mal at egenix.com
Thu Apr 27 21:54:24 CEST 2006

Phillip J. Eby wrote:

At 06:47 PM 4/27/2006 +0200, M.-A. Lemburg wrote:

Just read that you are hijacking site.py for setuptools' "just works" purposes. "hijacking" isn't the word I'd use; "wrapping" is what it actually does. The standard site.py is executed, there is just some pre- and post-processing of sys.path.

Whatever you call it: you're fiddling with the standard Python distribution again, without letting your users know about it.

Note that sitecustomize.py is intended for site local changes that admin wants to implement.

Please be aware that by allowing .pth files in all PYTHONPATH directories you are opening up a security hole - anyone with write-permission to one of these .pth files could manipulate other user's use of Python. FUD. Write access to a PYTHONPATH-listed directory already implies complete control over the user's Python environment. This doesn't introduce any issues that aren't implicit in the very existence of PYTHONPATH.

Really ? How often do you check the contents of a .pth file ? How often do you check sys.path of a running interpreter to see whether some .pth file has added extra entries that you weren't aware of ?

Note that I was talking about the .pth file being writable, not the directory. Even if they are not writable by non-admins, the .pth files can point to directories which are.

Combine that with the -m option of the Python interpreter and you have the basis for a nice exploit.

That's the reason why only site-packages .pth files are taken into account, since normally only root has write access to this directory. False. On OS X, Python processes any .pth files that are found in the ~/Library/Python2.x/site-packages directory.

Hmm, that was new to me. OTOH, it's a very special case and falls under the "normally only root" clause :-) - Mac users are often enough their own admins; I suppose that's why Jack added this:

        if sys.platform == 'darwin':
            # for framework builds *only* we add the standard Apple
            # locations. Currently only per-user, but /Library and
            # /Network/Library could be added too

(Which means, by the way, that OS X users don't need most of these hacks; they just point their install directory to their personal site-packages, and it already Just Works. Setuptools only introduces PYTHONPATH hacks to make this work on other platforms.)

The added startup time for scanning PYTHONPATH for .pth files and processing them is also not mentioned in the documentation. Every Python invocation would have to pay for this - regardless of whether eggs are used or not. FUD again. This happens if and only if: 1. You used easyinstall to install a package in such a way that the .pth file is required (i.e., you don't choose multi-version mode or single-version externally-managed mode) 2. You include the affected directory in PYTHONPATH So the idea that "every Python invocation would have to pay for this" is false. People who care about the performance have plenty of options for controlling this. Is there a nice HOWTO that explains what to do if you care more about performance than convenience? No. Feel free to contribute one.

Here's a HOWTO to optimize startup time, without loosing convenience:

If you run a lot of scripts, you'll also want to do keep this in mind:

More startup time can be saved by using an approach like the one described in mxCGIPython which adds the complete Python stdlib to the interpreter binary, effectively turning the Python installation into a single file:

http://www.egenix.com/files/python/mxCGIPython.html

I really wish that we could agree on an installation format for package (meta) data that does not rely on ZIP files. There is one already, and it's used if you select single-version externally-managed mode explicitly, or if you install using --root.

No, I'm talking about a format which has the same if not more benefits as what you're trying to achieve with the .egg file approach, but without all the magic and hacks.

It's not like this wouldn't be possible to achieve.

All the unnecessary magic that you have in your design would just go away - together with most of the issues people on this list have with it. Would you settle for a way to make a one-time ~/.pydistutils.cfg entry that will make setuptools act the way you want it to? That is, a way to make setuptools-based packages default to --single-version-externally-managed mode for installation on a given machine or machine(s)? That way, you'll never have to wonder whether a package uses setuptools or not, you can just "setup.py install" and be happy.

Not really.

I would like Python to remain a place that's free of unnecessary magic in the first place, not by having to configure setuptools to disable it's magic.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Apr 27 2006)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list