[Python-Dev] Internal documentation for egg formats now available (original) (raw)

M.-A. Lemburg mal at egenix.com
Fri Apr 28 18:20:44 CEST 2006

Phillip J. Eby wrote:

At 09:54 PM 4/27/2006 +0200, M.-A. Lemburg wrote:

Note that I was talking about the .pth file being writable, not the directory. Please stop this vague, handwaving FUD. You have yet to explain how this situation is supposed to arise. Is there some platform on which files with a .pth extension are automatically writable by users *when .py files are not also*? If not, then you are talking out of your... um, hat. If files are writable by other users by default, then .py files are too. Once again: *no new vector*.

Even if they are not writable by non-admins, the .pth files can point to directories which are. Uh huh. And how does that happen, exactly? Um, the user puts them there? What is your point, exactly? That people can do things insecurely if they're allowed near a computer?

You don't seem to want to see the problem:

By modifying site.py to take .pth files on all directories in PYTHONPATH into account, you extend the possibility to add "symbolic" links to a lot more directories than those which are normally monitored by the sysadmin.

Every entry in a .pth file is added to sys.path, it even allows imports to happen at .pth file scanning time.

A single malicious .pth file on PYTHONPATH could trick the user into running Python modules that he isn't really aware of.

Adding directories to PYTHONPATH which are writable by others is easy, just add '/tmp' as entry in some .pth file. Then put e.g. a modified os.py into /tmp and have it execute 'rm -rf /' at import time.

In summary: I don't think that allowing .pth files on all PYTHONPATH directories is the right way to go.

In order to solve your problem with users not being able to install user local copies of packages, we should instead discuss the possibility of having a standard directory on PYTHONPATH that is being searched for such packages and then allow .pth files only in that directory (which is under user control).

Please consider discussing such things on python-dev before implementing a hack that works around standard Python behavior.

Here's a HOWTO to optimize startup time, without loosing convenience: I meant, a HOWTO for setuptools users who care about this, although at the moment I have heard only from one -- and you're not, AFAIK, actually a setuptools user.

You asked for a HOWTO, I gave you one :-)

I have played with setuptools and it lots of interesting things, most of which are great and I applaud you for taking the effort in creating the tool.

However, as you know, I do have a few issues with it that I'd like to see resolved, which is why I keep trying to convince you of considering them (ever since you started in 2004 with this).

No, I'm talking about a format which has the same if not more benefits as what you're trying to achieve with the .egg file approach, but without all the magic and hacks.

It's not like this wouldn't be possible to achieve. That may or may not be true. Perhaps if you had participated in the original call to the distutils-sig for developing such a format (back in December 2004), perhaps the design would've been more to your liking. Oh wait... you did: http://mail.python.org/pipermail/distutils-sig/2004-December/004351.html

Indeed. And I suggested that you reconsider the idea to use ZIP files for installation (rather than just distribution):

http://mail.python.org/pipermail/distutils-sig/2004-December/004349.html

And if you replace 'syspathtools.use()' in that email, with 'pkgresources.require()', then it describes *exactly how setuptools works with .egg directories today*.

Interesting that you used that idea, because back then you didn't reply to the email. Looks like I deserve some credit ;-)

If you've already implemented this (which I wasn't aware of, since when I played with setuptools it kept installing .egg ZIP files), then why don't you make .egg directories the standard installation scheme, instead of insisting on having .egg ZIP files in site-packages/ ?

I've now played with it again and found that for some packages (e.g. kid and Paste) it installs these as .egg directories (horray!).

For other packages such as elementtree which are not available as egg files, it still creates egg files (by first downloading the source package, then creating an egg file and installing that).

It also uses the egg files for quite a few packages that are distributed as egg files.

So far, I've not found a pattern to this.

I wonder why you don't always create .egg directories.

Apparently, you hadn't yet thought of any the objections that you are now raising to the very scheme that you yourself proposed, until somebody else took the trouble to actually implement it.

Where am I doing that ?

And now you want to say that I never listen to or implement your proposals? Please. Your email is the first documentation on record of how this system works!

I'll repeat it again (hopefully the last time):

I believe that these requests are very reasonable.

Not really. Then I won't bother adding it, since nobody else asked for it. But don't ever say that I didn't offer you a real solution to the behavior you complained about. Meanwhile, I will take your choice as prima facie evidence that the things you are griping about have no real impact on you, and you are simply trying to make other people conform to your views of how things "should" be, rather than being interested in solving actual problems. It also makes it clear that your opinion about setuptools default installation behavior isn't relevant to any future Python-Dev discussion about setuptools' inclusion in the standard library, because it's: 1. Obviously not a real problem for you (else you'd accept the offer of a feature that would permanently solve it for you) 2. Not something that anybody else has complained about since I made --root automatically activate distutils compatibility In short, the credibility of your whining about this point and my supposed failure to accomodate you is now thoroughly debunked. I added an option to enable this behavior, I made other options enable the behavior where it could be reasonably assumed valid, and I offered you an option you could use to globally disable it for any package using setuptools, so that it would never affect you again. (And all of this... to disable the behavior that implements a scheme that you yourself proposed as the best way to do this sort of thing!)

Phillip, you're always side-stepping answering questions (and cutting away quoted email text doesn't help either).

Adding dozens of options to your system doesn't help remedy the problem, it only makes things worse.

Of course, there are always ways for me to have my way of doing things - from hacking setuptools to simply avoiding setuptools altogether.

But that's not what I want.

I would like setuptools to play nice with the rest of the Python world and become part of the standard distribution.

However, the level of magic introduced by setuptools is simply not what we're used to in the Python world, at least not around here (meaning python-dev).

The few suggestions I made would not hurt setuptools much, but would help put it back into the "explicit is better than implicit" camp, simply because a lot of magic could be removed.

As an example, you could avoid all the problems you currently have with C extensions in .egg ZIP files. You currently unzip just the C extension to a temporary cache directory and add proxies for in the original packages (a strategy that may or may not work depending on how the C extension is tied to the rest of the package).

By installing the .egg ZIP file as .egg directory, you'd avoid this completely.

Another example: the code that tries to find packages on the web. Screen scrapping is not really what I'd consider a safe way to find a package - the user might end up installing a completely different package.

Your argument is that you'd like things to work without the user having to pop up a browser and find the right URL for a package (your usually "just works" argument).

You should be aware that both package authors and users are capable of learning and adapting.

If we tell package authors to put a link to their eggs on PyPI along with an MD5 sum, then the users be in a much better position and could feel more secure about what they download and install.

It may take a while, but in the end will make the Python world better and safer for everyone, while still making the user experience a good one.

Plus, you could rip out all the code that currently tries to read web-sites and find packages that resemble the package name, reducing magic.

Have a nice weekend,

Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Apr 28 2006)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list