| msg48111 - (view) |
Author: Bob Ippolito (bob.ippolito) *  |
Date: 2005-04-01 05:24 |
| The current site.py has three major deficiencies: (1) All site dirs must exist on the filesystem: Since PEP 302 (New Import Hooks) was adopted, this is not necessarily true. sys.meta_path and sys.path_hooks can have valid uses for non- existent paths. Even the standard zipimport hook supports in-zip- file paths (i.e. foo.zip/bar). (2) The directories added to sys.path by .pth files are not scanned for further .pth files. If they were, you could make life much easier on developers and users of multi-user systems. For example, it would be possible for an administrator to drop in a .pth file into the system-wide site-packages to allow users to have their own local site-packages folder. Currently, you could try this, but it wouldn't work because many packages such as PIL, Numeric, and PyObjC take advantage of .pth files during their installation. (3) To support the above use case, .pth files should be allowed to use os.path.expanduser(), so you can toss a tilde in front and do the right thing. Currently, the only way to support (2) is to use an ugly "import" pth hook. Attached is a patch to CVS HEAD that: (1) Removes the os.path.exists() / os.path.isdir() restrictions (2) Changes the .pth reader to use addsitedir() (top down) rather than sys.path.append() (3) makepath() uses os.path.expanduser() after the os.path.join(). |
|
|
| msg48112 - (view) |
Author: Just van Rossum (jvr) *  |
Date: 2005-04-07 08:33 |
| Logged In: YES user_id=92689 Patch looks good, the new functionality is great to have, recommend apply. |
|
|
| msg48113 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2005-06-03 00:14 |
| Logged In: YES user_id=6380 Hm. Looking for .pth files only in site-packages was intentional; it will slow down startup considerably if you have to do an os.listdir() of every directory on sys.path (even worse if some are on NFS!). I'd recommend against that part. |
|
|
| msg48114 - (view) |
Author: Bob Ippolito (bob.ippolito) * |
Date: 2005-06-03 00:21 |
| Logged In: YES user_id=139309 If you read closer, it only scans directories added by .pth files, not all directories on sys.path. This is to facilitate better encapsulation, so an admin user can create a .pth file that says "~/lib/python2.4/site- packages" and any user that starts Python will have their own site- packages dir, where .pth files work, without resorting to PYTHONPATH. This way you can use distutils to install packages into your homedir, and they will actually work as expected. PYTHONPATH is utterly disastrous if you regularly deal with several different Python interpreters. |
|
|
| msg48115 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2005-06-03 00:28 |
| Logged In: YES user_id=6380 It still looks like a potentially considerable slowdown (since every directory added by a .pth file is scanned recursively), and I'm doubting the usefulness. Why should a site admin have the power to make Python search in an arbitrary place in my home directory? That could cause surprises too! When you are regularly switching between multiple Python interpreters, wouldn't it make more sense to have a few aliases (or shell scripts, .bat files, etc.) that set PYTHONPATH without exporting it? Methinks that many people have dealt with this issue *without* feeling the need to patch site.py. (BTW I'm okay with the other changes, from your description.) |
|
|
| msg48116 - (view) |
Author: Bob Ippolito (bob.ippolito) * |
Date: 2005-06-03 00:41 |
| Logged In: YES user_id=139309 Well in practice, .pth files aren't used a lot (most people will see maybe wxPython, PIL, and Numeric), so those three os.listdir() probably aren't going to be significant. If users are creating pth files themselves (which they probably should be doing, during development), then they'd likely take advantage of the new features. As far as power for site admins goes.. why should they have power to install Python at all? Lots of things an admin could do can cause surprises. PYTHONPATH is only useful when dealing with the command prompt directly, which is probably not the case for users of Mac OS X and Windows. It shouldn't have to be, anyway. Mac OS X doesn't even ship with a GUI interface to edit login environment variables, and the normal rc scripts aren't run in the context of LaunchServices. Also, consider the situation where a user of some shared web server is writing CGI scripts that depend on Numeric installed in their home directory. They'd need to know to add the following to the top of *EVERY* cgi script: import site import os site.addsitedir(os.path.expanduser("~/lib/python" + sys.version[:3] + "/ site-packages")) If the admin had configured a directory to put things by installing a .pth file that pointed there, they simply would have to tell users what flags to pass to distutils. For Mac OS X, there is a workaround, in that ~/Library/Python/2.X/site- packages is a second site dir, but this doesn't need to be a workaround and should be available elsewhere. |
|
|
| msg48117 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2005-06-03 00:43 |
| Logged In: YES user_id=6380 Sorry, I'm still -1, but I don't have the time to discuss it further. All your examples seem to be Mac OS X based; perhaps you can come up with a Mac OS X specific solution? |
|
|
| msg48118 - (view) |
Author: Bob Ippolito (bob.ippolito) * |
Date: 2005-06-03 00:44 |
| Logged In: YES user_id=139309 Did you read my whole comment? I did come up with an example that is not Mac OS X based: CGIs on a web server. |
|
|
| msg48119 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2005-06-03 00:49 |
| Logged In: YES user_id=6380 Do such shared web servers really exist? It sounds like an invitation to being hacked to me. Given that this is a last resort (it would make more sense to petition the ISP to install Numeric) I don't see that patching sys.path at the top of the cgi script is so bad. |
|
|
| msg48120 - (view) |
Author: Bob Ippolito (bob.ippolito) * |
Date: 2005-06-03 00:52 |
| Logged In: YES user_id=139309 And if the listdir performance is the only killer, the "recursive site" protocol could be changed slightly -- it could depend on the existence of some file (e.g. "python-site"), so the os.listdir() turns into a false os.path.exists() in the general case (where it is not intended to be added as another site directory). Alternatively, some token in the .pth file could say "this points to another site dir". Currently that (undocumented) token actually exists, but it looks like this: import site; site.addsitedir("....") |
|
|
| msg48121 - (view) |
Author: Bob Ippolito (bob.ippolito) * |
Date: 2005-06-03 00:57 |
| Logged In: YES user_id=139309 Such shared web servers do exist, and are actually quite common for low-cost hosting (i.e. pythonmac.org is sitting on one of these right now - dreamhost.com). However, they setuid first so that they are not a security hole waiting to happen. This is done with Apache using the suEXEC module <http:// httpd.apache.org/docs/suexec.html>. Petitioning the ISP to go through the trouble of auditing and installing some package for all users doesn't do you much good if you're trying to get something up and running that day. |
|
|
| msg59784 - (view) |
Author: Christian Heimes (christian.heimes) * |
Date: 2008-01-12 04:28 |
| I'm going to check the patch for my new PEP about per user site directories. |
|
|
| msg62773 - (view) |
Author: Christian Heimes (christian.heimes) * |
Date: 2008-02-23 17:02 |
| The feature request is superseded by my PEP 370 (http://www.python.org/dev/peps/pep-0370/). As far as I understand your proposal the PEP covers your use cases. |
|
|