[Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) (original) (raw)

Paul Moore p.f.moore at gmail.com
Thu Mar 22 18:59:34 CET 2012

On 22 March 2012 14:17, VanL <van.lindberg at gmail.com> wrote:

As this has been brought up a couple times in this subthread, I figured that I would lay out the rationale here.

I'm repeating myself here after I promised not to. My apologies, but I don't think this posting captures the debate completely. One reason I suggested a PEP is to better ensure that the arguments both pro and con were captured, as that is a key part of the PEP process.

One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"),

First of all, this difference is almost entirely invisible. Apart from possibly setting PATH (once!) users should not be digging around in the Python installation directory. Certainly on Windows, it is a very unusual application that expects users to even care how the application is laid out internally. And I suspect that is also true on Unix and Mac.

Secondly, the layouts are not as similar as you claim here, if I understand things correctly. on Unix Python is installed in /usr/local/bin so there isn't even a "Python installation directory" there. And Macs use some sort of Mac-specific bundle technology as I understand it. To be honest, I don't think that there's a lot of similarity even with your proposed changes.

but a full listing of the layouts shows substantial differences in layout and capitalization across platforms.

That's true, but largely incidental. And given that (a) Windows is case insensitive and (b) the capitalisation, although inconsistent, follows platform standards (Unix all lowercase, Windows capitalised) it makes little practical difference.

Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not.

Given that on Windows the Python library is usually in something like C:\Python32\Lib whereas on Unix it's in /usr/lib/python3.2 (I think), the difference is reasonable because the Python base location (C:\Python32 on Windows vs /usr on Unix) is version specific in one case but not the other. To keep the correspondence complete, you should be suggesting installing in /python32 on Unix (which I doubt would gain much support :-))

This may seem like a harmless inconsistency, and if that were all it was, I wouldn't care. (That said, cross-platform consistency is its own good). But it becomes a real pain when combined with tools like virtualenv or the new pyvenv to create cross-platform development environments.

The issue with virtualenv and pyvenv may be more significant. But you're only mentioning those incidentally. As a straw-man suggestion, why can virtualenv not be changed to maintain a platform-neutral layout in spite of what core Python does? This is a straw-man because I know the answer, but can we get the point out in the open - it's related to how distutils installs code, and that in turn hits you straight up against the distutils freeze. If distutils' behaviour is the issue here, then argue for more flexibility in packaging, and use that extra flexibility to argue for changes to virtualenv and pyvenv to maintain a standard cross-platform layout. Breaking the Python installation layout isn't the only option here, and I'd like to see a clear analysis of the tradeoffs. (I also get a sense of undue haste - "we can change the Python layout for 3.3, but changing packaging and virtualenv is a much longer process"...)

In particular, I regularly do development on both Windows and a Mac, and then deploy on Linux. I do this in virtualenvs, so that I have a controlled and regular environment. I keep them in sync using source control.

The problem comes when I have executable scripts that I want to include in my dvcs - I can't have it in the obvious place - the binaries directory - because the name of the directory changes when you move between platforms. More concretely, I can't hg add "Scripts/runner.py? on my windows environment (where it is put in the PATH by virtualenv) and thendo a pull on Mac or Linux and have it end up properly in "bin/runner.py" which is the correct PATH for those platforms.

This presupposes that your development workflow - developing in place in the virtualenv itself - is "the obvious approach". From what I've seen of tools like virtualenvwrapper, you're not alone in this. And I'm pretty new to using virtualenv so I wouldn't like to claim to be any expert. But can I respectfully suggest that other ways of working wouldn't hit these issues? WhatI do is develop my project in a project specific directory, just as I would if I were using the system Python. And I have an activated virtualenv located somewhere else that I install required 3rd party modules in, etc. I then do standard "pip install" to install and test my project, etc etc. This adds an "install to test" cycle (or you use something like setuptools' "deploy" techniques, which I know nothing about myself) but in exchange you are independent of the virtualenv layout (as pip install puts your scripts in the appropriate Scripts/bin directory). Also, by doing this you test your installation process, which you don't if you develop in place.

Again, you may not like this way of working, and that's fine. But can you acknowledge (and document) that "change your way of working" is another alternative to "change Python".

This applies anytime there are executable scripts that you want to manage using source control across platforms. Django projects regularly have these, and I suspect we will be seeing more of this with the new "project" support in virtualenvwrapper.

... if you develop inside the virtualenv.

While a few people have wondered why I would want this -- hopefully answered above -- I have not heard any opposition to this part of the proposal.

See above. There has been opposition from a number of people. It's relatively mild, simply because it's a niche area and doesn't cause huge pain, but it's there. And you seem (based on the above analysis) to be overstating the benefits, so the summary here is weighted heavily in favour of change.

Also, you have made no attempt that I've seen to address the question of why this is important enough to break backward compatibility. Maybe it is - but why? Backward compatibility is a very strong rule, and should be broken only with good justification. Consistency, and "it makes my way of working easier" really shouldn't be sufficient.

Has anyone checked whether this will affect people like Enthought and ActiveState who distribute their own versions of Python? Is ActiveState's PPM tool affected?

This first proposal is just to make the names of the directories match across platforms. There are six keys defined in the installer files (sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib', 'platlib', 'headers', 'scripts',  and 'data'.

Currently on Windows, there are two different layouts defined:  'nt': {  'stdlib': '{base}/Lib',  'platstdlib': '{base}/Lib',  'purelib': '{base}/Lib/site-packages',  'platlib': '{base}/Lib/site-packages',  'include': '{base}/Include',  'platinclude': '{base}/Include',  'scripts': '{base}/Scripts',  'data'   : '{base}',  },  'ntuser': {  'stdlib': '{userbase}/Python{pyversionnodot}',  'platstdlib': '{userbase}/Python{pyversionnodot}',  'purelib': '{userbase}/Python{pyversionnodot}/site-packages',  'platlib': '{userbase}/Python{pyversionnodot}/site-packages',  'include': '{userbase}/Python{pyversionnodot}/Include',  'scripts': '{userbase}/Scripts',  'data'   : '{userbase}',  },

The proposal is to make all the layouts change to:  'nt': {  'stdlib': '{base}/lib',  'platstdlib': '{base}/lib',  'purelib': '{base}/lib/site-packages',  'platlib': '{base}/lib/site-packages',  'include': '{base}/include',  'platinclude': '{base}/include',  'scripts': '{base}/bin',  'data'   : '{base}',  }, The change here is that 'Scripts' will change to 'bin' and the capitalization will be removed. Also, "user installs" of Python will have the same internal layout as "system installs" of Python. This will also, not coincidentally, match the install layout for posix, at least with regard to the 'bin', 'lib', and 'include' directories.

Note - that is not "Regularizing the layout". You have not made any changes to OS/2 (which matches Windows at the moment). And it doesn't match Posix at all. I won't copy a large chunk of sysconfig.py in here, but I'm tempted to, because your proposal as described really doesn't match the reality of sysconfig._INSTALL_SCHEMES. I'd encourage people to go and read sysconfig.py for details. There really isn't much consistency at the moment, and fiddling with Windows but nothing else doesn't regularise anything.

Again, I have not heard anyone objecting to this part of the proposal as it is laid out here. (Paul had a concern with the lib directory earlier, but he said he was ok with the above).

That's somewhat odd, as I did hear a number of concerns. But it was certainly not easy to tell which related to which part of the proposal. And all of the objections were mild, mostly because it's not a huge practical issue either way.

Please let me know if you have any problems or concerns with this part 1.

Personally, my main concerns are around procedure and policy. The more the discussion goes on, the more I feel that there should be a PEP to capture the details of the debate clearly. Too much is getting lost in the noise. And I think you should provide a clear statement of why this issue is important enough to justify violating the backward compatibility policies.

As Mark said (I think it was Mark...) if this had been proposed for 3.0, it would have been fine. Now we're at 3.2 with 3.3 close to release, and it just seems too late to be worth the risk.

One plus point about your posting this separately. It's made me think through the issue in a bit more detail, and I'm now a solid -1 on the proposal.

Paul.

More information about the Python-Dev mailing list