[Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) (original) (raw)

VanL van.lindberg at gmail.com
Thu Mar 22 19:57:17 CET 2012

Hi Paul,

To start with, I appreciate your comments, and it is worth having both sides expressed.

On 3/22/2012 12:59 PM, Paul Moore wrote:

I'm repeating myself here after I promised not to. My apologies, but I don't think this posting captures the debate completely. One reason I suggested a PEP is to better ensure that the arguments both pro and con were captured, as that is a key part of the PEP process.

I would be happy to write up a PEP.

First of all, this difference is almost entirely invisible. Apart from possibly setting PATH (once!) users should not be digging around in the Python installation directory. Certainly on Windows, it is a very unusual application that expects users to even care how the application is laid out internally. And I suspect that is also true on Unix and Mac.

This is a good point; it is mostly visible in the virtualenvs. If it only changed in virtualenvs, I would be happy. The policy, though, is that the virtualenv follows the platform policy.

Secondly, the layouts are not as similar as you claim here, if I understand things correctly. on Unix Python is installed in /usr/local/bin so there isn't even a "Python installation directory" there. And Macs use some sort of Mac-specific bundle technology as I understand it. To be honest, I don't think that there's a lot of similarity even with your proposed changes.

I was summarizing here because, frankly, there are hardly any OS/2 users, so it would be mostly Windows users affected by this change. Also as noted, I suggest that all platforms standardize on bin, lib, and include, just as I laid out.

That said, while I think that the above is a good idea, my personal ambitions are more modest: If the names of the top-level directories only were changed to 'bin', 'lib', and 'include' - never mind differences under 'lib' - I would be happy. In fact, even the one change of 'Scripts' to 'bin' everywhere would get 90% of my uses.

The issue with virtualenv and pyvenv may be more significant. But you're only mentioning those incidentally.

I am approaching it from the platform level because of the policy that virtualenvs match the platform install layout. If instead virtualenv layouts were standardized, that would end up making me just as happy.

(I also get a sense of undue haste - "we can change the Python layout for 3.3, but changing packaging and virtualenv is a much longer process"...)

Honestly, I didn't expect that much resistance. None of the people I talked to in person even cared, or if they did, they thought that consistency was a benefit. But now that virtualenvs are going in in 3.3, I see this as the last good chance to change this.

This presupposes that your development workflow - developing in place in the virtualenv itself - is "the obvious approach". From what I've seen of tools like virtualenvwrapper, you're not alone in this. [...] But can you acknowledge (and document) that "change your way of working" is another alternative to "change Python".

Acknowledged. What you say is true - but people wanted to know what the benefit would be. I laid out my concrete use-case as a rationale. And as you note, I am not alone in this type of development. Sure, I care here because it affects my style of development, and there are other styles that have other benefits (and tradeoffs). I don't see that this part of the proposal would negatively affect those styles.

While a few people have wondered why I would want this -- hopefully answered above -- I have not heard any opposition to this part of the proposal. See above. There has been opposition from a number of people. It's relatively mild, simply because it's a niche area and doesn't cause huge pain, but it's there. And you seem (based on the above analysis) to be overstating the benefits, so the summary here is weighted heavily in favour of change.

If I have misrepresented anyone, I am sorry - but to the best of my understanding, no one had (prior to you, right now) objected to this part of the proposal. Mark, at least, specified that his concern was with the moving of the python binary and that he didn't care about this part. I believe Tim indicated that too, but perhaps I have on my rose-colored glasses and misunderstood him.

Also, you have made no attempt that I've seen to address the question of why this is important enough to break backward compatibility. Maybe it is - but why? Backward compatibility is a very strong rule, and should be broken only with good justification. Consistency, and "it makes my way of working easier" really shouldn't be sufficient.

In general, yes, I agree with you. However, the break with backwards compatibility is, as you point out, minor, and there is a benefit to consistency - especially given virtualenv-centric development.

Has anyone checked whether this will affect people like Enthought and ActiveState who distribute their own versions of Python? Is ActiveState's PPM tool affected?

I have been running like this for several years across multiple Python versions, so I have experience with the "breakage" from this part of the proposal. I have found four packages that would need to be updated: Pip, virtualenv, PyPM, and Egginst would need 1-2 line patches. I have these patches, I would/could provide them. Generally these tools have something like:

if platform == 'win32': bin_dir = 'Scripts' else: bin_dir = 'bin'

The patches just remove the special casing - bin_dir just gets set to 'bin'.

Note - that is not "Regularizing the layout". You have not made any changes to OS/2 (which matches Windows at the moment). And it doesn't match Posix at all.

See my 'summarizing' content above - layouts should match. I also didn't want to post chunks of sysconfig.py. You also missed distutils.command.install, which is subtly different.

The OS X framework is already posixy inside, and virtualenvs on Mac OS X follow the posix-user layout. It is true that some Linux distributions place lib, include, etc in the system-wide directories. However, altinstalls that are confined to a directory (i.e., have a 'layout' in the sense that I am describing.

My proposal would prohibit people from having multiple versions installed on top of each other, but they would clobber each other anyway with the current setup.

Again, I have not heard anyone objecting to this part of the proposal as it is laid out here. (Paul had a concern with the lib directory earlier, but he said he was ok with the above). That's somewhat odd, as I did hear a number of concerns. But it was certainly not easy to tell which related to which part of the proposal. And all of the objections were mild, mostly because it's not a huge practical issue either way.

True. And again, I tried not to misrepresent anyone, but I have not heard anyone (including you) who would actually have code broken by this change. Do you have code? I really, really would like to know. I have not found anything that needs a patch except for the four packages above.

Personally, my main concerns are around procedure and policy. The more the discussion goes on, the more I feel that there should be a PEP to capture the details of the debate clearly. Too much is getting lost in the noise. And I think you should provide a clear statement of why this issue is important enough to justify violating the backward compatibility policies. As Mark said (I think it was Mark...) if this had been proposed for 3.0, it would have been fine. Now we're at 3.2 with 3.3 close to release, and it just seems too late to be worth the risk. One plus point about your posting this separately. It's made me think through the issue in a bit more detail, and I'm now a solid -1 on the proposal.

I have been trying at various PyCons and in various conversations to move this for years. No one cares. The current urgency is driven by pyvenv - changes now will be much, much easier than changes later.

Again, I am happy to write a PEP. If I were to summarize (on this issue only):

  1. The current backwards compatibility hit is minimal; I would be happy to contact and provide patches to the four packages I have found (and anyone else who wants one). Backwards compatibility in the future will probably be harder to deal with.
  2. There are advantages to cross-platform consistency and to virtualenv-based development. I believe that these will grow in the future.
  3. Most people won't care. To the extent that people notice, I think they will appreciate the consistency.

More information about the Python-Dev mailing list