[Python-Dev] magic in setuptools (Was: setuptools in the stdlib) (original) (raw)

"Martin v. Löwis" martin at v.loewis.de
Thu Apr 20 09:42:20 CEST 2006

I don't understand it. Have you read the manuals?

You mean,

http://peak.telecommunity.com/DevCenter/setuptools

Yes, I did. However, this would only enable me to use it (should I ever find a need). What I really need to understand is how all this stuff works inside.

Now, I haven't actually tried to learn it so far. Fredrik's question was not "who thinks he could learn it" (I certainly could), I read his question as "who does understand it, as of today". I can't faithfully answer "yes" to that question.

So far, I believe nobody has said "yes, I do understand it, and could maintain it if Phillip Eby became unavailable".

Please define "magic".

"Functionality with no obvious implementation, or functionality implemented in an unobvious way".

Better, please point to the API functions or classes, or the setup commands documented in the manual, to show me what these things are that appear to be magic.

Here are some that I either already knew, or discovered just now:

  1. setuptools manages to replace pydoc on earlier Python versions. MAL assumed it does so by overwriting the files. This would be bad practice, but is the obvious solution. You then told him that this is not what actually happens (so it's already magic); instead, you arrange that Python finds your version before it finds its own. Again, this is still magic: How does it do that? If you append to sys.path, Python would still find its own version before it finds yours. So perhaps you insert another path at sys.path[0]? This would also be bad practice, but it would be the next-obvious approach. But I fully expect that you are now telling me that this is still not how it works.

  2. setuptools manages to install code deep inside a subdirectory of site-python, and manages to install multiple versions simultaneously. How does that work? The first part (packages outside sys.path) can be solved with a .pth file (which I already consider bad practice, as it predates the package concept of Python); but that can't work for multiple versions: magic.

    I (think I) know the answer: there is a single .pth file, and that is edited on package installation time. This is a completely non-obvious approach. If Python needs to support multiple versions of a package (and there is apparently a need), that should be supported "properly", with a clear design, in a PEP.

  3. namespace packages. From the documentation, I see that you pass namespace_packages= to setup, and that you put

    import('pkg_resources').declare_namespace(name)

    into the namespace package. How does that work? The documentation only says

    "This code ensures that the namespace package machinery is operating and that the current package is registered as a namespace package."

    Also, why not

    import pkg_resources pkg_resources.declare_namespace(name)

    What does declare_namespace actually do?

  4. zip_safe. Documentation says

    "If this argument is not supplied, the bdist_egg command will have to analyze all of your project's contents for possible problems each time it buids an egg."

    How? Sounds like it is solving the halting problem, if it manages to find possible problems.

There do exist implementation details that are not included in user documentation, but this is going to be true of any system. These details are sometimes necessarily complex due to distutils limitations, behavioral quirks of deployed packages using distutils, and the sometimes baroque variations in sys.path resolution across platforms, Python versions, and invocation methods.

I trust that there is a good reason for each of them. However, for inclusion in the standard library, some of these should go away:

This magic might do the "right thing" in many cases, and it might indeed help the user that the magic is present, yet I still do believe that magic is inherently evil: explicit is better than implicit. Are documented defaults "implicit" or "magic"?

Documented defaults are explicit.

With respect to you and MAL, I think that some of your judgments regarding setuptools may have perhaps been largely formed at a time last year when, among other things:

* That documentation section I just referenced didn't exist yet

I don't think I ever complained about the lack of documentation. I would (personally) not trust the documentation, anyway, but always look at the code to find out how it actually works. I read documentation primarily to find out what the intent was.

These are significant changes that are directly relevant to the objections that you and he raised (regarding startup time, path length, tools compatibility, etc.), and which I added because of those objections. I think that they are appropriate responses to the issues raised, in that they allow the audience that cares about them (you, MAL, and system packagers in general) to avoid the problems that those features' absence caused.

And I like these changes indeed - my (current) complaints are not about the functionality provided.

Instead, I'm concerned about maintainability of this code.

It would probably helpful if you would both be as specific as possible in your objections so that they can be addressed individually. If you don't want setuptools in 2.5, let's be clear either on the specific objections. If the objection is to the very idea of setuptools, that's fine too, as long as we're clear that's what the objection is.

setuptools combines several independent aspects, as far as I can tell:

  1. automatic download of packages (actually ez_setup). This is a nice feature, and should be included. +1
  2. enhancements to the development process, and improvements to the extensibility of distutils. Also nice, but should be part of distutils itself. +0 in its current form, +1 if it were distutils features.
  3. "package resources". I dislike the term resources (it is not about natural gas, water, main memory, or processor cycles, right?); instead, this seems to provide access to "embedded" data files. Apparently, there is a need for it, so be it. -0
  4. .egg files. -1
  5. entry points. I don't understand that concept
  6. package dependencies. +1
  7. namespace packages. Apparently, there is a need, although I believe that flat is better than nested. -0

However, these are all besides my original objections:

So I would ask that you please make a list of what you would have to see changed in setuptools before you would consider it viable for stdlib inclusion, or simply admit that there are no changes that would satisfy you, or that you don't know enough about it to say, or that you'd like it to be kicked back to distutils-sig for more discussion ad infinitum, or whatever your actual objections are.

See above: now that Anthony approved it, I can accept it. There is nothing you could possibly do to ease my concerns about maintainability, or my dislike of a Python-specific package format.

Meanwhile, this discussion has used up the time that I otherwise would have spent writing 2.5 documentation today (e.g., for the pkgutil additions).

Same here. I spent several hours now typing setuptools email. Still, I don't think this is wasted time (or else I would not have typed this message).

Regards, Martin

More information about the Python-Dev mailing list