[Python-Dev] Status of packaging in 3.3 (original) (raw)

Paul Moore p.f.moore at gmail.com
Wed Jun 20 13:11:03 CEST 2012

On 20 June 2012 10:12, Antoine Pitrou <solipsis at pitrou.net> wrote:

I think the whole idea that distutils should be frozen and improvements should only go in distutils2 has been misled. Had distutils been improved instead, many of those enhancements would already have been available in 3.2 (and others would soon be released in 3.3).

The problem seems to be that in the past, any changes in distutils have been met with cries of "you can't do that", basically because the lack of a clearly distinguished extension API means that the assumption is that for any bit of the internals, someone somewhere has monkeypatched or relied on it. The issue is compounded by the fact that a lot of distutils is undocumented, or at least badly documented, so saying "if it's not documented, it's internal" doesn't work :-(

Maybe if we could be a bit more aggressive in saying what counts as "internal" and resist the cries of "but I use it", modifying distutils might be a more viable option, But there doesn't seem to be anyone willing to take and defend that step. IIRC, Tarek proposed distutils2/packaging after getting frustrated with how little he could usefully do on distutils itself.

Deciding to remove packaging from 3.3 is another instance of the same mistake, IMO.

I see your point, but without sufficient developer resource, the question is whether packaging is in a usable state at all. Nobody but Éric is actually working on packaging (and as he says, even he is not able to at the moment), so what alternatives are there? I guess one extra option not mentioned by Éric is to make the packaging issues into release blocker bugs. That would effectively stall the release of 3.3 until packaging could be brought to an acceptable state, effectively a form of blackmail. I can't imagine anyone wants to take that approach. And yet, some of the existing bugs would clearly be release blockers if they were in any other part of Python.

I think the first question is, do we need an enhanced distutils in the stdlib? As far as I can see, this one has been answered strongly, in the affirmative, a few times now. And yet, the need seems to be a diffuse thing, with no real champion (Tarek and Éric have both tried to take that role, and both appear to have been unable to invest the necessary amount of time - which doesn't surprise me, it seems to be a massive task).

Removing packaging from 3.3, in my mind acknowledges that as it stands the approach was a failed experiment[1]. Better to get it taken out before it appears in a released version of Python. We need to rethink the approach. I see a number of options going forward, all of which are based round the need to ensure enough developer involvement, so that Tarek and Éric get help, and don't simply burn out before we have anything useful.

  1. Reconsider the decision to freeze distutils, with a view to migrating incrementally to the feature set we want from packaging. That'll be hard as we'd need to take a much stronger line on making changes that could break existing code (stronger in the sense of "so we broke your code, tough - you were using undocumented/internal APIs"). And I suspect Tarek wouldn't be willing to take this route, so we immediately lose one resource. Maybe the other core developers could take up the slack, though. For example, Antoine, you seem to be implying that you would have worked on distutils if this had happened.

  2. Free up distutils2 to develop as an external package, and then have a PEP proposing its inclusion in the stdlib in due course, when it is ready and has been proven in the wild. The benefit here is that I assume that as a separate project, becoming a committer would be easier than becoming a Python core developer, so there's a wider pool of developers. The downside is that the timescales would be a lot longer (I doubt we'd see anything in 3.4 this way, and maybe not even 3.5).

  3. Write a PEP describing precisely what the packaging module will do, get consensus/agreement, and then restart development based on a solid scope and spec. This is the correct route for getting something added direct to the stdlib, and it's a shame it didn't happen in the first place for packaging. Having said that, the PEP would likely be huge, given the scope of packaging, and would require a lot of time from a champion. There's no guarantee that championing a PEP wouldn't burn someone out just as rapidly as developing the module itself :-( And also, given that the packaging documentation is one of its weak areas, I'd have to say I have concerns as to whether a PEP would come together in any case... The assumption here, though, is that the PEP process creates the debate, and results in interested parties coming together in the discussion. If we can keep that momentum, we get a pool of interested developers who may well assist in the coding aspects.

The one option I don't like is taking packaging out, releasing 3.3, and then putting it straight back in as is, and simply carry on as now, in the hope that it'll be ready for 3.4. I honestly doubt that the only issue is that we've run out of time before 3.3. There are more fundamental problems that need to be addressed as well - specifically the reliance on one individual to bear all of the load.

Just my thoughts, Paul.

[1] That reads really harshly. I don't mean to criticise any of the work that's been done, I'm a huge fan of the idea of packaging, and its goals. The "experiment" in this case is around process - developing something as big and important as packaging with limited developer resource, relatively directly in the core (bringing it in from distutils2 sooner rather than later) and working from a series of smaller PEPs focused on particular details, rather than an overall one covering the whole package.

More information about the Python-Dev mailing list