[Python-Dev] Re: Stability and change (original) (raw)

Alex Martelli aleax@aleax.it
Mon, 8 Apr 2002 10:20:04 +0200

On Monday 08 April 2002 00:08, Guido van Rossum wrote:

Alex, please try brevity. :-)

> It would help to some degree if the "stable" releases could be shown > to have SOME kind of enhancement. Not as much as if language > changes which did not break existing correct code could also be > there, but some. Modules such as email are a big part of the draw > of 2.2, for example (not quite as big as language-level enhancements, > but big). Now there's a good proposal. But it costs time and effort that takes away from our main mission (and PythonLabs is already under a lot of pressure).

You know PythonLabs's missions and goals better than I do. If such goals weigh "making a new and better Python" far more highly than helping spread and preserve the use of Python, then you may well be right that the extra effort for (e.g.) making email part of the standard distribution vs the current "maintaining a separate distro" would be a bad allocation of scarce resources.

I wonder if there's a commercial market for this? Maybe ActiveState could back-port important packages or language changes and insert them into ActivePython?

If I understand correctly, ActiveState has backed off (because of not sufficient revenue returns) from "owning" Hammond's win32all extensions, which are truly indispensable to firms using Python for Windows-only development. If win32all did not "have a commercial market" (sufficient to justify ActiveState's investments), then I don't see what other packages could.

[proposal to semi-backport bool snipped]

> Thus it would help us send the message I think prospective users > want to hear -- that Python stands for stability AND change, that > there is not necessarily a contradiction between the two goals. In > a very minor way, to be sure. Given the requirements you give elsewhere in your message, I'm not sure how to satisfy all these constraints. Let's be concrete. Should we backport stuff to 2.1 (the earliest release I'm willing to consider)? If so, what? Look at the "What's New?" section in http://www.python.org/2.2/. The new Windows installer has already been backported to 2.1. Type-class unification is obviously out. So are interators and generators (too much effort). So is // division (I think). Returning longs instead of raising OverflowError on int operations would probably break too much code, so it's better not to

Something requiring much effort is not necessarily "obviously out": it depends on how important you judge the results of those effort versus the results you could have by spending the effort elsewhere. In terms of "how it looks to actual and prospective users of Python" rather than "how much effort would it be to implement", iterators (not generators) and // division look good -- enabling "for x in dict" and "if x in dict" are two features of instant appeal -- language enhancements that do not break previously correct code.

It's of course possible to construct a program that would break due to any such change, because it expects and relies upon a specific exception being raised when such usage is attempted -- when the usage becomes legal and meaningful, crack. But in most cases that would be a "sea-lawyer" level issue. You're probably right that, alas, overflow on int operations wouldn't be feasible to remove (I don't know). Syntax changes that make some source code legal where it didn't use to be, e.g. // , are probably at the other extreme -- in practice one would not rely on e.g. compile('a//b') throwing a SyntaxException and thus break if it doesn't.

To "be concrete" as you ask, I think that targeting 2.1 as the stable release would not be the best allocation of effort, even though I see others disagree. Basically, the argument against that would be that 2.2 was announced as the new stable release, not as a bleeding edge experimental one, and making it "retroactively experimental" would send exactly the wrong message. So, I think effort would be better spent ensuring that enhancements for 2.3 are well divided between [a] those which are safe to have in some 2.2.* as well, vs [b] those which ARE experimental (break previously correct code) and thus should only go into 2.3.

backport. That leaves the email package and xmlrpclib (both available as separate distros!), and large file support -- which I believe did get ported to 2.1.2.

The fact that email and xmlrpclib are already installable as separate distros on 2.1.* suggests the amount of effort to put them into the standard library of 2.1.something might be modest; therefore, even if the advantage is also modest, it might still be worth doing.

> More relevantly, I'm not sure I had managed to explain exactly why I > think "stability and change" IS important to Python adoption in a > certain category of software shops.

Unfortunately, it's not clear what adaptation in that (apparently very change-resistant) category buys Python. IOW why should we, mostly volunteers working for the community, care?

Assuming one cares about Python being widely used, "capturing" a new "customer" that is highly change-resistant has better returns than getting one who's very change-prone. Exactly because of that user's inertia, they'll stick with Python once they've decided for it -- some other shop who'll gladly have a carefree fling with Python is just as liable to leave it for Ruby, or whatever, tomorrow.

Apparently we DO want to promote the spread of Python (otherwise the purpose of the python marketing list and the PSF becomes murky to me). We believe the world will be a better place if more development uses Python rather than different languages. That goal is well served by helping Python acceptance by development shops that do care about stability, because such shops are a substantial fraction of our industry.

Over the decades, the people who are now the technical managers and senior developers in such shops have seen too much of their work laid waste by incompatible changes in platforms, and have thus become quite sensitive to that specific technological risk; they've seen some other fraction of their efforts similarly wasted by having been spent on platforms who became "really stable" (i.e., dead) and are thus also sensitized to that antisymmetric risk. These are survival traits in the software jungle. Our industry suffers from so many "bandwagons" (in platforms, methodologies, tools, etc) that learning when NOT to jump on one -- developing a nose for the symptoms of the two opposed risks, excessive/churning rate of change and the "sunset boulevard" syndrome of a slowly dying technology -- make a shop's survival more likely. So, I think we need to avoid triggering either reflex (in sensible cases: we can't really make Logajan happy, I think).

That is, IF we want to promote Python's spread -- if we do believe we'll be happier if Python is widely accepted rather than less used than it deserves to be. I'm reasonably sure of that, myself. The more Python is widespread, the more of a "Python market" commercial ventures perceive as existing, the better my chances to find a good Python port to some weird new platform, the better some other enthusiast's chances to find jobs based on Python (or mine, should my current arrangements fail or terminate), the better tools I'm likely to have my pick amongst, and so forth. I didn't think this needed to be re-examined on this thread, but I'm quite game if it does.

> What numbering or naming scheme we adopt has importance in > determining what we communicate, but before that, deciding what we > DO want to communicate is probably more important.

Indeed. But I'd love it if all we need to do is change our communication patterns, because that takes much less effort than fully maintaining two separate release tracks.

I understand that doing nothing except changing communication patterns entails less effort than other changes. Exactly because of that, the other changes send a stronger message: they bespeak real commitment to "stability AND change" because they don't come for free. Economy Nobel prizes last year all went to economists whose major work has been on this subject: asymmetric-information situations and message-sending.

For example, that's how advertising, sponsoring sports, etc, mostly work: a firm sends a message "we really believe in this product" by splurging a lot of money in promoting it, because it's substantial effort -- effort it would make no sense for them to spend if they (who have more info on the product than the prospective buyer) really knew it was a dud. I'm summarizing and oversimplifying, but that's the gist.

> I think such a "retroactive" rebranding of 2.2 as "bleeding-edge" would > be a serious blow to those of us who have accepted it as stable > enough as our base over the last 3 months -- in books, magazine > articles, advocacy within our firms and outside of them, etc. For > example, AB Strakt has decided 2.2 was stable enough for us (and > it was being presented as such) -- our new code now uses > "for x in somedict:", "if x in somedict:", etc, for example -- now > hearing that it wasn't, and 2.2.X micro-releases might introduce > backwards-incompatible changes, would be a real problem indeed.

I'm happy to hear that 2.2 is stable for you -- exactly as I intended and expected. But what confuses me is that while you're happy with 2.2's stability, you don't want to testify of this fact to the more conservative shops. Rather than telling them "honestly, I cannot recommend 2.2", why don't you tell them "I've used 2.2 for three months and it's as solid as a rock"?

I do testify that all new code I've written or ported to 2.2 has run without a hitch so far. That's quite different from saying that existing code written for 2.1 ran unchanged with 2.2, because it didn't -- nor do I think it reasonably COULD have, mind you. E.g., any time I introspected an instance via dir() I only used to get per-instance attributes -- now I get attributes it gets from its class too; it's an enhancement, but it does fool any such introspection that I had.

Unit-tests might help, but unfortunately I relied on doctest for those, so I get a zillion "breakage" reports that aren't all that helpful. Guess I misused doctest (but it's SO much handier than unittest that it's an "attractive nuisance", consarn it!-).

Once you do manage to identify the real breakage points, it's not (generally) all that hard to turn 2.1 code into 2.2 code. People who maintain both 2.1 and 2.2 versions of their packages are no doubt better placed to evaluate that. But the message that "only little broke, and fixing it was not too terrible" is not strong enough for the shop looking for stability -- which I guess help explains 2.1's relative longevity, no?

> I perceive Booleans as small (albeit negatively), but both our CTO and > principal investor have vehemently posted against them, so it would > be hard to 'sell' them even if I were quite favorable.

If I didn't suspect that you, the CTI and the P.I. together are 75% of the company, I would say that they are micromanaging you. :-)

No: there's 12 of us, but we also have a strong commitment to no "code ownership". We need to agree on coding standards that are strong enough that we can't tell who wrote what piece -- at least, that's the ideal (and part of XP). And the standards must get consensus, which is no picnic (but the alternatives are worse).

> Still, if that was the one hurdle to jump to get to a "stable and > changing" release, who knows. But if jumping that small hurdle only > gets to more of the same, either unstability or feature-frozen > versions, the carrot may be too small.

I suspect that the people who complain that Python's rate of change is too fast are misguided. There's a lot of fuss on the newsgroup about potentially breaking code and undesired changes, and before you know it, everybody just believes that Python is unstable and changes too fast. But if you try it, you find very little incompatibility, and what's there is easily dealt with. The people on the newsgroup who complain that the introduction of bool is such a major change that it would only be acceptable in a major release (i.e. 3.0) vastly overestimate its impact, which is minuscule (zero in most cases).

The impact is small (that's my evaluation) in terms of maintenance of existing code (not necessarily all that small in teaching the language or writing about it, but that's another issue). Total breakage of doctests is probably the worst part (maybe it's just what we needed to stop using doctests and go for other approaches instead). Still, "zero in most cases" is NOT the situation with doctest usage today: I do believe I was not the only one to fall into its attractive-nuisance trap.

Logajan's main gripe (and also McMillan's biggest effort in keeping his codebase cross-version portable) was about things that changed between 1.5.2 and 2.0 -- that was a major release. And at the time we didn't have PEPs 4-6, there was no warning module, and so on.

So maybe there's only (or mostly) a problem of perception?

Mostly, yes, and I do agree that Logajan's impossible to satisfy -- I see no amount of effort that could have kept him happy, given his expressed complaints.

But if you're implying that said perception (by prospective users) is unfounded, I disagree. It has a solid basis in reality. Some previously correct code does break at each minor release (doctests first -- as apparently is their "canary in a coal mine" intended role -- but that is quite visible and contributes highly to perception). To some extent ("sea-lawyering" level) that may be inevitable. Moreover, the "status" perceived for releases such as 2.1.2, etc, seems correct to me: you do NOT want to invest substantial effort on them -- isn't this what you're saying throughout this? So, the perception "not quite dead but close, BDFL doesn't care much at all about them" is correct, no?

It may be possible to change perception without changing reality, but I have some doubt. "You can fool some of the people all of the time", etc, etc:-).

Alex