[Python-Dev] PEP 413: Faster evolution of the Python Standard Library (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Fri Feb 24 18:24:27 CET 2012

To allow the PEP 407 authors to focus on making the case for doing full CPython releases every 6 months (including language spec updates), I've created PEP 413 as a competing proposal.

It leaves the current language versioning (and rate of change) alone, while adding an additional date based standard library version number and adopting a new development workflow that will allow "standard library" releases to be made alongside each new maintenance release.

PEP text is below, or you can read it online: http://www.python.org/dev/peps/pep-0413/

Cheers, Nick.

PEP: 413 Title: Faster evolution of the Python Standard Library Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Nick Coghlan <ncoghlan at gmail.com> Status: Draft Type: Process Content-Type: text/x-rst Created: 2012-02-24 Post-History: 2012-02-24 Resolution: TBD

Abstract

This PEP proposes the adoption of a new date-based versioning scheme for the standard library (distinct from, but coupled to, the existing language versioning scheme) that allows accelerated releases of the Python standard library, while maintaining (or even slowing down) the current rate of change in the core language definition.

Like PEP 407, it aims to adjust the current balance between measured change that allows the broader community time to adapt and being able to keep pace with external influences that evolve more rapidly than the current release cycle can handle (this problem is particularly notable for standard library elements that relate to web technologies).

However, it's more conservative in its aims than PEP 407, seeking to restrict the increased pace of development to builtin and standard library interfaces, without affecting the rate of change for other elements such as the language syntax and version numbering as well as the CPython binary API and bytecode format.

Rationale

To quote the PEP 407 abstract:

Finding a release cycle for an open-source project is a delicate exercise
in managing mutually contradicting constraints: developer manpower,
availability of release management volunteers, ease of maintenance for
users and third-party packagers, quick availability of new features (and
behavioural changes), availability of bug fixes without pulling in new
features or behavioural changes.

The current release cycle errs on the conservative side. It is adequate
for people who value stability over reactivity. This PEP is an attempt to
keep the stability that has become a Python trademark, while offering a
more fluid release of features, by introducing the notion of long-term
support versions.

I agree with the PEP 407 authors that the current release cycle of the standard library is too slow to effectively cope with the pace of change in some key programming areas (specifically, web protocols and related technologies, including databases, templating and serialisation formats).

However, I have written this competing PEP because I believe that the approach proposed in PEP 407 of offering full, potentially binary incompatible releases of CPython every 6 months places too great a burden on the wider Python ecosystem.

Under the current CPython release cycle, distributors of key binary extensions will often support Python releases even after the CPython branches enter "security fix only" mode (for example, Twisted currently ships binaries for 2.5, 2.6 and 2.7, NumPy and SciPy suport those 3 along with 3.1 and 3.2, PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit binaries for 2.6 and 2.7, etc).

If CPython were to triple (or more) its rate of releases, the developers of those libraries (many of which are even more resource starved than CPython) would face an unpalatable choice: either adopt the faster release cycle themselves (up to 18 simultaneous binary releases for PyGame!), drop older Python versions more quickly, or else tell their users to stick to the CPython LTS releases (thus defeating the entire point of speeding up the CPython release cycle in the first place).

Similarly, many support tools for Python (e.g. syntax highlighters) can take quite some time to catch up with language level changes.

At a cultural level, the Python community is also accustomed to a certain meaning for Python version numbers - they're linked to deprecation periods, support periods, all sorts of things. PEP 407 proposes that collective knowledge all be swept aside, without offering a compelling rationale for why such a course of action is actually necessary (aside from, perhaps, making the lives of the CPython core developers a little easier at the expense of everyone else).

But, if we go back to the primary rationale for increasing the pace of change (i.e. more timely support for web protocols and related technologies), we can note that those only require standard library changes. That means many (perhaps even most) of the negative effects on the wider community can be avoided by explicitly limiting which parts of CPython are affected by the new release cycle, and allowing other parts to evolve at their current, more sedate, pace.

Proposal

This PEP proposes the addition of a new sys.stdlib_info attribute that records a date based standard library version above and beyond the underlying interpreter version::

sys.stdlib_info(year=2012, month=8, micro=0, releaselevel='final', serial=0)

This information would also be included in the sys.version string::

Python 3.3.0 (12.08.0, default:c1a07c8092f7+, Feb 17 2012, 23:03:41)
[GCC 4.6.1]

When maintenance releases are created, two new versions of Python would actually be published on python.org (using the first 3.3 maintenance release, planned for February 2013 as an example)::

3.3.1 + 12.08.1  # Maintenance release
3.3.1 + 13.02.0  # Standard library release

A standard library release would just be the corresponding maintenance release, with the following additional, backwards compatible changes:

A further 6 months later, the next 3.3 maintenance release would again be accompanied by a new standard library release::

3.3.2 + 12.08.2  # Maintenance release
3.3.2 + 13.08.1  # Standard library release

Again, the standard library release would be binary compatible with the previous language release, merely offering additional features at the Python level.

Finally, 18 months after the release of 3.3, a new language release would be made around the same time as the final 3.3 maintenance release:

3.3.3 + 12.08.3  # Maintenance release
3.4.0 + 14.02.0  # Language release

Language releases would then contain all the features that are not permitted in standard library releases:

The 3.4 release cycle would then follow a similar pattern to that for 3.3::

3.4.1 + 14.02.1  # Maintenance release
3.4.1 + 14.08.0  # Standard library release
3.4.2 + 14.02.2  # Maintenance release
3.4.2 + 15.02.0  # Standard library release
3.4.3 + 14.02.3  # Maintenance release
3.5.0 + 15.08.0  # Language release

Effects

Effect on development cycle

Similar to PEP 407, this PEP will break up the delivery of new features into more discrete chunks. Instead of whole raft of changes landing all at once in a language release, each language release will be limited to 6 months worth of standard library changes, as well as any changes associated with new syntax.

Effect on workflow

This PEP proposes the creation of a single additional branch for use in the normal workflow. After the release of 3.3, the following branches would be in use::

2.7 # Maintenance branch, no change 3.3 # Maintenance branch, as for 3.2 3.3-compat # New branch, backwards compatible changes default # Language changes, standard library updates that depend on them

When working on a new feature, developers will need to decide whether or not it is an acceptable change for a standard library release. If so, then it should be checked in on 3.3-compat and then merged to default. Otherwise it should be checked in directly to default.

Effect on bugfix cycle

The effect on the bug fix cycle is essentially the same as that on the workflow for new features - there is one additional branch to pass through before the change reaches default branch.

Effect on the community

PEP 407 has this to say about the effects on the community:

People who value stability can just synchronize on the LTS releases which,
with the proposed figures, would give a similar support cycle (both in
duration and in stability).

I believe this statement is just plain wrong. Life isn't that simple. Instead, developers of third party modules and frameworks will come under pressure to support the full pace of the new release cycle with binary updates, teachers and book authors will receive complaints that they're only covering an "old" version of Python ("You're only using 3.3, the latest is 3.5!"), etc.

As the minor version number starts climbing 3 times faster than it has in the past, I believe perceptions of language stability would also fall (whether such opinions were justified or not).

I believe isolating the increased pace of change to the standard library, and clearly delineating it with a separate date-based version number will greatly reassure the rest of the community that no, we're not suddenly asking them to triple their own rate of development. Instead, we're merely going to ship standard library updates for the next language release in three 6-monthly installments rather than delaying them all, even those that are backwards compatible with the previously released version of Python.

The community benefits list in PEP 407 are equally applicable to this PEP, at least as far as the standard library is concerned:

People who value reactivity and access to new features (without taking the
risk to install alpha versions or Mercurial snapshots) would get much more
value from the new release cycle than currently.

People who want to contribute new features or improvements would be more
motivated to do so, knowing that their contributions will be more quickly
available to normal users.

If the faster release cycle encourages more people to focus on contributing to the standard library rather than proposing changes to the language definition, I don't see that as a bad thing.

Handling News Updates

What's New?

The "What's New" documents would be split out into separate documents for standard library releases and language releases. If the major version number only continues to increase once every decade or so, resolving the eventual numbering conflict can be safely deemed somebody elses problem :)

NEWS

Merge conflicts on the NEWS file is already a hassle. Since this PEP proposes introduction of an additional branch into the normal workflow, resolving this becomes even more critical. While Mercurial phases will help to some degree, it would be good to eliminate the problem entirely.

One suggestion from Barry Warsaw is to adopt a non-conflicting separate-files-per-change approach, similar to that used by Twisted [2_].

For this PEP, one possible layout for such an approach (adopted following the release of 3.3.0+12.8.0 using the existing NEWS process) might look like::

Misc/ lang_news/ 3.3.1/ 3.4.0/ stdlib_news/ 12.08.1/ builtins/ extensions/ library/ documentation/ tests/ 13.02.0/ builtins/ extensions/ library/ documentation/ tests/ NEWS # Now autogenerated from lang_news and stdlib_news

Putting the version information in the directory heirarchy isn't strictly necessary (since the NEWS file generator could figure out from the version history), but does make it easy for humans to keep the different versions in order.

Why isn't PEP 384 enough?

PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited subset of the full C ABI that is guaranteed to remain stable. Extensions built against the stable ABI should be able to support all subsequent Python versions with the same binary.

This will help new projects to avoid coupling their C extension modules too closely to a specific version of CPython. For existing modules, however, migrating to the stable ABI can involve quite a lot of work (especially for extension modules that define a lot of classes). With limited development resources available, any time spent on such a change is time that could otherwise have been spent working on features that are offer more direct benefits to end users.

Why not separate out the standard library entirely?

Because it's a lot of work for next to no pay-off. CPython without the standard library is useless (the build chain won't even finish). You can't create a standalone pure Python standard library, because too many "modules" are actually tightly linked in to the internal details of the respective interpreters (e.g. weakref, gc, sys).

Creating a separate development branch that is kept compatible with the previous feature release should provide most of the benefits of a separate standard library repository with only a fraction of the pain.

Acknowledgements

Thanks go to the PEP 407 authors for starting this discussion, as well as to those authors and Larry Hastings for initial discussions of the proposal made in this PEP.

References

.. [1] PEP 407: New release cycle and introducing long-term support versions http://www.python.org/dev/peps/pep-0407/

.. [2] Twisted's "topfiles" approach to NEWS generation http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles

Copyright

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list