What’s New in Python 2.7 — Python v2.7 documentation (original) (raw)

Author: A.M. Kuchling (amk at amk.ca)
Release: 2.7
Date: July 03, 2010

This article explains the new features in Python 2.7. Python 2.7 was released on July 7, 2010.

Numeric handling has been improved in many ways, for both floating-point numbers and for the Decimal class. There are some useful additions to the standard library, such as a greatly enhanced unittest module, the argparse module for parsing command-line options, convenient OrderedDictand Counter classes in the collections module, and many other improvements.

Python 2.7 is planned to be the last of the 2.x releases, so we worked on making it a good release for the long term. To help with porting to Python 3, several new features from the Python 3.x series have been included in 2.7.

This article doesn’t attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.7 athttp://docs.python.org. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature or the issue on http://bugs.python.org in which a change was discussed. Whenever possible, “What’s New in Python” links to the bug/patch item for each change.

The Future for Python 2.x

Python 2.7 is intended to be the last major release in the 2.x series. The Python maintainers are planning to focus their future efforts on the Python 3.x series.

This means that 2.7 will remain in place for a long time, running production systems that have not been ported to Python 3.x. Two consequences of the long-term significance of 2.7 are:

Python 3.1 Features

Much as Python 2.6 incorporated features from Python 3.0, version 2.7 incorporates some of the new features in Python 3.1. The 2.x series continues to provide tools for migrating to the 3.x series.

A partial list of 3.1 features that were backported to 2.7:

Other new Python3-mode warnings include:

PEP 372: Adding an Ordered Dictionary to collections

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, 2.7 introduces a newOrderedDict class in the collections module.

The OrderedDict API provides the same interface as regular dictionaries but iterates over keys and values in a guaranteed order depending on when a key was first inserted:

from collections import OrderedDict d = OrderedDict([('first', 1), ... ('second', 2), ... ('third', 3)]) d.items() [('first', 1), ('second', 2), ('third', 3)]

If a new entry overwrites an existing entry, the original insertion position is left unchanged:

d['second'] = 4 d.items() [('first', 1), ('second', 4), ('third', 3)]

Deleting an entry and reinserting it will move it to the end:

del d['second'] d['second'] = 5 d.items() [('first', 1), ('third', 3), ('second', 5)]

The popitem() method has an optional _last_argument that defaults to True. If last is True, the most recently added key is returned and removed; if it’s False, the oldest key is selected:

od = OrderedDict([(x,0) for x in range(20)]) od.popitem() (19, 0) od.popitem() (18, 0) od.popitem(last=False) (0, 0) od.popitem(last=False) (1, 0)

Comparing two ordered dictionaries checks both the keys and values, and requires that the insertion order was the same:

od1 = OrderedDict([('first', 1), ... ('second', 2), ... ('third', 3)]) od2 = OrderedDict([('third', 3), ... ('first', 1), ... ('second', 2)]) od1 == od2 False

Move 'third' key to the end

del od2['third']; od2['third'] = 3 od1 == od2 True

Comparing an OrderedDict with a regular dictionary ignores the insertion order and just compares the keys and values.

How does the OrderedDict work? It maintains a doubly-linked list of keys, appending new keys to the list as they’re inserted. A secondary dictionary maps keys to their corresponding list node, so deletion doesn’t have to traverse the entire linked list and therefore remains O(1).

The standard library now supports use of ordered dictionaries in several modules.

See also

PEP 372 - Adding an ordered dictionary to collections

PEP written by Armin Ronacher and Raymond Hettinger; implemented by Raymond Hettinger.

PEP 378: Format Specifier for Thousands Separator

To make program output more readable, it can be useful to add separators to large numbers, rendering them as 18,446,744,073,709,551,616 instead of 18446744073709551616.

The fully general solution for doing this is the locale module, which can use different separators (“,” in North America, “.” in Europe) and different grouping sizes, but locale is complicated to use and unsuitable for multi-threaded applications where different threads are producing output for different locales.

Therefore, a simple comma-grouping mechanism has been added to the mini-language used by the str.format() method. When formatting a floating-point number, simply include a comma between the width and the precision:

'{:20,.2f}'.format(18446744073709551616.0) '18,446,744,073,709,551,616.00'

When formatting an integer, include the comma after the width:

'{:20,d}'.format(18446744073709551616) '18,446,744,073,709,551,616'

This mechanism is not adaptable at all; commas are always used as the separator and the grouping is always into three-digit groups. The comma-formatting mechanism isn’t as general as the localemodule, but it’s easier to use.

See also

PEP 378 - Format Specifier for Thousands Separator

PEP written by Raymond Hettinger; implemented by Eric Smith.

PEP 389: The argparse Module for Parsing Command Lines

The argparse module for parsing command-line arguments was added as a more powerful replacement for theoptparse module.

This means Python now supports three different modules for parsing command-line arguments: getopt, optparse, andargparse. The getopt module closely resembles the C library’s getopt() function, so it remains useful if you’re writing a Python prototype that will eventually be rewritten in C.optparse becomes redundant, but there are no plans to remove it because there are many scripts still using it, and there’s no automated way to update these scripts. (Making the argparseAPI consistent with optparse‘s interface was discussed but rejected as too messy and difficult.)

In short, if you’re writing a new script and don’t need to worry about compatibility with earlier versions of Python, useargparse instead of optparse.

Here’s an example:

import argparse

parser = argparse.ArgumentParser(description='Command-line example.')

Add optional switches

parser.add_argument('-v', action='store_true', dest='is_verbose', help='produce verbose output') parser.add_argument('-o', action='store', dest='output', metavar='FILE', help='direct output to FILE instead of stdout') parser.add_argument('-C', action='store', type=int, dest='context', metavar='NUM', default=0, help='display NUM lines of added context')

Allow any number of additional arguments.

parser.add_argument(nargs='*', action='store', dest='inputs', help='input filenames (default is stdin)')

args = parser.parse_args() print args.dict

Unless you override it, -h and --help switches are automatically added, and produce neatly formatted output:

-> ./python.exe argparse-example.py --help usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]

Command-line example.

positional arguments: inputs input filenames (default is stdin)

optional arguments: -h, --help show this help message and exit -v produce verbose output -o FILE direct output to FILE instead of stdout -C NUM display NUM lines of added context

As with optparse, the command-line switches and arguments are returned as an object with attributes named by the dest parameters:

-> ./python.exe argparse-example.py -v {'output': None, 'is_verbose': True, 'context': 0, 'inputs': []}

-> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2 {'output': '/tmp/output', 'is_verbose': True, 'context': 4, 'inputs': ['file1', 'file2']}

argparse has much fancier validation than optparse; you can specify an exact number of arguments as an integer, 0 or more arguments by passing '*', 1 or more by passing '+', or an optional argument with '?'. A top-level parser can contain sub-parsers to define subcommands that have different sets of switches, as in svn commit, svn checkout, etc. You can specify an argument’s type as FileType, which will automatically open files for you and understands that '-' means standard input or output.

See also

argparse documentation

The documentation page of the argparse module.

Upgrading optparse code

Part of the Python documentation, describing how to convert code that uses optparse.

PEP 389 - argparse - New Command Line Parsing Module

PEP written and implemented by Steven Bethard.

PEP 391: Dictionary-Based Configuration For Logging

The logging module is very flexible; applications can define a tree of logging subsystems, and each logger in this tree can filter out certain messages, format them differently, and direct messages to a varying number of handlers.

All this flexibility can require a lot of configuration. You can write Python statements to create objects and set their properties, but a complex set-up requires verbose but boring code.logging also supports a fileConfig()function that parses a file, but the file format doesn’t support configuring filters, and it’s messier to generate programmatically.

Python 2.7 adds a dictConfig() function that uses a dictionary to configure logging. There are many ways to produce a dictionary from different sources: construct one with code; parse a file containing JSON; or use a YAML parsing library if one is installed. For more information see Configuration functions.

The following example configures two loggers, the root logger and a logger named “network”. Messages sent to the root logger will be sent to the system log using the syslog protocol, and messages to the “network” logger will be written to a network.log file that will be rotated once the log reaches 1MB.

import logging import logging.config

configdict = { 'version': 1, # Configuration schema in use; must be 1 for now 'formatters': { 'standard': { 'format': ('%(asctime)s %(name)-15s ' '%(levelname)-8s %(message)s')}},

'handlers': {'netlog': {'backupCount': 10, 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/logs/network.log', 'formatter': 'standard', 'level': 'INFO', 'maxBytes': 1000000}, 'syslog': {'class': 'logging.handlers.SysLogHandler', 'formatter': 'standard', 'level': 'ERROR'}},

Specify all the subordinate loggers

'loggers': { 'network': { 'handlers': ['netlog'] } },

Specify properties of the root logger

'root': { 'handlers': ['syslog'] }, }

Set up configuration

logging.config.dictConfig(configdict)

As an example, log two error messages

logger = logging.getLogger('/') logger.error('Database not found')

netlogger = logging.getLogger('network') netlogger.error('Connection failed')

Three smaller enhancements to the logging module, all implemented by Vinay Sajip, are:

See also

PEP 391 - Dictionary-Based Configuration For Logging

PEP written and implemented by Vinay Sajip.

PEP 3106: Dictionary Views

The dictionary methods keys(), values(), anditems() are different in Python 3.x. They return an object called a view instead of a fully materialized list.

It’s not possible to change the return values of keys(),values(), and items() in Python 2.7 because too much code would break. Instead the 3.x versions were added under the new names viewkeys(), viewvalues(), and viewitems().

d = dict((i*10, chr(65+i)) for i in range(26)) d {0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'} d.viewkeys() dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])

Views can be iterated over, but the key and item views also behave like sets. The & operator performs intersection, and |performs a union:

d1 = dict((i10, chr(65+i)) for i in range(26)) d2 = dict((i*.5, i) for i in range(1000)) d1.viewkeys() & d2.viewkeys() set([0.0, 10.0, 20.0, 30.0]) d1.viewkeys() | range(0, 30) set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])

The view keeps track of the dictionary and its contents change as the dictionary is modified:

vk = d.viewkeys() vk dict_keys([0, 130, 10, ..., 250]) d[260] = '&' vk dict_keys([0, 130, 260, 10, ..., 250])

However, note that you can’t add or remove keys while you’re iterating over the view:

for k in vk: ... d[k*2] = k ... Traceback (most recent call last): File "", line 1, in RuntimeError: dictionary changed size during iteration

You can use the view methods in Python 2.x code, and the 2to3 converter will change them to the standard keys(),values(), and items() methods.

See also

PEP 3106 - Revamping dict.keys(), .values() and .items()

PEP written by Guido van Rossum. Backported to 2.7 by Alexandre Vassalotti; issue 1967.

PEP 3137: The memoryview Object

The memoryview object provides a view of another object’s memory content that matches the bytes type’s interface.

import string m = memoryview(string.letters) m <memory at 0x37f850> len(m) # Returns length of underlying object 52 m[0], m[25], m[26] # Indexing returns one byte ('a', 'z', 'A') m2 = m[0:26] # Slicing returns another memoryview m2 <memory at 0x37f080>

The content of the view can be converted to a string of bytes or a list of integers:

m2.tobytes() 'abcdefghijklmnopqrstuvwxyz' m2.tolist() [97, 98, 99, 100, 101, 102, 103, ... 121, 122]

memoryview objects allow modifying the underlying object if it’s a mutable object.

m2[0] = 75 Traceback (most recent call last): File "", line 1, in TypeError: cannot modify read-only memory b = bytearray(string.letters) # Creating a mutable object b bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ') mb = memoryview(b) mb[0] = '*' # Assign to view, changing the bytearray. b[0:5] # The bytearray has been changed. bytearray(b'*bcde')

See also

PEP 3137 - Immutable Bytes and Mutable Buffer

PEP written by Guido van Rossum. Implemented by Travis Oliphant, Antoine Pitrou and others. Backported to 2.7 by Antoine Pitrou; issue 2396.

Other Language Changes

Some smaller changes made to the core Python language are:

The contextlib.nested() function provides a very similar function, so it’s no longer necessary and has been deprecated.
(Proposed in http://codereview.appspot.com/53094; implemented by Georg Brandl.)

Interpreter Changes

A new environment variable, PYTHONWARNINGS, allows controlling warnings. It should be set to a string containing warning settings, equivalent to those used with the -W switch, separated by commas. (Contributed by Brian Curtin; issue 7301.)

For example, the following setting will print warnings every time they occur, but turn warnings from the Cookie module into an error. (The exact syntax for setting an environment variable varies across operating systems and shells.)

export PYTHONWARNINGS=all,error:::Cookie:0

Optimizations

Several performance enhancements have been added:

New and Improved Modules

As in every release, Python’s standard library received a number of enhancements and bug fixes. Here’s a partial list of the most notable changes, sorted alphabetically by module name. Consult theMisc/NEWS file in the source tree for a more complete list of changes, or look through the Subversion logs for all the details.

New module: importlib

Python 3.1 includes the importlib package, a re-implementation of the logic underlying Python’s import statement.importlib is useful for implementors of Python interpreters and to users who wish to write new importers that can participate in the import process. Python 2.7 doesn’t contain the completeimportlib package, but instead has a tiny subset that contains a single function, import_module().

import_module(name, package=None) imports a module. name is a string containing the module or package’s name. It’s possible to do relative imports by providing a string that begins with a .character, such as ..utils.errors. For relative imports, the_package_ argument must be provided and is the name of the package that will be used as the anchor for the relative import. import_module() both inserts the imported module into sys.modules and returns the module object.

Here are some examples:

from importlib import import_module anydbm = import_module('anydbm') # Standard absolute import anydbm <module 'anydbm' from '/p/python/Lib/anydbm.py'>

Relative import

file_util = import_module('..file_util', 'distutils.command') file_util <module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'>

importlib was implemented by Brett Cannon and introduced in Python 3.1.

New module: sysconfig

The sysconfig module has been pulled out of the Distutils package, becoming a new top-level module in its own right.sysconfig provides functions for getting information about Python’s build process: compiler switches, installation paths, the platform name, and whether Python is running from its source directory.

Some of the functions in the module are:

Consult the sysconfig documentation for more details and for a complete list of functions.

The Distutils package and sysconfig are now maintained by Tarek Ziadé, who has also started a Distutils2 package (source repository athttp://hg.python.org/distutils2/) for developing a next-generation version of Distutils.

Updated module: unittest

The unittest module was greatly enhanced; many new features were added. Most of these features were implemented by Michael Foord, unless otherwise noted. The enhanced version of the module is downloadable separately for use with Python versions 2.4 to 2.6, packaged as the unittest2 package, fromhttp://pypi.python.org/pypi/unittest2.

When used from the command line, the module can automatically discover tests. It’s not as fancy as py.test ornose, but provides a simple way to run tests kept within a set of package directories. For example, the following command will search the test/ subdirectory for any importable test files named test*.py:

python -m unittest discover -s test

Consult the unittest module documentation for more details. (Developed in issue 6001.)

The main() function supports some other new options:

The progress messages now show ‘x’ for expected failures and ‘u’ for unexpected successes when run in verbose mode. (Contributed by Benjamin Peterson.)

Test cases can raise the SkipTest exception to skip a test (issue 1034053).

The error messages for assertEqual(),assertTrue(), and assertFalse()failures now provide more information. If you set thelongMessage attribute of your TestCase classes to True, both the standard error message and any additional message you provide will be printed for failures. (Added by Michael Foord; issue 5663.)

The assertRaises() method now returns a context handler when called without providing a callable object to run. For example, you can write this:

with self.assertRaises(KeyError): {}['foo']

(Implemented by Antoine Pitrou; issue 4444.)

Module- and class-level setup and teardown fixtures are now supported. Modules can contain setUpModule() and tearDownModule()functions. Classes can have setUpClass() andtearDownClass() methods that must be defined as class methods (using @classmethod or equivalent). These functions and methods are invoked when the test runner switches to a test case in a different module or class.

The methods addCleanup() anddoCleanups() were added.addCleanup() lets you add cleanup functions that will be called unconditionally (after setUp() ifsetUp() fails, otherwise after tearDown()). This allows for much simpler resource allocation and deallocation during tests (issue 5679).

A number of new methods were added that provide more specialized tests. Many of these methods were written by Google engineers for use in their test suites; Gregory P. Smith, Michael Foord, and GvR worked on merging them into Python’s version of unittest.

unittest.main() now takes an optional exit argument. If False, main() doesn’t call sys.exit(), allowingmain() to be used from the interactive interpreter. (Contributed by J. Pablo Fernández; issue 3379.)

TestResult has new startTestRun() andstopTestRun() methods that are called immediately before and after a test run. (Contributed by Robert Collins; issue 5728.)

With all these changes, the unittest.py was becoming awkwardly large, so the module was turned into a package and the code split into several files (by Benjamin Peterson). This doesn’t affect how the module is imported or used.

Updated module: ElementTree 1.3

The version of the ElementTree library included with Python was updated to version 1.3. Some of the new features are:

Outputs 1...

print ET.tostring(new)

Outputs ['\n ', '1', ' ', '2', ' ', '3', '\n']

print list(t.itertext())

Fredrik Lundh develops ElementTree and produced the 1.3 version; you can read his article describing 1.3 athttp://effbot.org/zone/elementtree-13-intro.htm. Florent Xicluna updated the version included with Python, after discussions on python-dev and in issue 6472.)

Build and C API Changes

Changes to Python’s build process and to the C API include:

Capsules

Python 3.1 adds a new C datatype, PyCapsule, for providing a C API to an extension module. A capsule is essentially the holder of a C void * pointer, and is made available as a module attribute; for example, the socket module’s API is exposed as socket.CAPI, and unicodedata exposes ucnhash_CAPI. Other extensions can import the module, access its dictionary to get the capsule object, and then get the void * pointer, which will usually point to an array of pointers to the module’s various API functions.

There is an existing data type already used for this,PyCObject, but it doesn’t provide type safety. Evil code written in pure Python could cause a segmentation fault by taking aPyCObject from module A and somehow substituting it for thePyCObject in module B. Capsules know their own name, and getting the pointer requires providing the name:

void *vtable;

if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") { PyErr_SetString(PyExc_ValueError, "argument type invalid"); return NULL; }

vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI");

You are assured that vtable points to whatever you’re expecting. If a different capsule was passed in, PyCapsule_IsValid() would detect the mismatched name and return false. Refer toProviding a C API for an Extension Module for more information on using these objects.

Python 2.7 now uses capsules internally to provide various extension-module APIs, but the PyCObject_AsVoidPtr() was modified to handle capsules, preserving compile-time compatibility with the CObject interface. Use ofPyCObject_AsVoidPtr() will signal aPendingDeprecationWarning, which is silent by default.

Implemented in Python 3.1 and backported to 2.7 by Larry Hastings; discussed in issue 5630.

Port-Specific Changes: Windows

Port-Specific Changes: Mac OS X

Port-Specific Changes: FreeBSD

Other Changes and Fixes

Porting to Python 2.7

This section lists previously described changes and other bugfixes that may require changes to your code:

In the standard library:

For C extensions:

For applications that embed Python:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, Hugh Secker-Walker.