What’s New In Python 3.6 (original) (raw)

Editors:

Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io>

This article explains the new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016. See thechangelog for a full list of changes.

See also

PEP 494 - Python 3.6 Release Schedule

Summary – Release highlights

New syntax features:

New library modules:

CPython implementation improvements:

Significant improvements in the standard library:

Security improvements:

Windows improvements:

New Features

PEP 498: Formatted string literals

PEP 498 introduces a new kind of string literals: f-strings, orformatted string literals.

Formatted string literals are prefixed with 'f' and are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using theformat() protocol:

name = "Fred" f"He said his name is {name}." 'He said his name is Fred.' width = 10 precision = 4 value = decimal.Decimal("12.34567") f"result: {value:{width}.{precision}}" # nested fields 'result: 12.35'

PEP 526: Syntax for variable annotations

PEP 484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:

primes: List[int] = []

captain: str # Note: no initial value!

class Starship: stats: Dict[str, int] = {}

Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the__annotations__ attribute of a class or module.

In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the __annotations__ attribute.

See also

PEP 526 – Syntax for variable annotations.

PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, and Guido van Rossum. Implemented by Ivan Levkivskyi.

Tools that use or will use the new syntax:mypy,pytype, PyCharm, etc.

PEP 515: Underscores in Numeric Literals

PEP 515 adds the ability to use underscores in numeric literals for improved readability. For example:

1_000_000_000_000_000 1000000000000000 0x_FF_FF_FF_FF 4294967295

Single underscores are allowed between digits and after any base specifier. Leading, trailing, or multiple underscores in a row are not allowed.

The string formatting language also now has support for the '_' option to signal the use of an underscore for a thousands separator for floating-point presentation types and for integer presentation type 'd'. For integer presentation types 'b','o', 'x', and 'X', underscores will be inserted every 4 digits:

'{:_}'.format(1000000) '1_000_000' '{:_x}'.format(0xFFFFFFFF) 'ffff_ffff'

See also

PEP 515 – Underscores in Numeric Literals

PEP written by Georg Brandl and Serhiy Storchaka.

PEP 525: Asynchronous Generators

PEP 492 introduced support for native coroutines and async / awaitsyntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:

async def ticker(delay, to): """Yield numbers from 0 to to every delay seconds.""" for i in range(to): yield i await asyncio.sleep(delay)

The new syntax allows for faster and more concise code.

See also

PEP 525 – Asynchronous Generators

PEP written and implemented by Yury Selivanov.

PEP 530: Asynchronous Comprehensions

PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions:

result = [i async for i in aiter() if i % 2]

Additionally, await expressions are supported in all kinds of comprehensions:

result = [await fun() for fun in funcs if await condition()]

See also

PEP 530 – Asynchronous Comprehensions

PEP written and implemented by Yury Selivanov.

PEP 487: Simpler customization of class creation

It is now possible to customize subclass creation without using a metaclass. The new __init_subclass__ classmethod will be called on the base class whenever a new subclass is created:

class PluginBase: subclasses = []

def __init_subclass__(cls, **kwargs):
    super().__init_subclass__(**kwargs)
    cls.subclasses.append(cls)

class Plugin1(PluginBase): pass

class Plugin2(PluginBase): pass

In order to allow zero-argument super() calls to work correctly from__init_subclass__() implementations, custom metaclasses must ensure that the new __classcell__ namespace entry is propagated totype.__new__ (as described in Creating the class object).

PEP 487: Descriptor Protocol Enhancements

PEP 487 extends the descriptor protocol to include the new optional__set_name__() method. Whenever a new class is defined, the new method will be called on all descriptors included in the definition, providing them with a reference to the class being defined and the name given to the descriptor within the class namespace. In other words, instances of descriptors can now know the attribute name of the descriptor in the owner class:

class IntField: def get(self, instance, owner): return instance.dict[self.name]

def __set__(self, instance, value):
    if not isinstance(value, int):
        raise ValueError(f'expecting integer in {self.name}')
    instance.__dict__[self.name] = value

# this is the new initializer:
def __set_name__(self, owner, name):
    self.name = name

class Model: int_field = IntField()

PEP 519: Adding a file system path protocol

File system paths have historically been represented as stror bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python’s standard library.

To fix this situation, a new interface represented byos.PathLike has been defined. By implementing the__fspath__() method, an object signals that it represents a path. An object can then provide a low-level representation of a file system path as a str orbytes object. This means an object is consideredpath-like if it implementsos.PathLike or is a str or bytes object which represents a file system path. Code can use os.fspath(),os.fsdecode(), or os.fsencode() to explicitly get astr and/or bytes representation of a path-like object.

The built-in open() function has been updated to acceptos.PathLike objects, as have all relevant functions in theos and os.path modules, and most other functions and classes in the standard library. The os.DirEntry class and relevant classes in pathlib have also been updated to implement os.PathLike.

The hope is that updating the fundamental functions for operating on file system paths will lead to third-party code to implicitly support all path-like objects without any code changes, or at least very minimal ones (e.g. callingos.fspath() at the beginning of code before operating on a path-like object).

Here are some examples of how the new interface allows forpathlib.Path to be used more easily and transparently with pre-existing code:

import pathlib with open(pathlib.Path("README")) as f: ... contents = f.read() ... import os.path os.path.splitext(pathlib.Path("some_file.txt")) ('some_file', '.txt') os.path.join("/a/b", pathlib.Path("c")) '/a/b/c' import os os.fspath(pathlib.Path("some_file.txt")) 'some_file.txt'

(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.)

See also

PEP 519 – Adding a file system path protocol

PEP written by Brett Cannon and Koos Zevenhoven.

PEP 495: Local Time Disambiguation

In most world locations, there have been and will be times when local clocks are moved back. In those times, intervals are introduced in which local clocks show the same time twice in the same day. In these situations, the information displayed on a local clock (or stored in a Python datetime instance) is insufficient to identify a particular moment in time.

PEP 495 adds the new fold attribute to instances ofdatetime.datetime and datetime.time classes to differentiate between two moments in time for which local times are the same:

u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc) for i in range(4): ... u = u0 + i*HOUR ... t = u.astimezone(Eastern) ... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold) ... 04:00:00 UTC = 00:00:00 EDT 0 05:00:00 UTC = 01:00:00 EDT 0 06:00:00 UTC = 01:00:00 EST 1 07:00:00 UTC = 02:00:00 EST 0

The values of the fold attribute have the value 0 for all instances except those that represent the second (chronologically) moment in time in an ambiguous case.

See also

PEP 495 – Local Time Disambiguation

PEP written by Alexander Belopolsky and Tim Peters, implementation by Alexander Belopolsky.

PEP 529: Change Windows filesystem encoding to UTF-8

Representing filesystem paths is best performed with str (Unicode) rather than bytes. However, there are some situations where using bytes is sufficient and correct.

Prior to Python 3.6, data loss could result when using bytes paths on Windows. With this change, using bytes to represent paths is now supported on Windows, provided those bytes are encoded with the encoding returned bysys.getfilesystemencoding(), which now defaults to 'utf-8'.

Applications that do not use str to represent paths should useos.fsencode() and os.fsdecode() to ensure their bytes are correctly encoded. To revert to the previous behaviour, setPYTHONLEGACYWINDOWSFSENCODING or callsys._enablelegacywindowsfsencoding().

See PEP 529 for more information and discussion of code modifications that may be required.

PEP 528: Change Windows console encoding to UTF-8

The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin,sys.stdout and sys.stderr now default to utf-8 encoding.

This change only applies when using an interactive console, and not when redirecting files or pipes. To revert to the previous behaviour for interactive console use, set PYTHONLEGACYWINDOWSSTDIO.

See also

PEP 528 – Change Windows console encoding to UTF-8

PEP written and implemented by Steve Dower.

PEP 520: Preserving Class Attribute Definition Order

Attributes in a class definition body have a natural ordering: the same order in which the names appear in the source. This order is now preserved in the new class’s __dict__ attribute.

Also, the effective default class execution namespace (returned fromtype.__prepare__()) is now an insertion-order-preserving mapping.

See also

PEP 520 – Preserving Class Attribute Definition Order

PEP written and implemented by Eric Snow.

PEP 468: Preserving Keyword Argument Order

**kwargs in a function signature is now guaranteed to be an insertion-order-preserving mapping.

See also

PEP 468 – Preserving Keyword Argument Order

PEP written and implemented by Eric Snow.

New dict implementation

The dict type now uses a “compact” representation based on a proposal by Raymond Hettingerwhich was first implemented by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5.

The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5).

(Contributed by INADA Naoki in bpo-27350. Ideaoriginally suggested by Raymond Hettinger.)

PEP 523: Adding a frame evaluation API to CPython

While Python provides extensive support to customize how code executes, one place it has not done so is in the evaluation of frame objects. If you wanted some way to intercept frame evaluation in Python there really wasn’t any way without directly manipulating function pointers for defined functions.

PEP 523 changes this by providing an API to make frame evaluation pluggable at the C level. This will allow for tools such as debuggers and JITs to intercept frame evaluation before the execution of Python code begins. This enables the use of alternative evaluation implementations for Python code, tracking frame evaluation, etc.

This API is not part of the limited C API and is marked as private to signal that usage of this API is expected to be limited and only applicable to very select, low-level use-cases. Semantics of the API will change with Python as necessary.

See also

PEP 523 – Adding a frame evaluation API to CPython

PEP written by Brett Cannon and Dino Viehland.

PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows setting the Python memory allocators and installing debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:

Checking if the GIL is held is also a new feature of Python 3.6.

See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators.

It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. This is helpful when using external memory debuggers like Valgrind on a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use thetracemalloc module to get the traceback where a memory block was allocated.

Example of fatal error on buffer overflow usingpython3.6 -X tracemalloc=5 (store 5 frames in traces):

Debug memory block at address p=0x7fbcd41666f8: API 'o' 4 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): at tail+0: 0x02 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #1233329 to debug malloc/realloc. Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first): File "test/test_bytes.py", line 323 File "unittest/case.py", line 600 File "unittest/case.py", line 648 File "unittest/suite.py", line 122 File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first): File "test/test_bytes.py", line 323 in test_hex File "unittest/case.py", line 600 in run File "unittest/case.py", line 648 in call File "unittest/suite.py", line 122 in run File "unittest/suite.py", line 84 in call File "unittest/suite.py", line 122 in run File "unittest/suite.py", line 84 in call ...

(Contributed by Victor Stinner in bpo-26516 and bpo-26564.)

DTrace and SystemTap probing support

Python can now be built --with-dtrace which enables static markers for the following events in the interpreter:

This can be used to instrument running interpreters in production, without the need to recompile specific debug builds or providing application-specific profiling/debugging code.

More details in Instrumenting CPython with DTrace and SystemTap.

The current implementation is tested on Linux and macOS. Additional markers may be added in the future.

(Contributed by Łukasz Langa in bpo-21590, based on patches by Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

Other Language Changes

Some smaller changes made to the core Python language are:

New Modules

secrets

The main purpose of the new secrets module is to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar.

Warning

Note that the pseudo-random generators in the random module should NOT be used for security purposes. Use secretson Python 3.6+ and os.urandom() on Python 3.5 and earlier.

See also

PEP 506 – Adding A Secrets Module To The Standard Library

PEP written and implemented by Steven D’Aprano.

Improved Modules

array

Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended. This is consistent with the behavior of other mutable sequences.

Contributed by Serhiy Storchaka in bpo-26492.

ast

The new ast.Constant AST node has been added. It can be used by external AST optimizers for the purposes of constant folding.

Contributed by Victor Stinner in bpo-26146.

asyncio

Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.

Notable changes in the asyncio module since Python 3.5.0 (all backported to 3.5.x due to the provisional status):

binascii

The b2a_base64() function now accepts an optional _newline_keyword argument to control whether the newline character is appended to the return value. (Contributed by Victor Stinner in bpo-25357.)

cmath

The new cmath.tau (τ) constant has been added. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)

New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infjand cmath.nanj to match the format used by complex repr. (Contributed by Mark Dickinson in bpo-23229.)

collections

The new Collection abstract base class has been added to represent sized iterable container classes. (Contributed by Ivan Levkivskyi, docs by Neil Girdhar in bpo-27598.)

The new Reversible abstract base class represents iterable classes that also provide the __reversed__() method. (Contributed by Ivan Levkivskyi in bpo-25987.)

The new AsyncGenerator abstract base class represents asynchronous generators. (Contributed by Yury Selivanov in bpo-28720.)

The namedtuple() function now accepts an optional keyword argument module, which, when specified, is used for the __module__ attribute of the returned named tuple class. (Contributed by Raymond Hettinger in bpo-17941.)

The verbose and rename arguments fornamedtuple() are now keyword-only. (Contributed by Raymond Hettinger in bpo-25628.)

Recursive collections.deque instances can now be pickled. (Contributed by Serhiy Storchaka in bpo-26482.)

concurrent.futures

The ThreadPoolExecutorclass constructor now accepts an optional thread_name_prefix argument to make it possible to customize the names of the threads created by the pool. (Contributed by Gregory P. Smith in bpo-27664.)

contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for __enter__() which returnsself and leaves __exit__() an abstract method. A matching class has been added to the typing module astyping.ContextManager. (Contributed by Brett Cannon in bpo-25609.)

datetime

The datetime and time classes have the new fold attribute used to disambiguate local time when necessary. Many functions in the datetime have been updated to support local time disambiguation. See Local Time Disambiguation section for more information. (Contributed by Alexander Belopolsky in bpo-24773.)

The datetime.strftime() anddate.strftime() methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in bpo-12006.)

The datetime.isoformat() function now accepts an optional timespec argument that specifies the number of additional components of the time value to include. (Contributed by Alessandro Cucci and Alexander Belopolsky in bpo-19475.)

The datetime.combine() now accepts an optional tzinfo argument. (Contributed by Alexander Belopolsky in bpo-27661.)

decimal

New Decimal.as_integer_ratio()method that returns a pair (n, d) of integers that represent the givenDecimal instance as a fraction, in lowest terms and with a positive denominator:

Decimal('-3.14').as_integer_ratio() (-157, 50)

(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928.)

distutils

The default_format attribute has been removed fromdistutils.command.sdist.sdist and the formatsattribute defaults to ['gztar']. Although not anticipated, any code relying on the presence of default_format may need to be adapted. See bpo-27819 for more details.

email

The new email API, enabled via the policy keyword to various constructors, is no longer provisional. The email documentation has been reorganized and rewritten to focus on the new API, while retaining the old documentation for the legacy API. (Contributed by R. David Murray in bpo-24277.)

The email.mime classes now all accept an optional policy keyword. (Contributed by Berker Peksag in bpo-27331.)

The DecodedGenerator now supports the _policy_keyword.

There is a new policy attribute,message_factory, that controls what class is used by default when the parser creates new message objects. For theemail.policy.compat32 policy this is Message, for the new policies it is EmailMessage. (Contributed by R. David Murray in bpo-20476.)

encodings

On Windows, added the 'oem' encoding to use CP_OEMCP, and the 'ansi'alias for the existing 'mbcs' encoding, which uses the CP_ACP code page. (Contributed by Steve Dower in bpo-27959.)

enum

Two new enumeration base classes have been added to the enum module:Flag and IntFlags. Both are used to define constants that can be combined using the bitwise operators. (Contributed by Ethan Furman in bpo-23591.)

Many standard library modules have been updated to use theIntFlags class for their constants.

The new enum.auto value can be used to assign values to enum members automatically:

from enum import Enum, auto class Color(Enum): ... red = auto() ... blue = auto() ... green = auto() ... list(Color) [<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>]

faulthandler

On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable(). (Contributed by Victor Stinner inbpo-23848.)

fileinput

hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in bpo-25788.)

hashlib

hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)

BLAKE2 hash functions were added to the module. blake2b()and blake2s() are always available and support the full feature set of BLAKE2. (Contributed by Christian Heimes in bpo-26798 based on code by Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)

The SHA-3 hash functions sha3_224(), sha3_256(),sha3_384(), sha3_512(), and SHAKE hash functionsshake_128() and shake_256() were added. (Contributed by Christian Heimes in bpo-16113. Keccak Code Package by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and Ronny Van Keer.)

The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer. (Contributed by Christian Heimes in bpo-27928.)

http.client

HTTPConnection.request() andendheaders() both now support chunked encoding request bodies. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

idlelib and IDLE

The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.

‘Modernizing’ includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy inbpo-24225. Most idlelib patches since have been and will be part of the process.)

In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.

New in 3.6.2:

Multiple fixes for autocompletion. (Contributed by Louie Lu in bpo-15786.)

New in 3.6.3:

Module Browser (on the File menu, formerly called Class Browser), now displays nested functions and classes in addition to top-level functions and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy in bpo-1612262.)

The IDLE features formerly implemented as extensions have been reimplemented as normal features. Their settings have been moved from the Extensions tab to other dialog tabs. (Contributed by Charles Wohlganger and Terry Jan Reedy in bpo-27099.)

The Settings dialog (Options, Configure IDLE) has been partly rewritten to improve both appearance and function. (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)

New in 3.6.4:

The font sample now includes a selection of non-Latin characters so that users can better see the effect of selecting a particular font. (Contributed by Terry Jan Reedy in bpo-13802.) The sample can be edited to include other characters. (Contributed by Serhiy Storchaka in bpo-31860.)

New in 3.6.6:

Editor code context option revised. Box displays all context lines up to maxlines. Clicking on a context line jumps the editor to that line. Context colors for custom themes is added to Highlights tab of Settings dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in bpo-33642,bpo-33768, and bpo-33679.)

On Windows, a new API call tells Windows that tk scales for DPI. On Windows 8.1+ or 10, with DPI compatibility properties of the Python binary unchanged, and a monitor resolution greater than 96 DPI, this should make text and lines sharper. It should otherwise have no effect. (Contributed by Terry Jan Reedy in bpo-33656.)

New in 3.6.7:

Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in bpo-1529353.)

importlib

Import now raises the new exception ModuleNotFoundError(subclass of ImportError) when it cannot find a module. Code that current checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)

importlib.util.LazyLoader now callscreate_module() on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter andimportlib.machinery.ExtensionFileLoader couldn’t be used withimportlib.util.LazyLoader.

importlib.util.cache_from_source(),importlib.util.source_from_cache(), andimportlib.util.spec_from_file_location() now accept apath-like object.

inspect

The inspect.signature() function now reports the implicit .0 parameters generated by the compiler for comprehension and generator expression scopes as if they were positional-only parameters calledimplicit0. (Contributed by Jelle Zijlstra in bpo-19611.)

To reduce code churn when upgrading from Python 2.7 and the legacyinspect.getargspec() API, the previously documented deprecation ofinspect.getfullargspec() has been reversed. While this function is convenient for single/source Python 2/3 code bases, the richerinspect.signature() interface remains the recommended approach for new code. (Contributed by Nick Coghlan in bpo-27172)

json

json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32. (Contributed by Serhiy Storchaka in bpo-17909.)

logging

The new WatchedFileHandler.reopenIfNeeded()method has been added to add the ability to check if the log file needs to be reopened. (Contributed by Marian Horban in bpo-24884.)

math

The tau (τ) constant has been added to the math and cmathmodules. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)

multiprocessing

Proxy Objects returned bymultiprocessing.Manager() can now be nested. (Contributed by Davin Potts in bpo-6766.)

os

See the summary of PEP 519 for details on how theos and os.path modules now supportpath-like objects.

scandir() now supports bytes paths on Windows.

A new close() method allows explicitly closing ascandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir()iterator is neither exhausted nor explicitly closed a ResourceWarningwill be emitted in its destructor. (Contributed by Serhiy Storchaka in bpo-25994.)

On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.

The Linux getrandom() syscall (get random bytes) is now exposed as the newos.getrandom() function. (Contributed by Victor Stinner, part of the PEP 524)

pathlib

pathlib now supports path-like objects. (Contributed by Brett Cannon in bpo-27186.)

See the summary of PEP 519 for details.

pdb

The Pdb class constructor has a new optional readrc argument to control whether .pdbrc files should be read.

pickle

Objects that need __new__ called with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in bpo-24164.)

pickletools

pickletools.dis() now outputs the implicit memo index for theMEMOIZE opcode. (Contributed by Serhiy Storchaka in bpo-25382.)

pydoc

The pydoc module has learned to respect the MANPAGERenvironment variable. (Contributed by Matthias Klose in bpo-8637.)

help() and pydoc can now list named tuple fields in the order they were defined rather than alphabetically. (Contributed by Raymond Hettinger in bpo-24879.)

random

The new choices() function returns a list of elements of specified size from the given population with optional weights. (Contributed by Raymond Hettinger in bpo-18844.)

re

Added support of modifier spans in regular expressions. Examples:'(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON';'(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'. (Contributed by Serhiy Storchaka in bpo-433028.)

Match object groups can be accessed by __getitem__, which is equivalent to group(). So mo['name'] is now equivalent tomo.group('name'). (Contributed by Eric Smith in bpo-24454.)

Match objects now supportindex-like objects as group indices. (Contributed by Jeroen Demeyer and Xiang Zhang in bpo-27177.)

readline

Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in bpo-26870.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in bpo-25011 and bpo-25209.)

shlex

The shlex has muchimproved shell compatibilitythrough the new punctuation_chars argument to control which characters are treated as punctuation. (Contributed by Vinay Sajip in bpo-1521950.)

site

When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in bpo-26587).

sqlite3

sqlite3.Cursor.lastrowid now supports the REPLACE statement. (Contributed by Alex LordThorsen in bpo-16864.)

socket

The ioctl() function now supports theSIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in bpo-26536.)

The getsockopt() constants SO_DOMAIN,SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported. (Contributed by Christian Heimes in bpo-26907.)

The setsockopt() now supports thesetsockopt(level, optname, None, optlen: int) form. (Contributed by Christian Heimes in bpo-27744.)

The socket module now supports the address familyAF_ALG to interface with Linux Kernel crypto API. ALG_*,SOL_ALG and sendmsg_afalg() were added. (Contributed by Christian Heimes in bpo-27744 with support from Victor Stinner.)

New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added. (Contributed by Omar Sandoval, bpo-26273).

socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server andwsgiref.simple_server, now support the context managerprotocol. (Contributed by Aviv Palivoda in bpo-26404.)

The wfile attribute ofStreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling write() is now guaranteed to send the data in full. (Contributed by Martin Panter in bpo-26721.)

ssl

ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)

3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. (Contributed by Christian Heimes in bpo-27850 and bpo-27766.)

SSLContext has better default configuration for options and ciphers. (Contributed by Christian Heimes in bpo-28043.)

SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in bpo-19500 based on a draft by Alex Warhawk.)

The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority.

All constants and flags have been converted to IntEnum andIntFlags. (Contributed by Christian Heimes in bpo-28025.)

Server and client-side specific TLS protocols for SSLContextwere added. (Contributed by Christian Heimes in bpo-28085.)

Added ssl.SSLContext.post_handshake_auth to enable andssl.SSLSocket.verify_client_post_handshake() to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in gh-78851.)

statistics

A new harmonic_mean() function has been added. (Contributed by Steven D’Aprano in bpo-27181.)

struct

struct now supports IEEE 754 half-precision floats via the 'e'format specifier. (Contributed by Eli Stevens, Mark Dickinson in bpo-11734.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or explicitly call the wait() method to read the exit status of the child process. (Contributed by Victor Stinner inbpo-26741.)

The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin, stdout and stderr streams. (Contributed by Steve Dower in bpo-6135.)

sys

The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in bpo-27781.)

On Windows the return value of the getwindowsversion() function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in bpo-27932.)

telnetlib

telnetlib.Telnet is now a context manager (contributed by Stéphane Wirtel in bpo-25485).

time

The struct_time attributes tm_gmtoff andtm_zone are now available on all platforms.

timeit

The new Timer.autorange() convenience method has been added to call Timer.timeit()repeatedly so that the total run time is greater or equal to 200 milliseconds. (Contributed by Steven D’Aprano in bpo-6422.)

timeit now warns when there is substantial (4x) variance between best and worst times. (Contributed by Serhiy Storchaka in bpo-23552.)

tkinter

Added methods trace_add(),trace_remove() and trace_info()in the tkinter.Variable class. They replace old methodstrace_variable(), trace(),trace_vdelete() andtrace_vinfo() that use obsolete Tcl commands and might not work in future versions of Tcl. (Contributed by Serhiy Storchaka in bpo-22115).

traceback

Both the traceback module and the interpreter’s builtin exception display now abbreviate long sequences of repeated lines in tracebacks as shown in the following example:

def f(): f() ... f() Traceback (most recent call last): File "", line 1, in File "", line 1, in f File "", line 1, in f File "", line 1, in f [Previous line repeated 995 more times] RecursionError: maximum recursion depth exceeded

(Contributed by Emanuel Barry in bpo-26823.)

tracemalloc

The tracemalloc module now supports tracing memory allocations in multiple different address spaces.

The new DomainFilter filter class has been added to filter block traces by their address space (domain).

(Contributed by Victor Stinner in bpo-26588.)

typing

Since the typing module is provisional, all changes introduced in Python 3.6 have also been backported to Python 3.5.x.

The typing module has a much improved support for generic type aliases. For example Dict[str, Tuple[S, T]] is now a valid type annotation. (Contributed by Guido van Rossum in Github #195.)

The typing.ContextManager class has been added for representing contextlib.AbstractContextManager. (Contributed by Brett Cannon in bpo-25609.)

The typing.Collection class has been added for representing collections.abc.Collection. (Contributed by Ivan Levkivskyi in bpo-27598.)

The typing.ClassVar type construct has been added to mark class variables. As introduced in PEP 526, a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. (Contributed by Ivan Levkivskyi in Github #280.)

A new TYPE_CHECKING constant that is assumed to beTrue by the static type checkers, but is False at runtime. (Contributed by Guido van Rossum in Github #230.)

A new NewType() helper function has been added to create lightweight distinct types for annotations:

from typing import NewType

UserId = NewType('UserId', int) some_id = UserId(524313)

The static type checker will treat the new type as if it were a subclass of the original type. (Contributed by Ivan Levkivskyi in Github #189.)

unicodedata

The unicodedata module now uses data from Unicode 9.0.0. (Contributed by Benjamin Peterson.)

unittest.mock

The Mock class has the following improvements:

urllib.request

If a HTTP request has a file or iterable body (other than a bytes object) but no Content-Length header, rather than throwing an error, AbstractHTTPHandler now falls back to use chunked transfer encoding. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

urllib.robotparser

RobotFileParser now supports the Crawl-delay andRequest-rate extensions. (Contributed by Nikolay Bogoychev in bpo-16099.)

venv

venv accepts a new parameter --prompt. This parameter provides an alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak and ported to 3.6 by Stéphane Wirtel in bpo-22829.)

warnings

A new optional source parameter has been added to thewarnings.warn_explicit() function: the destroyed object which emitted aResourceWarning. A source attribute has also been added towarnings.WarningMessage (contributed by Victor Stinner inbpo-26568 and bpo-26567).

When a ResourceWarning warning is logged, the tracemalloc module is now used to try to retrieve the traceback where the destroyed object was allocated.

Example with the script example.py:

import warnings

def func(): return open(file)

f = func() f = None

Output of the command python3.6 -Wd -X tracemalloc=5 example.py:

example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'> f = None Object allocated at (most recent call first): File "example.py", lineno 4 return open(file) File "example.py", lineno 6 f = func()

The “Object allocated at” traceback is new and is only displayed iftracemalloc is tracing Python memory allocations and if thewarnings module was already imported.

winreg

Added the 64-bit integer type REG_QWORD. (Contributed by Clement Rouault in bpo-23026.)

winsound

Allowed keyword arguments to be passed to Beep,MessageBeep, and PlaySound (bpo-27982).

xmlrpc.client

The xmlrpc.client module now supports unmarshalling additional data types used by the Apache XML-RPC implementation for numerics and None. (Contributed by Serhiy Storchaka in bpo-26885.)

zipfile

A new ZipInfo.from_file() class method allows making a ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() method can be used to check if the ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in bpo-26039.)

The ZipFile.open() method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in bpo-26039.)

zlib

The compress() and decompress() functions now accept keyword arguments. (Contributed by Aviv Palivoda in bpo-26243 and Xiang Zhang in bpo-16764 respectively.)

Optimizations

Build and C API Changes

Other Improvements

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7. Starting in Python 3.6, the use ofasync or await as names will generate a DeprecationWarning.

Deprecated Python behavior

Raising the StopIteration exception inside a generator will now generate a DeprecationWarning, and will trigger a RuntimeErrorin Python 3.7. See PEP 479: Change StopIteration handling inside generators for details.

The __aiter__() method is now expected to return an asynchronous iterator directly instead of returning an awaitable as previously. Doing the former will trigger a DeprecationWarning. Backward compatibility will be removed in Python 3.7. (Contributed by Yury Selivanov in bpo-27243.)

A backslash-character pair that is not a valid escape sequence now generates a DeprecationWarning. Although this will eventually become aSyntaxError, that will not be for several Python releases. (Contributed by Emanuel Barry in bpo-27364.)

When performing a relative import, falling back on __name__ and__path__ from the calling module when __spec__ or__package__ are not defined now raises an ImportWarning. (Contributed by Rose Ames in bpo-25791.)

Deprecated Python modules, functions and methods

asynchat

The asynchat has been deprecated in favor of asyncio. (Contributed by Mariatta in bpo-25002.)

asyncore

The asyncore has been deprecated in favor of asyncio. (Contributed by Mariatta in bpo-25002.)

dbm

Unlike other [dbm](../library/dbm.html#module-dbm "dbm: Interfaces to various Unix "database" formats.") implementations, the dbm.dumb module creates databases with the 'rw' mode and allows modifying the database opened with the 'r' mode. This behavior is now deprecated and will be removed in 3.8. (Contributed by Serhiy Storchaka in bpo-21708.)

distutils

The undocumented extra_path argument to thedistutils.Distribution constructor is now considered deprecated and will raise a warning if set. Support for this parameter will be removed in a future Python release. See bpo-27919 for details.

grp

The support of non-integer arguments in getgrgid() has been deprecated. (Contributed by Serhiy Storchaka in bpo-26129.)

importlib

The importlib.machinery.SourceFileLoader.load_module() andimportlib.machinery.SourcelessFileLoader.load_module() methods are now deprecated. They were the only remaining implementations ofimportlib.abc.Loader.load_module() in importlib that had not been deprecated in previous versions of Python in favour ofimportlib.abc.Loader.exec_module().

The importlib.machinery.WindowsRegistryFinder class is now deprecated. As of 3.6.0, it is still added to sys.meta_path by default (on Windows), but this may change in future releases.

os

Undocumented support of general bytes-like objectsas paths in os functions, compile() and similar functions is now deprecated. (Contributed by Serhiy Storchaka in bpo-25791 and bpo-26754.)

re

Support for inline flags (?letters) in the middle of the regular expression has been deprecated and will be removed in a future Python version. Flags at the start of a regular expression are still allowed. (Contributed by Serhiy Storchaka in bpo-22493.)

ssl

OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. In the future the ssl module will require at least OpenSSL 1.0.2 or 1.1.0.

SSL-related arguments like certfile, keyfile and check_hostnamein ftplib, http.client, imaplib, poplib, and smtplib have been deprecated in favor of context. (Contributed by Christian Heimes in bpo-28022.)

A couple of protocols and functions of the ssl module are now deprecated. Some features will no longer be available in future versions of OpenSSL. Other features are deprecated in favor of a different API. (Contributed by Christian Heimes in bpo-28022 and bpo-26470.)

tkinter

The tkinter.tix module is now deprecated. tkinter users should use tkinter.ttk instead.

venv

The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in bpo-25154.)

xml

Deprecated functions and types of the C API

Undocumented functions PyUnicode_AsEncodedObject(),PyUnicode_AsDecodedObject(), PyUnicode_AsEncodedUnicode()and PyUnicode_AsDecodedUnicode() are deprecated now. Use the generic codec based API instead.

Deprecated Build Options

The --with-system-ffi configure flag is now on by default on non-macOS UNIX platforms. It may be disabled by using --without-system-ffi, but using the flag is deprecated and will not be accepted in Python 3.7. macOS is unaffected by this change. Note that many OS distributors already use the --with-system-ffi flag when building their system Python.

Removed

API and Feature Removals

Porting to Python 3.6

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in ‘python’ Command Behavior

Changes in the Python API

Changes in the C API

CPython bytecode changes

There have been several major changes to the bytecode in Python 3.6.

Notable changes in Python 3.6.2

New make regen-all build target

To simplify cross-compilation, and to ensure that CPython can reliably be compiled without requiring an existing version of Python to already be available, the autotools-based build system no longer attempts to implicitly recompile generated files based on file modification times.

Instead, a new make regen-all command has been added to force regeneration of these files when desired (e.g. after an initial version of Python has already been built based on the pregenerated versions).

More selective regeneration targets are also defined - seeMakefile.pre.in for details.

(Contributed by Victor Stinner in bpo-23404.)

Added in version 3.6.2.

Removal of make touch build target

The make touch build target previously used to request implicit regeneration of generated files by updating their modification times has been removed.

It has been replaced by the new make regen-all target.

(Contributed by Victor Stinner in bpo-23404.)

Changed in version 3.6.2.

Notable changes in Python 3.6.4

The PyExc_RecursionErrorInst singleton that was part of the public API has been removed as its members being never cleared may cause a segfault during finalization of the interpreter. (Contributed by Xavier de Gaye in bpo-22898 and bpo-30697.)

Notable changes in Python 3.6.5

The locale.localeconv() function now sets temporarily the LC_CTYPElocale to the LC_NUMERIC locale in some cases. (Contributed by Victor Stinner in bpo-31900.)

Notable changes in Python 3.6.7

xml.dom.minidom and xml.sax modules no longer process external entities by default. See also gh-61441.

In 3.6.7 the tokenize module now implicitly emits a NEWLINE token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in bpo-33899.)

Notable changes in Python 3.6.10

Due to significant security concerns, the reuse_address parameter ofasyncio.loop.create_datagram_endpoint() is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov inbpo-37228.)

Notable changes in Python 3.6.13

Earlier Python versions allowed using both ; and & as query parameter separators in urllib.parse.parse_qs() andurllib.parse.parse_qsl(). Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affectscgi.parse() and cgi.parse_multipart() as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967.)

Notable changes in Python 3.6.14

A security fix alters the ftplib.FTP behavior to not trust the IPv4 address sent from the remote server when setting up a passive data channel. We reuse the ftp server IP address instead. For unusual code requiring the old behavior, set a trust_server_pasv_ipv4_addressattribute on your FTP instance to True. (See gh-87451)

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates RFC 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser urllib.parse() preventing such attacks. The removal characters are controlled by a new module level variableurllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See gh-88048)