[Python-Dev] PEP 565: show DeprecationWarning in main (round 2) (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sat Nov 25 00:33:50 EST 2017

• Previous message (by thread): [Python-Dev] PEP 559 - built-in noop()
• Next message (by thread): [Python-Dev] PEP 565: show DeprecationWarning in __main__ (round 2)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

This is a new version of the proposal to show DeprecationWarning in main.

The proposal itself hasn't changed (it's still recommending a new entry in the default filter list), but there have been several updates to the PEP text based on further development work and comments in the initial thread:

• there's now a linked issue and reference implementation
• it turns out we don't currently support the definition of module based filters at startup time, so I've explicitly noted the relevant enhancement that turned out to be necessary (allowing plain-string-or-compiled-regex in stored filter definitions where we currently only allow compiled regexes)
• I've noted the intended changes to the warnings-related documentation
• I've noted a couple of other relevant changes that Victor already implemented for 3.7
• I've noted that the motivation for the change in 2.7 & 3.1 covered all Python applications, not just developer tools (developer tools just provide a particularly compelling example of why "revert to the Python 2.6 behaviour" isn't a good answer)

Cheers, Nick.

================= PEP: 565 Title: Show DeprecationWarning in main Author: Nick Coghlan <ncoghlan at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Nov-2017 Python-Version: 3.7 Post-History: 12-Nov-2017, 25-Nov-2017

Abstract

In Python 2.7 and Python 3.2, the default warning filters were updated to hide DeprecationWarning by default, such that deprecation warnings in development tools that were themselves written in Python (e.g. linters, static analysers, test runners, code generators), as well as any other applications that merely happened to be written in Python, wouldn't be visible to their users unless those users explicitly opted in to seeing them.

However, this change has had the unfortunate side effect of making DeprecationWarning markedly less effective at its primary intended purpose: providing advance notice of breaking changes in APIs (whether in CPython, the standard library, or in third party libraries) to users of those APIs.

To improve this situation, this PEP proposes a single adjustment to the default warnings filter: displaying deprecation warnings attributed to the main module by default.

This change will mean that code entered at the interactive prompt and code in single file scripts will revert to reporting these warnings by default, while they will continue to be silenced by default for packaged code distributed as part of an importable module.

The PEP also proposes a number of small adjustments to the reference interpreter and standard library documentation to help make the warnings subsystem more approachable for new Python developers.

Specification

The current set of default warnings filters consists of::

ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning

The default unittest test runner then uses warnings.catch_warnings() warnings.simplefilter('default') to override the default filters while running test cases.

The change proposed in this PEP is to update the default warning filter list to be::

default::DeprecationWarning:__main__
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning

This means that in cases where the nominal location of the warning (as determined by the stacklevel parameter to warnings.warn) is in the __main__ module, the first occurrence of each DeprecationWarning will once again be reported.

This change will lead to DeprecationWarning being displayed by default for:

• code executed directly at the interactive prompt
• code executed directly as part of a single-file script

While continuing to be hidden by default for:

• code imported from another module in a zipapp archive's __main__.py file
• code imported from another module in an executable package's __main__ submodule
• code imported from an executable script wrapper generated at installation time based on a console_scripts or gui_scripts entry point definition

As a result, API deprecation warnings encountered by development tools written in Python should continue to be hidden by default for users of those tools

While not its originally intended purpose, the standard library documentation will also be updated to explicitly recommend the use of FutureWarning (rather than DeprecationWarning) for backwards compatibility warnings that are intended to be seen by users of an application.

This will give the following three distinct categories of backwards compatibility warning, with three different intended audiences:

• PendingDeprecationWarning: reported by default only in test runners that override the default set of warning filters. The intended audience is Python developers that take an active interest in ensuring the future compatibility of their software (e.g. professional Python application developers with specific support obligations).
• DeprecationWarning: reported by default for code that runs directly in the __main__ module (as such code is considered relatively unlikely to have a dedicated test suite), but relies on test suite based reporting for code in other modules. The intended audience is Python developers that are at risk of upgrades to their dependencies (including upgrades to Python itself) breaking their software (e.g. developers using Python to script environments where someone else is in control of the timing of dependency upgrades).
• FutureWarning: always reported by default. The intended audience is users of applications written in Python, rather than other Python developers (e.g. warning about use of a deprecated setting in a configuration file format).

Given its presence in the standard library since Python 2.3, FutureWarning would then also have a secondary use case for libraries and frameworks that support multiple Python versions: as a more reliably visible alternative to DeprecationWarning in Python 2.7 and versions of Python 3.x prior to 3.7.

Documentation Updates

The current reference documentation for the warnings system is relatively short on specific examples of possible settings for the -W command line option or the PYTHONWARNINGS environment variably that achieve particular end results.

The following improvements are proposed as part of the implementation of this PEP:

• Explicitly list the following entries under the description of the PYTHONWARNINGS environment variable::

    PYTHONWARNINGS=error # Convert to exceptions
    PYTHONWARNINGS=always # Warn every time
    PYTHONWARNINGS=default # Warn once per call location
    PYTHONWARNINGS=module # Warn once per calling module
    PYTHONWARNINGS=once # Warn once per Python process
    PYTHONWARNINGS=ignore # Never warn

• Explicitly list the corresponding short options (-We, -Wa, -Wd, -Wm,-Wo, -Wi) for each of the warning actions listed under the -W command line switch documentation

• Explicitly list the default filter set in the warnings module documentation, using the action::category and action::category:module notation

• Explicitly list the following snippet in the warnings.simplefilter documentation as a recommended approach to turning off all warnings by default in a Python application while still allowing them to be turned back on via PYTHONWARNINGS or the -W command line switch::

    if not sys.warnoptions:
        warnings.simplefilter("ignore")

None of these are new (they already work in all still supported Python versions), but they're not especially obvious given the current structure of the related documentation.

Reference Implementation

A reference implementation is available in the PR [4_] linked from the related tracker issue for this PEP [5_].

As a side-effect of implementing this PEP, the internal warnings filter list will start allowing the use of plain strings as part of filter definitions (in addition to the existing use of compiled regular expressions). When present, the plain strings will be compared for exact matches only. This approach allows the new default filter to be added during interpreter startup without requiring early access to the re module.

Motivation

As discussed in [1_] and mentioned in [2_], Python 2.7 and Python 3.2 changed the default handling of DeprecationWarning such that:

• the warning was hidden by default during normal code execution
• the unittest test runner was updated to re-enable it when running tests

The intent was to avoid cases of tooling output like the following::

$ devtool mycode/
/usr/lib/python3.6/site-packages/devtool/cli.py:1:

DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7 async = True ... actual tool output ...

Even when devtool is a tool specifically for Python programmers, this is not a particularly useful warning, as it will be shown on every invocation, even though the main helpful step an end user can take is to report a bug to the developers of devtool.

The warning is even less helpful for general purpose developer tools that are used across more languages than just Python, and almost entirely *un*helpful for applications that simply happen to be written in Python, and aren't necessarily intended for a developer audience at all.

However, this change proved to have unintended consequences for the following audiences:

• anyone using a test runner other than the default one built into unittest (the request for third party test runners to change their default warnings filters was never made explicitly, so many of them still rely on the interpreter defaults that are designed to suit deployed applications)
• anyone using the default unittest test runner to test their Python code in a subprocess (since even unittest only adjusts the warnings settings in the current process)
• anyone writing Python code at the interactive prompt or as part of a directly executed script that didn't have a Python level test suite at all

In these cases, DeprecationWarning ended up become almost entirely equivalent to PendingDeprecationWarning: it was simply never seen at all.

Limitations on PEP Scope

This PEP exists specifically to explain both the proposed addition to the default warnings filter for 3.7, and to more clearly articulate the rationale for the original change to the handling of DeprecationWarning back in Python 2.7 and 3.2.

This PEP does not solve all known problems with the current approach to handling deprecation warnings. Most notably:

• the default unittest test runner does not currently report deprecation warnings emitted at module import time, as the warnings filter override is only put in place during test execution, not during test discovery and loading.
• the default unittest test runner does not currently report deprecation warnings in subprocesses, as the warnings filter override is applied directly to the loaded warnings module, not to the PYTHONWARNINGS environment variable.
• the standard library doesn't provide a straightforward way to opt-in to seeing all warnings emitted by a particular dependency prior to upgrading it (the third-party warn module [3_] does provide this, but enabling it involves monkeypatching the standard library's warnings module).
• re-enabling deprecation warnings by default in main doesn't help in handling cases where software has been factored out into support modules, but those modules still have little or no automated test coverage. Near term, the best currently available answer is to run such applications with PYTHONWARNINGS=default::DeprecationWarning or python -W default::DeprecationWarning and pay attention to their stderr output. Longer term, this is really a question for researchers working on static analysis of Python code: how to reliably find usage of deprecated APIs, and how to infer that an API or parameter is deprecated based on warnings.warn calls, without actually running either the code providing the API or the code accessing it

While these are real problems with the status quo, they're excluded from consideration in this PEP because they're going to require more complex solutions than a single additional entry in the default warnings filter, and resolving them at least potentially won't require going through the PEP process.

For anyone interested in pursuing them further, the first two would be unittest module enhancement requests, the third would be a warnings module enhancement request, while the last would only require a PEP if inferring API deprecations from their contents was deemed to be an intractable code analysis problem, and an explicit function and parameter marker syntax in annotations was proposed instead.

The CPython reference implementation will also include the following related changes in 3.7:

• a new -X dev command line option that combines several developer centric settings (including -Wd) into one command line flag: https://bugs.python.org/issue32043
• changing the behaviour in debug builds to show more of the warnings that are off by default in regular interpeter builds: https://bugs.python.org/issue32088

References

.. [1] stdlib-sig thread proposing the original default filter change (https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)

.. [2] Python 2.7 notification of the default warnings filter change (https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)

.. [3] Emitting warnings based on the location of the warning itself (https://pypi.org/project/warn/)

.. [4] GitHub PR for PEP 565 implementation (https://github.com/python/cpython/pull/4458)

.. [5] Tracker issue for PEP 565 implementation (https://bugs.python.org/issue31975)

.. [6] python-dev discussion thread for this PEP (https://mail.python.org/pipermail/python-dev/2017-November/150477.html)

Copyright

This document has been placed in the public domain.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

• Previous message (by thread): [Python-Dev] PEP 559 - built-in noop()
• Next message (by thread): [Python-Dev] PEP 565: show DeprecationWarning in __main__ (round 2)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the Python-Dev mailing list