PEP 3151 – Reworking the OS and IO exception hierarchy | peps.python.org (original) (raw)

Author:

Antoine Pitrou

BDFL-Delegate:

Barry Warsaw

Status:

Final

Type:

Standards Track

Created:

21-Jul-2010

Python-Version:

3.3

Post-History:

Resolution:

Python-Dev message

Table of Contents

Abstract

The standard exception hierarchy is an important part of the Python language. It has two defining qualities: it is both generic and selective. Generic in that the same exception type can be raised - and handled - regardless of the context (for example, whether you are trying to add something to an integer, to call a string method, or to write an object on a socket, a TypeError will be raised for bad argument types). Selective in that it allows the user to easily handle (silence, examine, process, store or encapsulate…) specific kinds of error conditions while letting other errors bubble up to higher calling contexts. For example, you can choose to catch ZeroDivisionErrors without affecting the default handling of other ArithmeticErrors (such as OverflowErrors).

This PEP proposes changes to a part of the exception hierarchy in order to better embody the qualities mentioned above: the errors related to operating system calls (OSError, IOError, mmap.error, select.error, and all their subclasses).

Rationale

Lack of fine-grained exceptions

The current variety of OS-related exceptions doesn’t allow the user to filter easily for the desired kinds of failures. As an example, consider the task of deleting a file if it exists. The Look Before You Leap (LBYL) idiom suffers from an obvious race condition:

if os.path.exists(filename): os.remove(filename)

If a file named as filename is created by another thread or process between the calls to os.path.exists and os.remove, it won’t be deleted. This can produce bugs in the application, or even security issues.

Therefore, the solution is to try to remove the file, and ignore the error if the file doesn’t exist (an idiom known as Easier to Ask Forgiveness than to get Permission, or EAFP). Careful code will read like the following (which works under both POSIX and Windows systems):

try: os.remove(filename) except OSError as e: if e.errno != errno.ENOENT: raise

or even:

try: os.remove(filename) except EnvironmentError as e: if e.errno != errno.ENOENT: raise

This is a lot more to type, and also forces the user to remember the various cryptic mnemonics from the errno module. It imposes an additional cognitive burden and gets tiresome rather quickly. Consequently, many programmers will instead write the following code, which silences exceptions too broadly:

try: os.remove(filename) except OSError: pass

os.remove can raise an OSError not only when the file doesn’t exist, but in other possible situations (for example, the filename points to a directory, or the current process doesn’t have permission to remove the file), which all indicate bugs in the application logic and therefore shouldn’t be silenced. What the programmer would like to write instead is something such as:

try: os.remove(filename) except FileNotFoundError: pass

Compatibility strategy

Reworking the exception hierarchy will obviously change the exact semantics of at least some existing code. While it is not possible to improve on the current situation without changing exact semantics, it is possible to define a narrower type of compatibility, which we will call useful compatibility.

For this we first must explain what we will call careful and _careless_exception handling. Careless (or “naïve”) code is defined as code which blindly catches any of OSError, IOError, socket.error,mmap.error, WindowsError, select.error without checking the errnoattribute. This is because such exception types are much too broad to signify anything. Any of them can be raised for error conditions as diverse as: a bad file descriptor (which will usually indicate a programming error), an unconnected socket (ditto), a socket timeout, a file type mismatch, an invalid argument, a transmission failure, insufficient permissions, a non-existent directory, a full filesystem, etc.

(moreover, the use of certain of these exceptions is irregular; Appendix B exposes the case of the select module, which raises different exceptions depending on the implementation)

Careful code is defined as code which, when catching any of the above exceptions, examines the errno attribute to determine the actual error condition and takes action depending on it.

Then we can define useful compatibility as follows:

The rationale for this compromise is that careless code can’t really be helped, but at least code which “works” won’t suddenly raise errors and crash. This is important since such code is likely to be present in scripts used as cron tasks or automated system administration programs.

Careful code, on the other hand, should not be penalized. Actually, one purpose of this PEP is to ease writing careful code.

Step 1: coalesce exception types

The first step of the resolution is to coalesce existing exception types. The following changes are proposed:

Each of these changes doesn’t preserve exact compatibility, but it does preserve useful compatibility (see “compatibility” section above).

Each of these changes can be accepted or refused individually, but of course it is considered that the greatest impact can be achieved if this first step is accepted in full. In this case, the IO exception sub-hierarchy would become:

+-- OSError (replacing IOError, WindowsError, EnvironmentError, etc.) +-- io.BlockingIOError +-- io.UnsupportedOperation (also inherits from ValueError) +-- socket.gaierror +-- socket.herror +-- socket.timeout

Justification

Not only does this first step present the user a simpler landscape as explained in the rationale section, but it also allows for a better and more complete resolution of Step 2 (see Prerequisite).

The rationale for keeping OSError as the official name for generic OS-related exceptions is that it, precisely, is more generic than IOError.EnvironmentError is more tedious to type and also much lesser-known.

The survey in Appendix B shows that IOError is the dominant error today in the standard library. As for third-party Python code, Google Code Search shows IOError being ten times more popular than EnvironmentError in user code, and three times more popular than OSError[3]. However, with no intention to deprecate IOError in the middle term, the lesser popularity of OSError is not a problem.

Exception attributes

Since WindowsError is coalesced into OSError, the latter gains a winerrorattribute under Windows. It is set to None under situations where it is not meaningful, as is already the case with the errno, filename andstrerror attributes (for example when OSError is raised directly by Python code).

Deprecation of names

The following paragraphs outline a possible deprecation strategy for old exception names. However, it has been decided to keep them as aliases for the time being. This decision could be revised in time for Python 4.0.

built-in exceptions

Deprecating the old built-in exceptions cannot be done in a straightforward fashion by intercepting all lookups in the builtins namespace, since these are performance-critical. We also cannot work at the object level, since the deprecated names will be aliased to non-deprecated objects.

A solution is to recognize these names at compilation time, and then emit a separate LOAD_OLD_GLOBAL opcode instead of the regularLOAD_GLOBAL. This specialized opcode will handle the output of a DeprecationWarning (or PendingDeprecationWarning, depending on the policy decided upon) when the name doesn’t exist in the globals namespace, but only in the builtins one. This will be enough to avoid false positives (for example if someone defines their own OSError in a module), and false negatives will be rare (for example when someone accesses OSErrorthrough the builtins module rather than directly).

module-level exceptions

The above approach cannot be used easily, since it would require special-casing some modules when compiling code objects. However, these names are by construction much less visible (they don’t appear in the builtins namespace), and lesser-known too, so we might decide to let them live in their own namespaces.

Step 2: define additional subclasses

The second step of the resolution is to extend the hierarchy by defining subclasses which will be raised, rather than their parent, for specific errno values. Which errno values is subject to discussion, but a survey of existing exception matching practices (see Appendix A) helps us propose a reasonable subset of all values. Trying to map all errno mnemonics, indeed, seems foolish, pointless, and would pollute the root namespace.

Furthermore, in a couple of cases, different errno values could raise the same exception subclass. For example, EAGAIN, EALREADY, EWOULDBLOCK and EINPROGRESS are all used to signal that an operation on a non-blocking socket would block (and therefore needs trying again later). They could therefore all raise an identical subclass and let the user examine theerrno attribute if (s)he so desires (see below “exception attributes”).

Prerequisite

Step 1 is a loose prerequisite for this.

Prerequisite, because some errnos can currently be attached to different exception classes: for example, ENOENT can be attached to both OSError and IOError, depending on the context. If we don’t want to break useful compatibility, we can’t make an except OSError (or IOError) fail to match an exception where it would succeed today.

Loose, because we could decide for a partial resolution of step 2 if existing exception classes are not coalesced: for example, ENOENT could raise a hypothetical FileNotFoundError where an IOError was previously raised, but continue to raise OSError otherwise.

The dependency on step 1 could be totally removed if the new subclasses used multiple inheritance to match with all of the existing superclasses (or, at least, OSError and IOError, which are arguable the most prevalent ones). It would, however, make the hierarchy more complicated and therefore harder to grasp for the user.

New exception classes

The following tentative list of subclasses, along with a description and the list of errnos mapped to them, is submitted to discussion:

In addition, the following exception class is proposed for inclusion:

The following drawing tries to sum up the proposed additions, along with the corresponding errno values (where applicable). The root of the sub-hierarchy (OSError, assuming Step 1 is accepted in full) is not shown:

+-- BlockingIOError EAGAIN, EALREADY, EWOULDBLOCK, EINPROGRESS +-- ChildProcessError ECHILD +-- ConnectionError +-- BrokenPipeError EPIPE, ESHUTDOWN +-- ConnectionAbortedError ECONNABORTED +-- ConnectionRefusedError ECONNREFUSED +-- ConnectionResetError ECONNRESET +-- FileExistsError EEXIST +-- FileNotFoundError ENOENT +-- InterruptedError EINTR +-- IsADirectoryError EISDIR +-- NotADirectoryError ENOTDIR +-- PermissionError EACCES, EPERM +-- ProcessLookupError ESRCH +-- TimeoutError ETIMEDOUT

Naming

Various naming controversies can arise. One of them is whether all exception class names should end in “Error”. In favour is consistency with the rest of the exception hierarchy, against is concision (especially with long names such as ConnectionAbortedError).

Exception attributes

In order to preserve useful compatibility, these subclasses should still set adequate values for the various exception attributes defined on the superclass (for example errno, filename, and optionallywinerror).

Implementation

Since it is proposed that the subclasses are raised based purely on the value of errno, little or no changes should be required in extension modules (either standard or third-party).

The first possibility is to adapt the PyErr_SetFromErrno() family of functions (PyErr_SetFromWindowsErr() under Windows) to raise the appropriate OSError subclass. This wouldn’t cover, however, Python code raising OSError directly, using the following idiom (seen inLib/tempfile.py):

raise IOError(_errno.EEXIST, "No usable temporary file name found")

A second possibility, suggested by Marc-Andre Lemburg, is to adaptOSError.__new__ to instantiate the appropriate subclass. This has the benefit of also covering Python code such as the above.

Possible objections

Namespace pollution

Making the exception hierarchy finer-grained makes the root (or builtins) namespace larger. This is to be moderated, however, as:

An alternative would be to provide a separate module containing the finer-grained exceptions, but that would defeat the purpose of encouraging careful code over careless code, since the user would first have to import the new module instead of using names already accessible.

Earlier discussion

While this is the first time such as formal proposal is made, the idea has received informal support in the past [1]; both the introduction of finer-grained exception classes and the coalescing of OSError and IOError.

The removal of WindowsError alone has been discussed and rejected as part of another PEP, but there seemed to be a consensus that the distinction with OSError wasn’t meaningful. This supports at least its aliasing with OSError.

Implementation

The reference implementation has been integrated into Python 3.3. It was formerly developed in http://hg.python.org/features/pep-3151/ in branch pep-3151, and also tracked on the bug tracker athttp://bugs.python.org/issue12555. It has been successfully tested on a variety of systems: Linux, Windows, OpenIndiana and FreeBSD buildbots.

One source of trouble has been with the respective constructors of OSErrorand WindowsError, which were incompatible. The way it is solved is by keeping the OSError signature and adding a fourth optional argument to allow passing the Windows error code (which is different from the POSIX errno). The fourth argument is stored as winerror and its POSIX translation as errno. The PyErr_SetFromWindowsErr* functions have been adapted to use the right constructor call.

A slight complication is when the PyErr_SetExcFromWindowsErr* functions are called with OSError rather than WindowsError: the errnoattribute of the exception object would store the Windows error code (such as 109 for ERROR_BROKEN_PIPE) rather than its POSIX translation (such as 32 for EPIPE), which it does now. For non-socket error codes, this only occurs in the private _multiprocessing module for which there is no compatibility concern.

Note

For socket errors, the “POSIX errno” as reflected by the errno module is numerically equal to the Windows Socket error codereturned by the WSAGetLastError system call:

errno.EWOULDBLOCK 10035 errno.WSAEWOULDBLOCK 10035

Possible alternative

Pattern matching

Another possibility would be to introduce an advanced pattern matching syntax when catching exceptions. For example:

try: os.remove(filename) except OSError as e if e.errno == errno.ENOENT: pass

Several problems with this proposal:

Exceptions ignored by this PEP

This PEP ignores EOFError, which signals a truncated input stream in various protocol and file format implementations (for example GzipFile).EOFError is not OS- or IO-related, it is a logical error raised at a higher level.

This PEP also ignores SSLError, which is raised by the ssl module in order to propagate errors signalled by the OpenSSL library. Ideally,SSLError would benefit from a similar but separate treatment since it defines its own constants for error types (ssl.SSL_ERROR_WANT_READ, etc.). In Python 3.2, SSLError is already replaced with socket.timeoutwhen it signals a socket timeout (see issue 10272).

Endly, the fate of socket.gaierror and socket.herror is not settled. While they would deserve less cryptic names, this can be handled separately from the exception hierarchy reorganization effort.

Appendix A: Survey of common errnos

This is a quick inventory of the various errno mnemonics checked for in the standard library and its tests, as part of except clauses.

Common errnos with OSError

Common errnos with IOError

Common errnos with socket.error

All these errors may also be associated with a plain IOError, for example when calling read() on a socket’s file descriptor.

Common errnos with select.error

Appendix B: Survey of raised OS and IO errors

About VMSError

VMSError is completely unused by the interpreter core and the standard library. It was added as part of the OpenVMS patches submitted in 2002 by Jean-François Piéronne [4]; the motivation for including VMSError was that it could be raised by third-party packages.

Interpreter core

Handling of PYTHONSTARTUP raises IOError (but the error gets discarded):

$ PYTHONSTARTUP=foox ./python Python 3.2a0 (py3k:82920M, Jul 16 2010, 22:53:23) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. Could not open PYTHONSTARTUP IOError: [Errno 2] No such file or directory: 'foox'

PyObject_Print() raises IOError when ferror() signals an error on theFILE * parameter (which, in the source tree, is always either stdout or stderr).

Unicode encoding and decoding using the mbcs encoding can raise WindowsError for some error conditions.

Standard library

bz2

Raises IOError throughout (OSError is unused):

bz2.BZ2File("foox", "rb") Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory bz2.BZ2File("LICENSE", "rb").read() Traceback (most recent call last): File "", line 1, in IOError: invalid data stream bz2.BZ2File("/tmp/zzz.bz2", "wb").read() Traceback (most recent call last): File "", line 1, in IOError: file is not ready for reading

curses

Not examined.

dbm.gnu, dbm.ndbm

_dbm.error and _gdbm.error inherit from IOError:

dbm.gnu.open("foox") Traceback (most recent call last): File "", line 1, in _gdbm.error: [Errno 2] No such file or directory

fcntl

Raises IOError throughout (OSError is unused).

imp module

Raises IOError for bad file descriptors:

imp.load_source("foo", "foo", 123) Traceback (most recent call last): File "", line 1, in IOError: [Errno 9] Bad file descriptor

io module

Raises IOError when trying to open a directory under Unix:

open("Python/", "r") Traceback (most recent call last): File "", line 1, in IOError: [Errno 21] Is a directory: 'Python/'

Raises IOError or io.UnsupportedOperation (which inherits from the former) for unsupported operations:

open("LICENSE").write("bar") Traceback (most recent call last): File "", line 1, in IOError: not writable io.StringIO().fileno() Traceback (most recent call last): File "", line 1, in io.UnsupportedOperation: fileno open("LICENSE").seek(1, 1) Traceback (most recent call last): File "", line 1, in IOError: can't do nonzero cur-relative seeks

Raises either IOError or TypeError when the inferior I/O layer misbehaves (i.e. violates the API it is expected to implement).

Raises IOError when the underlying OS resource becomes invalid:

f = open("LICENSE") os.close(f.fileno()) f.read() Traceback (most recent call last): File "", line 1, in IOError: [Errno 9] Bad file descriptor

…or for implementation-specific optimizations:

f = open("LICENSE") next(f) 'A. HISTORY OF THE SOFTWARE\n' f.tell() Traceback (most recent call last): File "", line 1, in IOError: telling position disabled by next() call

Raises BlockingIOError (inheriting from IOError) when a call on a non-blocking object would block.

mmap

Under Unix, raises its own mmap.error (inheriting from EnvironmentError) throughout:

mmap.mmap(123, 10) Traceback (most recent call last): File "", line 1, in mmap.error: [Errno 9] Bad file descriptor mmap.mmap(os.open("/tmp", os.O_RDONLY), 10) Traceback (most recent call last): File "", line 1, in mmap.error: [Errno 13] Permission denied

Under Windows, however, it mostly raises WindowsError (the source code also shows a few occurrences of mmap.error):

fd = os.open("LICENSE", os.O_RDONLY) m = mmap.mmap(fd, 16384) Traceback (most recent call last): File "", line 1, in WindowsError: [Error 5] Accès refusé sys.last_value.errno 13 errno.errorcode[13] 'EACCES'

m = mmap.mmap(-1, 4096) m.resize(16384) Traceback (most recent call last): File "", line 1, in WindowsError: [Error 87] Paramètre incorrect sys.last_value.errno 22 errno.errorcode[22] 'EINVAL'

multiprocessing

Not examined.

os / posix

The os (or posix) module raises OSError throughout, except under Windows where WindowsError can be raised instead.

ossaudiodev

Raises IOError throughout (OSError is unused):

ossaudiodev.open("foo", "r") Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory: 'foo'

readline

Raises IOError in various file-handling functions:

readline.read_history_file("foo") Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory readline.read_init_file("foo") Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory readline.write_history_file("/dev/nonexistent") Traceback (most recent call last): File "", line 1, in IOError: [Errno 13] Permission denied

select

As a side-note, not deriving from EnvironmentError means select.errordoes not get the useful errno attribute. User code must check args[0]instead:

signal.alarm(1); select.select([], [], []) 0 Traceback (most recent call last): File "", line 1, in select.error: (4, 'Interrupted system call') e = sys.last_value e error(4, 'Interrupted system call') e.errno == errno.EINTR Traceback (most recent call last): File "", line 1, in AttributeError: 'error' object has no attribute 'errno' e.args[0] == errno.EINTR True

signal

signal.ItimerError inherits from IOError.

socket

socket.error inherits from IOError.

sys

sys.getwindowsversion() raises WindowsError with a bogus error number if the GetVersionEx() call fails.

time

Raises IOError for internal errors in time.time() and time.sleep().

zipimport

zipimporter.get_data() can raise IOError.

Acknowledgments

Significant input has been received from Alyssa Coghlan.

References

This document has been placed in the public domain.