Issue 12451: open: avoid the locale encoding when possible (original) (raw)
Created on 2011-06-30 12:20 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (15)
Author: STINNER Victor (vstinner) * 
Date: 2011-06-30 12:20
open() uses the locale encoding in Python 3 when opening text file if the encoding argument is not specified (implicit). Some functions use locale encoding, but it's not the right encoding. I see at least three cases where the encoding should be changed:
- UTF-8 should be used instead for portability: it's a bug in the module
- ASCII must be used instead: the module doesn't support non-ASCII characters (old file formats, old network protocols, some fields of a document, etc.)
- ASCII can be used instead: it's just a micro-optimization, the ASCII encoding is a little bit faster
To detect the usage of the implicit locale encoding, some functions can be monkeypatched:
- builtins.open, io.open, _pyio.open
- io.TextIOWrapper, _pyio.TextIOWrapper
- more functions using directly or indirectly open/TextIOWrapper may be patched to emit the warning earlier
Attached open_hook.patch implements these hooks (hacks?) in the site module: it emits a ResourceWarning. Use python -Werror to raise an error if the locale encoding is used implicitly. If you really want to use the locale encoding, use encoding='locale' to make quiet the warning.
Quite all functions in Python uses the implicit locale encoding. For example, Python doesn't start with the patch and -Werror. If you use -Werror, you have to patch all calls to open()/TextIOWrapper to be able to locate real bugs, or the program will stop before hitting the real problems. Each time you have to check what is the real expected encoding, it takes a lot of time.
I started this huge project. I'm using ASCII most of the time (especially in Python tests), I don't know if it's correct. It will require a second step to ensure that the function really don't use/support non-ASCII characters.
I will use this issue for my commits, attach patches, and more generally discuss this topic.
Author: Roundup Robot (python-dev) 
Date: 2011-06-30 13:42
New changeset bd73edea78dc by Victor Stinner in branch '3.2': Issue #12451: distutils now opens the setup script in binary mode to read the http://hg.python.org/cpython/rev/bd73edea78dc
New changeset 8a7fd54cba01 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: distutils now opens the setup script in binary mode http://hg.python.org/cpython/rev/8a7fd54cba01
Author: Roundup Robot (python-dev)
Date: 2011-06-30 14:00
New changeset 1942f7c8f51c by Victor Stinner in branch '3.2': Issue #12451: pydoc.synopsis() now reads the encoding cookie if available, to http://hg.python.org/cpython/rev/1942f7c8f51c
New changeset 3e627877b5a9 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: pydoc.synopsis() now reads the encoding cookie if http://hg.python.org/cpython/rev/3e627877b5a9
Author: Roundup Robot (python-dev)
Date: 2011-06-30 15:39
New changeset bafc5c7d24b2 by Victor Stinner in branch '3.2': Issue #12451: doctest.debug_script() doesn't create a temporary file anymore to http://hg.python.org/cpython/rev/bafc5c7d24b2
New changeset 77c589b27e90 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: doctest.debug_script() doesn't create a temporary http://hg.python.org/cpython/rev/77c589b27e90
Author: STINNER Victor (vstinner) *
Date: 2011-06-30 15:41
See also issue #9561 for distutils: I just attached a new patch for PKG-INFO / .egg-info files.
Author: Roundup Robot (python-dev)
Date: 2011-06-30 16:11
New changeset 45e3dafb3dbe by Victor Stinner in branch '3.2': Issue #12451: The XInclude default loader of xml.etree now decodes files from http://hg.python.org/cpython/rev/45e3dafb3dbe
New changeset e8eea84a90dc by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: The XInclude default loader of xml.etree now decodes http://hg.python.org/cpython/rev/e8eea84a90dc
Author: Roundup Robot (python-dev)
Date: 2011-06-30 16:21
New changeset 68bc1a29ba5a by Victor Stinner in branch '3.2': Issue #12451: Open files in binary mode in some tests when the text file is not http://hg.python.org/cpython/rev/68bc1a29ba5a
New changeset 3969b6377f52 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: Open files in binary mode in some tests when the text http://hg.python.org/cpython/rev/3969b6377f52
Author: Roundup Robot (python-dev)
Date: 2011-06-30 16:25
New changeset c4388478f9b2 by Victor Stinner in branch 'default': Issue #12451: Open the test file in binary mode in test_bz2, the text file is http://hg.python.org/cpython/rev/c4388478f9b2
Author: STINNER Victor (vstinner) *
Date: 2011-06-30 16:46
See also issue #12454 for the mailbox module (.mh_sequences).
Author: Roundup Robot (python-dev)
Date: 2011-06-30 21:26
New changeset 0c49260e85a0 by Victor Stinner in branch 'default': Issue #12451: Add support.create_empty_file() http://hg.python.org/cpython/rev/0c49260e85a0
Author: Roundup Robot (python-dev)
Date: 2011-07-03 23:29
New changeset 81424281ee59 by Victor Stinner in branch '3.2': Issue #12451: xml.dom.pulldom: parse() now opens files in binary mode instead http://hg.python.org/cpython/rev/81424281ee59
New changeset c039c6b58907 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: xml.dom.pulldom: parse() now opens files in binary http://hg.python.org/cpython/rev/c039c6b58907
Author: Roundup Robot (python-dev)
Date: 2011-07-03 23:47
New changeset cd1759711357 by Victor Stinner in branch '3.2': Issue #12451: runpy: run_path() now opens the Python script in binary mode, http://hg.python.org/cpython/rev/cd1759711357
New changeset e240af1f0ae1 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: runpy: run_path() now opens the Python script in http://hg.python.org/cpython/rev/e240af1f0ae1
Author: Roundup Robot (python-dev)
Date: 2011-07-04 00:14
New changeset a1b4f1716b73 by Victor Stinner in branch '3.2': Issue #12451: pydoc: importfile() now opens the Python script in binary mode, http://hg.python.org/cpython/rev/a1b4f1716b73
New changeset 5ca136dccbf7 by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: pydoc: importfile() now opens the Python script in http://hg.python.org/cpython/rev/5ca136dccbf7
Author: Roundup Robot (python-dev)
Date: 2011-07-05 12:32
New changeset 8b62f5d722f4 by Victor Stinner in branch '3.2': Issue #12451: pydoc: html_getfile() now uses tokenize.open() to support Python http://hg.python.org/cpython/rev/8b62f5d722f4
New changeset 2fbfb7ea362f by Victor Stinner in branch 'default': (merge 3.2) Issue #12451: pydoc: html_getfile() now uses tokenize.open() to http://hg.python.org/cpython/rev/2fbfb7ea362f
Author: STINNER Victor (vstinner) *
Date: 2011-10-17 18:45
Ok, it should be enough :-)
History
Date
User
Action
Args
2022-04-11 14:57:19
admin
set
github: 56660
2011-10-17 18:45:19
vstinner
set
status: open -> closed
resolution: fixed
messages: +
2011-07-05 12:32:01
python-dev
set
messages: +
2011-07-04 00:14:57
python-dev
set
messages: +
2011-07-03 23:47:59
python-dev
set
messages: +
2011-07-03 23:29:13
python-dev
set
messages: +
2011-06-30 21:26:00
python-dev
set
messages: +
2011-06-30 16:51:16
Arfrever
set
nosy: + Arfrever
2011-06-30 16:46:58
vstinner
set
messages: +
2011-06-30 16:25:28
python-dev
set
messages: +
2011-06-30 16:21:45
python-dev
set
messages: +
2011-06-30 16:11:26
python-dev
set
messages: +
2011-06-30 15:41:25
vstinner
set
messages: +
2011-06-30 15:39:42
python-dev
set
messages: +
2011-06-30 14:00:04
python-dev
set
messages: +
2011-06-30 13:42:02
python-dev
set
nosy: + python-dev
messages: +
2011-06-30 12:20:50
vstinner
create