[Python-Dev] Issue #8610: Set default file system encoding to ascii on error? (original) (raw)

Victor Stinner victor.stinner at haypocalc.com
Fri May 7 13:05:52 CEST 2010

Python 3.0 introduced PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to UTF-8 if getting the file system encoding failed or the encoding is unknown (there is not nl_langinfo(CODESET) function).

UTF-8 is not a good choice for the fallback because it's incompatible with other encodings like Latin1. I would like to fallback to ASCII on error which is compatible with all encodings (thanks to surrogateescape).

I would like to ensure that sys.getfilesystemencoding() result cannot be None, because Python 3.2 uses it on Unix to decode os.environb and to encode filenames in subprocess. I can implement a fallback for os.environb and subprocess (and other functions calling sys.getfilesystemencoding()), but I prefer to have a reliable sys.getfilesystemencoding() function.

This change doesn't concern Windows and Mac OS X because the encoding is hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". I think that few (or no) user would notice this change.

-- Victor Stinner http://www.haypocalc.com/

More information about the Python-Dev mailing list