[Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Tue May 9 07:06:58 EDT 2017

On 9 May 2017 at 13:44, Nick Coghlan <ncoghlan at gmail.com> wrote:

On 8 May 2017 at 15:34, Nick Coghlan <ncoghlan at gmail.com> wrote:

On 7 May 2017 at 15:22, INADA Naoki <songofacandy at gmail.com> wrote:

## Background

Locale coercion in current PEP 538 has some downsides: * If user set LANG=C LCDATE=jaJP.UTF-8, locale coercion may overrides LCDATE. The fact it sets "LCALL" has previously been raised as a concern with PEP 538, so it probably makes sense to drop that aspect and just override "LANG". The scenarios where it makes a difference are incredibly obscure (involving non-default SSH locale forwarding settings for folks using SSH on Mac OS X to connect to remote Linux systems), while just setting "LANG" will be sufficient to address the "LANG=C" case that is the main driver for the PEP. It occurs to me we can even still handle the forwarded "LCCTYPE=UTF-8" case by changing the locale coercion to set LCCTYPE & LANG, rather than just setting LANG as I suggested above. That way LANG=C LCDATE=jaJP.UTF-8 would still respect the explicit LCDATE setting, LCCTYPE=C would be handled the same way as LANG=C, and LCALL=C would continue to provide a way to force the C locale even for LCCTYPE without needing to be aware of the Python specific PYTHONCOERCECLOCALE setting.

I've posted an updated reference implementation that works this way, and it turned out to have some rather nice benefits: not only did it make the handling of full locales (C.UTF-8, C.utf8) and partial locales (UTF-8) more consistent (allowing for a net deletion of code), it also meant I no longer needed a custom test case in _testembed to check the locale warning. Instead, the affected test cases now just set "LC_ALL" as a locale override that switches off CPython's locale coercion without also switching off the locale warning.

Code changes: https://github.com/ncoghlan/cpython/commit/476a78133c94d82e19b89f50036cecd9b4214e7a

Rather than posting the PEP updates here though, I'll start a new thread that explains what has changed since my initial posting to python-dev back in March.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list