[Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation (original) (raw)

Ned Deily nad at python.org
Mon Sep 17 21:42:41 EDT 2018

On Sep 17, 2018, at 21:20, Victor Stinner <vstinner at redhat.com> wrote:

tl; dr Nick, Ned, INADA-san: I modified 3.7.1 to add a new "-X coerceclocale=value" option and make sure that the C locale coercion cannot be when Python in embedded: are you ok with these changes?

Before 3.7.0 release, during the implementation of the UTF-8 Mode (PEP 540), I changed two things in Nick Coghlan's implementation of the C locale coercion (PEP 538): (1) PYTHONCOERCECLOCALE environment variable is now ignored when -E or -I command line option is used. (2) When Python is embeded, the C locale coercion is now enabled if the LCCTYPE locale is "C". Nick asked me to change the behavior: https://bugs.python.org/issue34589 I just pushed this change in the 3.7 branch which adds a new "-X coerceclocale=value" option: https://github.com/python/cpython/commit/144f1e2c6f4a24bd288c045986842c65cc289684 Examples using Pyhon 3.7 (future 3.7.1) with UTF-8 Mode disabled, to only test the C locale coercion: --- $ cat test.py import codecs, locale enc = locale.getpreferredencoding() enc = codecs.lookup(enc).name print(enc) $ export LCALL= LCCTYPE=C LANG= # Disable C locale coercion: get ASCII as expected $ PYTHONCOERCECLOCALE=0 ./python -X utf8=0 test.py ascii # -E ignores PYTHONCOERCECLOCALE=0: # C locale is coerced, we get UTF-8 $ PYTHONCOERCECLOCALE=0 ./python -E -X utf8=0 test.py utf-8 # -X coerceclocale=0 is not affected by -E: # C locale coercion disabled as expected, get ASCII as expected $ ./python -E -X utf8=0 -X coerceclocale=0 test.py ascii --- For (1), Nick's use case is to get Python 3.6 behavior (C locale not coerced) on Python 3.7 using PYTHONCOERCECLOCALE. Nick proposed to use PYTHONCOERCECLOCALE even with -E or -I, but I dislike introducing a special case for -E option. I chose to add a new "-X coerceclocale=0" to Python 3.7.1 to provide a solution for this use case. (Python 3.7.0 and older ignore this option.) Note: Python 3.7.0 is fine with PYTHONCOERCECLOCALE=0, we are only talking about the special case of -E and -I options. For (2), I modified Python 3.7.1 to make sure the C locale is never coerced when the C API is used to embed Python inside an application: PyInitialize() and PyMain(). The C locale can only be coerced by the official Python program ("python3.7"). I don't know if it should be possible to enable C locale coercion when Python is embedded. So I just made the change requested by Nick :-) I dislike doing such late changes in 3.7.1, especially since PEP 538 has been designed by Nick Coghlan, and we disagree on the fix. But Ned Deily, our Python 3.7 release manager, wants to see last 3.7 fixes merged before Tuesday, so here we are.

Just because the 3.7.1rc is scheduled doesn't mean we should throw something in, particularly if it's not fully reviewed and fully agreed upon. If it's important enough, we could delay the rc a few days ... or decide to wait for 3.7.2.

Nick, Ned, INADA-san: are you ok with these changes? The other choices for 3.7.1 are:

* Revert my change: C locale coercion can still be enabled when Python is embedded, -E option ignores PYTHONCOERCECLOCALE env var. * Revert my change and apply Nick's PR 9257: C locale coercion cannot be enabled when Python is embedded and -E option doesn't ignore PYTHONCOERCECLOCALE env var.

I spent months to fix the master branch to support all possible locales and encodings, and get a consistent CLI: https://vstinner.github.io/python3-locales-encodings.html So I'm not excited by Nick's PR which IMHO moves Python backward, especially it breaks the -E option contract: it doesn't ignore PYTHONCOERCECLOCALE env var.

I would like to see Nick review the merged 3.7 PR and have both him and you agree that this is the thing to do for 3.7.1. I also want to make sure we understand what affect this will have on 3.7.0 users. Let's not potentially make things worse.

I'm not planning to tag 3.7.1rc for at least another 18 hours. I'm marking bpo-34589 as "release blocker" and I will not proceed until this is resolved.

Thanks! --Ned

-- Ned Deily nad at python.org -- []

More information about the Python-Dev mailing list