[Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation (original) (raw)

Victor Stinner vstinner at redhat.com
Wed Sep 19 08:07:30 EDT 2018

IMHO the implementation is really a secondary concern here, the main question is: what is the correct behavior?

Nick: * Do we agree that we need to provide a way to disable C locale coercion (PEP 538) even when -E is used? * Do you agree that PyInitialize() and PyMain() must not enable the C locale coercion (PEP 538)? I understood that your reply is yes for the second question, since you insist to push your change which also prevent PyInitialize() and PyMain() to enable C locale coercion.

Hum, I'm not sure if I explained properly my opinion on these questions.

I consider that Python 3.7.0 introduced a regression compared to Python 3.6: it changes the LC_CTYPE locale for Python and all child processes and it's not possible to opt-out for that when using -E command line option. I proposed (and implemented) -X coerce_c_locale=0 for that. Unicode and locales are so hard to get right that I consider that it's important that we provide an option to opt-out,. Otherwise, someone will find an use case where Python 3.7 doesn't behave as expected and break one specific use case. I didn't notice a complain yet, but there are very few Python 3.7 users at this point. For example, very few Linux distributions use it yet.

I consider that PYTHONCOERCECLOCALE must not introduce an exception in -E: it must be ignored when -E or -I is used. For security reasons, it's important to really ignore all PYTHON* environment variables. "Unicode" (in general) has been abused in the past to exploit vulnerabilities in applications. Locales and encodings are so hard, that it's easy to mess up and introduce a vulnerability just caused by encodings. It's also important to get deterministic and reproducible programs.

For Py_Initialize() and Py_Main(): I have no opinion, so I rely on Nick's request to make sure that the C locale is not coerced when Python is embeded :-)

Victor

More information about the Python-Dev mailing list