[Python-Dev] Using environment variables problematic for embedded python. (original) (raw)

Campbell Barton ideasman42 at gmail.com
Thu Oct 4 15:01:49 CEST 2012

Checked Python 3.4 (latest hg checkout) and this area of the code hasn't changed: pythonrun,c initstdio()

reported the issue: http://bugs.python.org/issue16129

On Thu, Oct 4, 2012 at 2:51 PM, Guido van Rossum <guido at python.org> wrote:

This seems more fit for the tracker; can you file there? (Then post the issue link here.) I do think you have a legitimate use case to set the default encoding to utf-8. (Though there may be a way already.) Does Python 3.3 have te same bug?

On Wednesday, October 3, 2012, Campbell Barton wrote:

On Thu, Oct 4, 2012 at 1:35 PM, Campbell Barton <ideasman42 at gmail.com> wrote: > Hi, > > We've run into an issue recently with blender3d on ms-windows where we > want to enforce the encoding is UTF-8 with the embedded python > interpreter. > (the encoding defaults to cp437). > > I naively thought setting the environment variable before calling > PyInitialize() would work, but the way python DLL loads, it gets its > own environment variables that cant be modified directly [1]. > eg, putenv("PYTHONIOENCODING=utf-8:surrogateescape"); > > We had bug reports by windows users not able to export files because > the stdout errors on printing paths with unsupported encoding. [2],[3] > > --- > > Of course we could distribute blender with a bat file launcher that > sets env variables, or ask the user to set their env variable - but I > dont think this is really a good option. > > I tried overriding the stderr & stdout, but this caused another bug in > a part of out code that catches exception messages from the stderr. > [4] > import sys, io > sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', > errors='surrogateescape', linebuffering=True) > sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', > errors='surrogateescape', linebuffering=True) > > > > IMHO either of these solutions would be fine. > > * have a PyOSPutEnv() function, gettext has gettextputenv() to > workaround this problem. > > * manage this the same as PyGetPythonHome(), which can be defined by > the embedding application to override the default. > > > Id like to know if there is some known solution to workaround this > issue, if not - would either of these would be acceptable in python > (can write up a patch if it helps) > > Regards, > Campbell > > --- > > [1] > http://stackoverflow.com/questions/5153547/environment-variables-are-different-for-dll-than-exe > [2] http://projects.blender.org/tracker/index.php?func=detail&aid=32750 > [3] http://projects.blender.org/tracker/index.php?func=detail&aid=31555 > [4] http://projects.blender.org/tracker/?func=detail&aid=32720 To follow up and give a correction to overwriting sys.stdout/stderr, The issue seemed to be that stderr/stdout was later overwritten, loosing the original reference to the buffer (maybe refcount issue here?), either way it would silence the output.

import sys, io sys.stdout = sys.stdout = io.TextIOWrapper(io.open(sys.stdout.fileno(), "wb", -1), encoding='utf-8', errors='surrogateescape', newline="\n", linebuffering=True) sys.stderr = sys.stderr = io.TextIOWrapper(io.open(sys.stderr.fileno(), "wb", -1), encoding='utf-8', errors='surrogateescape', newline="\n", linebuffering=True) This all works as expected without bug [4] (above), however on exit I get an assert in MSVCR90.DLL's write.c (called from python32d.dll): VALIDATECLEAROSSERRRETURN((osfile(fh) & FOPEN), EBADF, -1); I'd rather not loose more time debugging why this assert happens, IMHO this is too low-level a way to change the encoing of stdio/stderr and error-prone too, so some way to reliably set PYTHONIOENCODING from a program embedding python is still needed. -- - Campbell

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

--

More information about the Python-Dev mailing list