[Python-Dev] Using environment variables problematic for embedded python. (original) (raw)

Campbell Barton ideasman42 at gmail.com
Thu Oct 4 06:35:02 CEST 2012

On Thu, Oct 4, 2012 at 1:35 PM, Campbell Barton <ideasman42 at gmail.com> wrote:

Hi,

We've run into an issue recently with blender3d on ms-windows where we want to enforce the encoding is UTF-8 with the embedded python interpreter. (the encoding defaults to cp437). I naively thought setting the environment variable before calling PyInitialize() would work, but the way python DLL loads, it gets its own environment variables that cant be modified directly [1]. eg, putenv("PYTHONIOENCODING=utf-8:surrogateescape"); We had bug reports by windows users not able to export files because the stdout errors on printing paths with unsupported encoding. [2],[3] --- Of course we could distribute blender with a bat file launcher that sets env variables, or ask the user to set their env variable - but I dont think this is really a good option. I tried overriding the stderr & stdout, but this caused another bug in a part of out code that catches exception messages from the stderr. [4] import sys, io sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='surrogateescape', linebuffering=True) sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', errors='surrogateescape', linebuffering=True)

IMHO either of these solutions would be fine. * have a PyOSPutEnv() function, gettext has gettextputenv() to workaround this problem. * manage this the same as PyGetPythonHome(), which can be defined by the embedding application to override the default. Id like to know if there is some known solution to workaround this issue, if not - would either of these would be acceptable in python (can write up a patch if it helps) Regards, Campbell --- [1] http://stackoverflow.com/questions/5153547/environment-variables-are-different-for-dll-than-exe [2] http://projects.blender.org/tracker/index.php?func=detail&aid=32750 [3] http://projects.blender.org/tracker/index.php?func=detail&aid=31555 [4] http://projects.blender.org/tracker/?func=detail&aid=32720

To follow up and give a correction to overwriting sys.stdout/stderr, The issue seemed to be that stderr/stdout was later overwritten, loosing the original reference to the buffer (maybe refcount issue here?), either way it would silence the output.

import sys, io sys.stdout = sys.stdout = io.TextIOWrapper(io.open(sys.stdout.fileno(), "wb", -1), encoding='utf-8', errors='surrogateescape', newline="\n", line_buffering=True) sys.stderr = sys.stderr = io.TextIOWrapper(io.open(sys.stderr.fileno(), "wb", -1), encoding='utf-8', errors='surrogateescape', newline="\n", line_buffering=True)

This all works as expected without bug [4] (above), however on exit I get an assert in MSVCR90.DLL's write.c (called from python32_d.dll): _VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

I'd rather not loose more time debugging why this assert happens, IMHO this is too low-level a way to change the encoing of stdio/stderr and error-prone too, so some way to reliably set PYTHONIOENCODING from a program embedding python is still needed.

--

More information about the Python-Dev mailing list