[Python-ideas] Replacing the standard IO streams (was Re: changing sys.stdout encoding) (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sun Jun 10 17:44:08 CEST 2012

On Mon, Jun 11, 2012 at 12:34 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:

On 10.06.12 00:22, Mark Lawrence wrote:

On 09/06/2012 21:02, Serhiy Storchaka wrote:

None of these methods are not guaranteed to work if the input or output have occurred before. That's a double negative so I'm not sure what you meant to say. Can you please rephrase it. I assume that English is not your native language, so I'll let you off :) open(sys.stdin.fileno()) is not guaranteed to work if the input or output have occurred before. And io.TextIOWrapper(sys.stdin.detach()) is not guaranteed to work if the input or output have occurred before. sys.stdin internal buffer can contains read by not used characters. sys.stdin.buffer internal buffer can contains read by not used bytes. With multibyte encoding sys.stdin.decoder internal buffer can contains uncompleted multibyte character.

Right, but the point of this discussion is to document the cleanest available way for an application to change these settings at application start (e.g. to support an "--encoding" parameter). Yes, there are potential issues if you use any of these mechanisms while there is data in the buffers, but that's a much harder problem and not one we're trying to solve here.

Regardless, the advantage of the "open + fileno" idiom is that it works for any level of change. If you want to force your streams to unbuffered binary IO rather than merely changing the encoding:

sys.stdin = open(sys.stdin.fileno(), 'rb', buffering=0, closefd=False)
sys.stdout = open(sys.stdout.fileno(), 'wb', buffering=0, closefd=False)
sys.stderr = open(sys.stderr.fileno(), 'wb', buffering=0, closefd=False)

Keep them as text, but force them to permissive utf-8, no matter how the interpreter originally created them?:

sys.stdin = open(sys.stdin.fileno(), 'r', encoding="utf-8",

errors="surrogateescape", closefd=False) sys.stdout = open(sys.stdout.fileno(), 'w', encoding="utf-8", errors="surrogateescape", closefd=False) sys.stderr = open(sys.stderr.fileno(), 'w', encoding="utf-8", errors="surrogateescape", closefd=False)

This approach also has the advantage of leaving sys.std(in/out/err) in a somewhat usable state.

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-ideas mailing list