[Python-Dev] Printing and unicode (original) (raw)

M.-A. Lemburg mal@lemburg.com
Wed, 13 Nov 2002 21:58:33 +0100

Guido van Rossum wrote:

Though it acts somewhat as if its default encoding is "ascii". This is somewhat inconsistent: you can write arbitrary Unicode strings, but the Unicode won't be converted to ASCII. ASCII is converted to Unicode though.

It is the only case of a "pure Unicode" stream in Python, where the underlying "native" sequence is not one of bytes, but one of Unicode characters. The real problem is that the "orientation" (wide or narrow strings) is determined by the things written into the stream. It might be more reasonable to have StringIO.ByteIO and StringIO.UnicodeIO constructors, which both accept an encoding= argument, and will convert objects of the wrong "orientation" using that encoding (defaulting to the system encoding). I'm not sure about those names, but I agree that the encoding should be forced when the StringIO instance is created. Given that using Unicode with these is currently fragile at best, maybe we should say that unless you give an encoding argument, it's a byte stream and doesn't allow Unicode at all? That would be consistent with cStringIO.

+1

The fact that StringIO works with Unicode (and then only in the case where you only pass Unicode to it) is more an implementation detail than a true feature. cStringIO doesn't have this implementation detail, so porting from StringIO to the much faster cStringIO doesn't work at all for Unicode.

I think that StringIO and cStringIO should be regarded as binary streams without any encoding knowledge. It is easy enough to wrap these into Unicode aware streams using the codecs.StreamReaderWriter class as is done in codecs.open().

That API already adds the .encoding attribute to the stream object, BTW.

-- Marc-Andre Lemburg CEO eGenix.com Software GmbH

eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/