[Python-Dev] Python in Unicode context (original) (raw)

M.-A. Lemburg mal at egenix.com
Thu Aug 5 11:00:22 CEST 2004

[Source code encoding and string literals]

I hope that my explanation above helps at seeing that source encoding and choice of string literals are not as independent as one may think. A choice that I surely do not have is to see bugs appear in programs merely because I changed the source encoding. Going from ISO 8859-1 to ISO 8859-15 for a Python source is probably fairly safe, because there is no need for switching the narrowness of strings. Going from ISO 8859-1 to UTF-8 is very unsafe, and editing all literal strings from narrow to wide, using `u' prefixes, becomes almost unavoidable.

Indeed. As always: explicit is better than implicit :-) The small "u" in front of the literal will tell all readers: this is Unicode text.

We might introduce more ways to switch string literal interpretation depending on module or interpreter process scope. However, the small u is here to stay and it's available now, so why not use it ?

Your concerns about programs breaking because of changes to the source encoding are valid, but not something that Python can address. You have the same problem with normal text documents: a spell checker might find wrong spellings of a word as a result of using a wrong encoding, but it is not fool proof and things get worse if you have multiple languages embedded in your program code.

As general advice for writing i18n compliant programs, I can only suggest to keep programs written using a single source encoding and language that appeals to the programmer and place all string literals under gettext or similar tool control.

I usually write programs in ASCII or Latin-1 and use English to write the string literals which then get mapped to user languages as necessary by means of gettext or custom translation logic.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Aug 05 2004)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list