[Python-Dev] PEP 383 update: utf8b is now the error handler (original) (raw)

Stephen J. Turnbull stephen at xemacs.org
Tue May 5 18:32:03 CEST 2009

Zooko O'Whielacronx writes:

How would an application make sure that they were producing only valid unicode?

That's very difficult. There are a couple of sources that I can think of, in Python: C modules, chr(), \u literals, and now codecs with the 'utf8b'. There may be others. You'd need to review your own code for all of them very carefully, and you'd have to validate all strings returned by non-validating APIs (which is all of them in Python now, although many of them can probably be trusted, such as codecs not using the 'utf8b' error handler).

How about add an option to os.listdir() named "errors" with default value 'utf8b'

Seems reasonable to me, but Martin's probably thought more carefully about it. I don't think its applicable to your use case, though, because you want to be able to access those files as well as display the names to the users, right? You won't be able to access those files if you receive the names already munged by the error handler.

More information about the Python-Dev mailing list