[Python-Dev] eval and triple quoted strings (original) (raw)

Benjamin Peterson benjamin at python.org
Tue Jun 18 02:49:10 CEST 2013

2013/6/17 Guido van Rossum <guido at python.org>:

On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/6/17 Guido van Rossum <guido at python.org>:

On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote: What exactly does the parser handles better than the io module? Is it just the coding cookies? I suppose that works as long as the file is encoded using as ASCII superset like the Latin-N variants or UTF-8. It would fail pretty badly if it was UTF-16 (and yes, that's an abominable encoding for other reasons :-).

The coding cookie is the main one. In fact, if you can't parse that, you don't really know what encoding to open the file with at all. There's also small things like BOM handling (you have to use the utf-16-sig encoding with TextIO to get it removed) and defaulting to UTF-8 (which the io module doesn't do) which is better left to the parser. Maybe there are some lessons here that the TextIO module could learn?

UTF-8 by default would be great, but that ship has sailed.

Reading Python coding cookies is outside the purview of TextIOWrapper. However, it would be good to have a function in the stdlib to read a python source file to Unicode; I've definitely implemented that several times.

-- Regards, Benjamin

More information about the Python-Dev mailing list