[Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Tue Sep 28 13:19:18 CEST 2010

On Mon, 27 Sep 2010 23:45:45 -0400 Steve Holden <steve at holdenweb.com> wrote:

On 9/27/2010 11:27 PM, Benjamin Peterson wrote: > 2010/9/27 Meador Inge <meadori at gmail.com>: >> which, as seen in the trace, is because the 'detectencoding' function in >> 'Lib/tokenize.py' searches for 'BOMUTF8' (a 'bytes' object) in the string >> to tokenize 'first' (a 'str' object). It seems to me that strings should >> still be able to be tokenized, but maybe I am missing something. >> Is the implementation of 'detectencoding' correct in how it attempts to >> determine an encoding or should I open an issue for this? > > Tokenize only works on bytes. You can open a feature request if you desire. > Working only on bytes does seem rather perverse.

I agree, the morality of bytes objects could have been better :)

More information about the Python-Dev mailing list