| msg71397 - (view) |
Author: Brett Cannon (brett.cannon) *  |
Date: 2008-08-19 01:19 |
| Turns out that PyTokenizer_FindEncoding() never properly succeeds because the tok_state used by it does not have tok->filename set, which is an error condition in the tokenizer. This error has been masked by the one place the function is used, imp.find_module() because a NULL return is never checked for an error, but instead just assumes the default source encoding suffices. |
|
|
| msg71398 - (view) |
Author: Brett Cannon (brett.cannon) * |
Date: 2008-08-19 01:20 |
| I have not bothered to check if this exists in 2.6, but I don't see why it would be any different. |
|
|
| msg71399 - (view) |
Author: Brett Cannon (brett.cannon) * |
Date: 2008-08-19 01:44 |
| Turns out that the NULL return value can signal an error that manifests itself as SyntaxError("encoding problem: with BOM") thanks to the lack of tok->filename being set in Parser/tokenizer.c:fp_setreadl() which is called by check_coding_spec() and assumes that since tok->encoding was never set (because fp_setreadl() returned an error value) that it had something to do with the BOM. The only reason this was found is because my bootstrapping of importlib into Py3K, at some point, triggers a PyErr_Occurred() which finally notices the error. |
|
|
| msg71407 - (view) |
Author: Brett Cannon (brett.cannon) * |
Date: 2008-08-19 05:25 |
| Attached is a patch that fixes where the error occurs. By opening the file by either file name or file descriptor, the problem goes away. Once this patch is accepted then PyErr_Occurred() should be added to all uses of PyTokenizer_FindEncoding(). |
|
|
| msg72392 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-09-03 16:41 |
| I don't understand the whole decoding machinery in the tokenizer, but the patch looks ok to me. (tested in debug mode under Linux and Windows) |
|
|
| msg72420 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2008-09-03 21:26 |
| The patch also looks pretty harmless to me. :) |
|
|
| msg72477 - (view) |
Author: Hyeshik Chang (hyeshik.chang) * |
Date: 2008-09-04 03:35 |
| pitrou, that's because Python source code can't be correctly tokenized when it's encoded in few odd encodings like iso-2022 or shift-jis which utilizes \, (, ) and " as second byte of two-byte character sequence. For example, '\x81\\' is HORIZONTAL BAR in shift-jis, exec('print "\x81\\"') fails. because of " is ignored by second byte of '\x81\\'. |
|
|
| msg72480 - (view) |
Author: Brett Cannon (brett.cannon) * |
Date: 2008-09-04 05:04 |
| Committed in r66209. |
|
|