Issue 4377: tokenize.detect_encoding() and Mac newline (original) (raw)

Issue4377

Created on 2008-11-21 12:32 by vstinner, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (4)
msg76176 - (view)	Author: STINNER Victor (vstinner) *	Date: 2008-11-21 12:32
I'm trying to fix IDLE to support Unicode (#4008 and #4323). Instead of IDLE builtin charset detection, I tried to use tokenize.detect_encoding() but this function doesn't work with script using Mac new line (b"\r"). Code to detect the encoding of a Python script: ---- def pythonEncoding(filename): with open(filename, 'rb') as fp: encoding, lines = detect_encoding(fp.readline) return encoding ---- Example to reproduce the problem with Mac script: ---- fp = BytesIO(b'# coding: ISO-8859-1\rprint("Bonjour ma ch\xe8re amie")\r') encoding, lines = detect_encoding(fp.readline) print(encoding, lines) ---- => Result: utf-8 [b'# coding: ISO-8859-1\rprint("Bonjour ma ch\xe8re amie")\r'] The problem occurs at "line_string = line.decode('ascii')". Since "line" contains a non-ASCII character (b"\xe8"), the conversion fails.
msg84126 - (view)	Author: STINNER Victor (vstinner) *	Date: 2009-03-24 23:44
See also related issue: #4628 (No universal newline support for compile() when using bytes).
msg100364 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-03-04 01:07
I finally wrote a patch using a small generator based on the .splitlines(1) method. Patch includes a test.
msg112017 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-07-29 22:18
Well, it looks like nobody cares (including me), so I close this issue.