Issue 15054: bytes literals erroneously tokenized (original) (raw)

With Python 2.7, both b'hello' and br'hello' are wrong. With Python 3.3, b'hello' is wrong.

$ python2.7 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello'" 1,0-1,7: STRING "'hello'" 1,7-1,8: OP ',' 1,9-1,17: STRING "u'hello'" 1,17-1,18: OP ',' 1,19-1,28: STRING "ur'hello'" 1,28-1,29: OP ',' 1,30-1,31: NAME 'b' 1,31-1,38: STRING "'hello'" 1,38-1,39: OP ',' 1,40-1,42: NAME 'br' 1,42-1,49: STRING "'hello'" 1,49-1,50: NEWLINE '\n' 2,0-2,0: ENDMARKER ''

$ python3.3 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello', rb'hello'" 1,0-1,7: STRING "'hello'"
1,7-1,8: OP ','
1,9-1,17: STRING "u'hello'"
1,17-1,18: OP ','
1,19-1,28: STRING "ur'hello'"
1,28-1,29: OP ','
1,30-1,31: NAME 'b'
1,31-1,38: STRING "'hello'"
1,38-1,39: OP ','
1,40-1,49: STRING "br'hello'"
1,49-1,50: OP ','
1,51-1,60: STRING "rb'hello'"
1,60-1,61: NEWLINE '\n'
2,0-2,0: ENDMARKER ''