Issue 36654: Add example to tokenize.tokenize (original) (raw)

The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the io.IOBase.readline() method of file objects. Each call to the function should return one line of input as bytes.

Add an example like this should be easier to understand:

example.py

def foo: pass

tokenize_example.py

import tokenize f = open('example.py', 'rb') token_gen = tokenize.tokenize(f.readline)

for token in token_gen: # Something like this # TokenInfo(type=1 (NAME), string='class', start=(1, 0), end=(1, 5), line='class Foo:\n') # TokenInfo(type=1 (NAME), string='Foo', start=(1, 6), end=(1, 9), line='class Foo:\n') # TokenInfo(type=53 (OP), string=':', start=(1, 9), end=(1, 10), line='class Foo:\n') print(token)