If a generator has its close() method called before any items are requested from it, a finally block in the generator function will not be executed. I encountered this when wrapping an open file to alter the result of iterating over it. Using a generator function with a try/finally block seemed like a simple way of acheiving this. Here's an example that logs each line as it's read: def logged_lines(f): try: for line in f: logging.warning(line.strip()) yield line finally: logging.warning('closing') f.close() If the generator is created and closed immediately, the underlying file-like object is left open: >>> f = urlopen('https://docs.python.org/') >>> lines = logged_lines(f) >>> lines.close() >>> f.closed False But once the first item is requested from the generator, close() will trigger cleanup: >>> lines = logged_lines(f) >>> next(lines) WARNING:root:>> lines.close() WARNING:root:closing >>> f.closed True Having read the documentation for yield expressions, I don't believe this behaviour to be non-conformant - but it still seems like a bit of a gotcha to me. Should this usage be warned against?
I think there is an issue in the way you designed your cleanup logic. So I think this issue is invalid. Usually, the code (funcion, class, ...) that *opens* the file should also be resposible of closing it. option 1) the caller opens and closes the file and wrapping the logged lines in a try/finally def logged_lines(f): try: for line in f: logging.warning(line.strip()) yield line finally: logging.warning('closing') f = open('yyy', 'r') try: for l in logged_lines(f): print(l) finally: f.close() option 2) the funcion opens and closes the file def logged_lines(fname): f = open('yyy', 'r') try: for line in f: logging.warning(line.strip()) yield line finally: logging.warning('closing') f.close() for l in logged_lines('yyy'): print(l)
This looks logical to me. The "finally" block is only entered if the "try" block is ever entered, but if you don't consume anything in the generator then the generator's code is never actually executed.
Ok, I can accept that. I think my mistake was to assume that because a generator has a close() method, I could treat it as a lightweight wrapper for another closeable object. But it's better to regard a generator function that wraps an iterable as something more akin to map() or filter(), and use a class if it's necessary to wrap a file such that close() is passed through. I happened to take a fresh look at this just the other day and it also occurred to me that the kind of composition I was trying to do can work if it's generators all the way down: def open_lines(name, mode='rt', buffering=-1): with open(name, mode, buffering) as f: for line in f: yield line def logged_lines(f): try: for line in f: logging.warning(line.strip()) yield line finally: f.close() lines = open_lines('yyy', 'r') if verbose: lines = logged_lines(lines) try: for line in lines: print(line) finally: lines.close() So a generator can transparently wrap a plain iterable or another generator, but not closeable objects in general. There's nothing really wrong with that, so I'm happy for this issue to be closed as invalid.
History
Date
User
Action
Args
2022-04-11 14:58:11
admin
set
github: 67416
2015-05-14 23:22:23
martin.panter
set
status: open -> closedresolution: not a bugstage: resolved
