[Python-Dev] Updates to PEP 471, the os.scandir() proposal (original) (raw)

Ethan Furman ethan at stoneleaf.us
Fri Jul 11 05:26:05 CEST 2014

On 07/09/2014 09:02 PM, Nick Coghlan wrote:

On 9 Jul 2014 17:14, "Ethan Furman" wrote:

I like the 'onerror' API better primarily because it gives a single point to deal with the errors. [...] The "onerror" approach can also deal with readdir failing, which the PEP currently glosses over.

Do we want this, though? I can see an error handler for individual entries, but if one of the *dir commands fails that would seem to be fairly catastrophic.

I'm somewhat inclined towards the current approach in the PEP, but I'd like to see an explanation of two aspects:

1. How a scandir variant with an 'onerror' option could be implemented given the version in the PEP

Here's a stab at it:

 def scandir_error(path, info=None, onerror=None):
     for entry in scandir(path):
         if info == 'type':
             try:
                 entry.is_dir()
             except OSError as exc:
                 if onerror is None:
                     raise
                 if not onerror(exc, entry):
                     continue
         elif info == 'lstat':
             try:
                 entry.lstat()
             except OSError as exc:
                 if onerror is None:
                     raise
                 if not onerror(exc, entry):
                     continue
         yield entry

Here it is again with an attempt to deal with opendir/readdir/closedir exceptions:

 def scandir_error(path, info=None, onerror=None):
     entries = scandir(path)
     try:
         entry = next(entries)
     except StopIteration:
         # pass it through
         raise
     except Exception as exc:
         if onerror is None:
             raise
         if not onerror(exc, 'what else here?'):
             # what do we do on False?
             # what do we do on True?
     else:
         for entry in scandir(path):
             if info == 'type':
                 try:
                     entry.is_dir()
                 except OSError as exc:
                     if onerror is None:
                         raise
                     if not onerror(exc, entry):
                         continue
             elif info == 'lstat':
                 try:
                     entry.lstat()
                 except OSError as exc:
                     if onerror is None:
                         raise
                     if not onerror(exc, entry):
                         continue
             yield entry

2. How the existing scandir module handles the 'onerror' parameter to its directory walking function

Here's the first third of it from the repo:

 def walk(top, topdown=True, onerror=None, followlinks=False):
     """Like os.walk(), but faster, as it uses scandir() internally."""
     # Determine which are files and which are directories
     dirs = []
     nondirs = []
     try:
         for entry in scandir(top):
             if entry.is_dir():
                 dirs.append(entry)
             else:
                 nondirs.append(entry)
     except OSError as error:
         if onerror is not None:
             onerror(error)
         return
     ...

-- Ethan

More information about the Python-Dev mailing list