Issue 36243: Python os.listdir fails with FileNotFoundError when directory exists (original) (raw)

I have the following code:

    def handle_empty_directories(dir):
        if os.path.exists(dir):
            shouter.shout("%s exists" % dir)
        else:
           shouter.shout("%s doesn't exists" % dir)
        entries = os.listdir(dir)
        if entries == []:
            open(os.path.join(dir, Commiter.hed_file), "w").close()
        else:
            if (len(entries) > 1) and (Commiter.hed_file in entries):
                os.remove(os.path.join(dir, Commiter.hed_file))
            for entry in entries:
                if entry not in Commiter.hed_ignore:
                    full_entry = os.path.join(dir, entry)
                    if (os.path.isdir(full_entry)):
                        Commiter.handle_empty_directories(full_entry)

Occasionally, the call to os.listdir(dir) fails with FileNotFoundError even though the os.path.exists(dir) call says that exists:

    08:57:56 - C:\r2g-wd\sport-6.0.5\SBS\SBS\Light\bin\com\ibm\ArtifactTechnology\ABS\ArtifactBroker exists
    Traceback (most recent call last):
      File "migration.py", line 169, in 
        migrate()
      File "migration.py", line 80, in migrate
        rtc.acceptchangesintoworkspace(rtc.getchangeentriestoaccept(changeentries, history))
      File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\rtcFunctions.py", line 304, in acceptchangesintoworkspace
        Commiter.handle_empty_directories(os.getcwd())
      File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\gitFunctions.py", line 126, in handle_empty_directories
        Commiter.handle_empty_directories(full_entry)
      File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\gitFunctions.py", line 126, in handle_empty_directories
        Commiter.handle_empty_directories(full_entry)
      File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\gitFunctions.py", line 126, in handle_empty_directories
        Commiter.handle_empty_directories(full_entry)
      [Previous line repeated 6 more times]
      File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\gitFunctions.py", line 116, in handle_empty_directories
        entries = os.listdir(dir)
    FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\r2g-wd\\sport-6.0.5\\SBS\\SBS\\Light\\bin\\com\\ibm\\ArtifactTechnology\\ABS\\ArtifactBroker'

I've also seen a similar FileNoteFound on the

            open(os.path.join(dir, Commiter.hed_file), "w").close()

line when os.path.exists(dir) says that the directory exists.

How can this happen? I'm running Python 3.7.2 64-bit on Windows 10.

How can this happen?

Easily -- this looks like a "Time Of Check To Time Of Use" bug. You check for the existence of a directory, and then a fraction of a second later you attempt to use that directory. But on a multi-processing operating system like Windows, Mac or Linux, a lot can happen in that fraction of a second, including the directory being deleted by another process.

https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use

I'm not saying that this absolutely cannot be a bug in Python, but the initial indication is that this is simply a TOCTTOU bug in your code.

Unfortunately, your code is unrunnable by us, as it if full of mysterious objects like "shouter.shout" which we have no access too.

In addition, there's a problem that "handle_empty_directories" calls itself recursively as if it were a method of some "Commiter" object:

Commiter.handle_empty_directories

but it has no "self" parameter, so I don't know what is going on there.

For us to investigate further, you need to replicate the error using much simple code that we can run ourselves.

Please read http://www.sscce.org/ for some hints in producing short, self-contained, runnable code so that we can replicate this error.

In the meantime, I'm going to change this to Pending. When you have replicated this with simpler code, please feel free to change the status back to Open.