Issue 33240: shutil.rmtree fails if inner folder is open in Windows Explorer (original) (raw)

Issue33240

Created on 2018-04-08 02:51 by yuliu, last changed 2022-04-11 14:58 by admin.

Messages (14)
msg315077 - (view)	Author: Yu Liu (yuliu)	Date: 2018-04-08 02:51
Given the following directory structure on a Windows machine: - foo - bar a call to `shutil.rmtree("foo")` will fail when the inner folder `bar` is opened in an Explorer. The error message indicates the `foo` directory is not empty, while after the execution, although it failed, the `foo` directory is empty. So the inner folder `bar` was removed successfully, but `foo` was not. And the error message is misleading. It will not fail when `foo` is opened in an Explorer, neither on Linux system.
msg315078 - (view)	Author: Eryk Sun (eryksun) *	Date: 2018-04-08 04:35
This is not an uncommon problem. If there's one or more existing references to a file or empty directory that were opened with shared delete access, then a delete operation will succeed, but the file or directory will not be unlinked from the parent directory. The filesystem only unlinks a file or directory if its "delete disposition" is set when the last reference is closed. Removing the parent directory thus requires a loop that retries the delete until it succeeds, i.e. until existing references are closed and the directory finally becomes empty and thus deletable. If the problem is caused by an anti-malware program, it should typically be resolved within a short time. Exactly how long to wait in a retry loop before failing the operation should be configurable. Maybe you can conduct a simple experiment to measure the wait time required in your case. Run the following with "bar" opened in Explorer. Substitute the real path of "foo" in PARENT_PATH. import os import time ERROR_DIR_NOT_EMPTY = 145 PARENT_PATH = 'foo' CHILD_PATH = os.path.join(PARENT_PATH, 'bar') os.rmdir(CHILD_PATH) t0 = time.perf_counter() while True: try: os.rmdir(PARENT_PATH) wait_time = time.perf_counter() - t0 break except OSError as e: if e.winerror != ERROR_DIR_NOT_EMPTY: raise print(wait_time)
msg315103 - (view)	Author: Yu Liu (yuliu)	Date: 2018-04-09 01:13
The result is 0.00026412295632975946 on my computer. Does this mean it is caused by an anti-malware program?
msg315105 - (view)	Author: Eryk Sun (eryksun) *	Date: 2018-04-09 02:41
A sub-millisecond wait is fairly quick, but it depends on the machine speed. I should have included a counter. Try the following. It's not reproducing the problem if num_retries doesn't get incremented. import os import time ERROR_DIR_NOT_EMPTY = 145 PARENT_PATH = 'foo' CHILD_PATH = os.path.join(PARENT_PATH, 'bar') os.rmdir(CHILD_PATH) num_retries = 0 t0 = time.perf_counter() while True: try: os.rmdir(PARENT_PATH) break except OSError as e: if e.winerror != ERROR_DIR_NOT_EMPTY: raise num_retries += 1 wait_time = time.perf_counter() - t0 print('num_retries:', num_retries) print('wait_time:', wait_time)
msg315110 - (view)	Author: Yu Liu (yuliu)	Date: 2018-04-09 07:02
This time I run it a couple of consecutive times manually. The result shows that the first time it retried 12 times and the wait time was 0.0004259171330071893. Then it seems to be steady, and the num_retried is 3 or 4, and the wait_time is about 0.00025. I also run it without `bar` opened in Explorer. The num_retried is 0 and the wait_time is 4.46745114706336e-05. Because of some reasons, I can't post the exact results here. I will run it on another computer later and post the results if you need all of the exact results.
msg315126 - (view)	Author: Yu Liu (yuliu)	Date: 2018-04-09 12:02
These are results on another slower machine. Note these results are attained on Windows 10, while the above on Windows 7. Just in case it has some influence. $ python test.py num_retries: 6 wait_time: 0.0008691726957804373 $ python test.py num_retries: 3 wait_time: 0.0007175661796639806 $ python test.py num_retries: 6 wait_time: 0.0007962191842657514 $ python test.py num_retries: 3 wait_time: 0.0006970480045504753 $ python test.py num_retries: 4 wait_time: 0.0009637842810260455 $ python test.py num_retries: 4 wait_time: 0.001005390580561765 $ python test.py num_retries: 3 wait_time: 0.000654301806397339 $ python test.py num_retries: 6 wait_time: 0.0008857012257329832 $ python test.py num_retries: 4 wait_time: 0.0009227479307990348 $ python test.py num_retries: 4 wait_time: 0.0008976701612158615 And this is result without `bar` opened in Explorer. $ python test.py num_retries: 0 wait_time: 0.00019834235943055225
msg315129 - (view)	Author: Eryk Sun (eryksun) *	Date: 2018-04-09 15:11
Your case probably isn't due to a anti-malware filesystem filter. Explorer keeps handles open to directories to get updates via ReadDirectoryChangesExW. It opens watched directories with shared delete access, so deleting the child succeeds. But as discussed above, the directory isn't unlinked from the parent until Explorer closes its handle. Apparently it's not fast enough on the systems you tested. As a workaround, you can define an onerror handler for use with shutil.rmtree() that retries the rmdir() call in a loop for up to a given timeout period, such as 10 ms. For convenience, a handler that retries unlink() and rmdir() could be distributed with shutil. For ease of use, it could be enabled by default on Windows.
msg315185 - (view)	Author: Giampaolo Rodola' (giampaolo.rodola) *	Date: 2018-04-11 07:19
> For convenience, a handler that retries unlink() and rmdir() could be distributed with shutil. For ease of use, it could be enabled by default on Windows. +1 on that. I bumped into this many times over the years as occasional and hardly reproducible test failures when cleaning up test files/dirs. The tricky part is how to distinguish a legitimate "directory is not empty" error though.
msg365768 - (view)	Author: Ofek Lev (Ofekmeister) *	Date: 2020-04-04 15:10
> For convenience, a handler that retries unlink() and rmdir() could be distributed with shutil. For ease of use, it could be enabled by default on Windows. Any update on that? I just spent a bunch of time debugging this on Windows.
msg388964 - (view)	Author: Eryk Sun (eryksun) *	Date: 2021-03-17 20:34
> Explorer keeps handles open to directories to get updates via > ReadDirectoryChangesExW. It opens watched directories with > shared delete access, so deleting the child succeeds. But as > discussed above, the directory isn't unlinked from the parent > until Explorer closes its handle. In Windows 10, NTFS allows deleting an empty directory that's currently opened with shared-delete access, so this race condition will be increasingly less common. For example: access = GENERIC_READ sharing = FILE_SHARE_READ | FILE_SHARE_WRITE	FILE_SHARE_DELETE disposition = OPEN_EXISTING flags = FILE_FLAG_BACKUP_SEMANTICS os.mkdir('spam') h = CreateFile('spam', access, sharing, None, disposition, flags, None) os.rmdir('spam') >>> print(GetFinalPathNameByHandle(h, 0)) \\?\C:\$Extend\$Deleted\004E00000000632F70819337 FAT filesystems do not support this capability, and may never support it, so the problem hasn't gone away completely.
msg389193 - (view)	Author: Gregory P. Smith (gregory.p.smith) *	Date: 2021-03-20 22:56
Isn't this just "how windows behaves" on some filesystems with little that we can do about it? The only real action item I can see here is that if it is _reasonable_ for us to detect the situation and improve the error message, that'd help the users (and reduce bugreports like this). Otherwise I suggest closing it as wont fix / working as the platform intends.
msg389200 - (view)	Author: Eryk Sun (eryksun) *	Date: 2021-03-20 23:47
> Isn't this just "how windows behaves" on some filesystems with > little that we can do about it? The suggestion was to include an error handler that retries unlink() and rmdir() -- particularly rmdir() -- a given number of times, probably with an exponential back off, before giving up and failing. This accounts for race conditions in which the delete succeeds but the file/directory can't be unlinked because it's currently open (with delete sharing). A lot of these cases are similar to ReadDirectoryChangesExW(), in which the owner of the open is immediately notified that the directory was deleted. If they're well behaved, like Explorer, they immediately close their handle to allow the directory to be unlinked by the system. But that may not be soon enough for the process that deleted the directory. The suggested retry loop would help to work around this race condition. A couple people were in favor of this being provided by the standard library, so everyone isn't forced to implement there own workaround for a common problem.
msg389201 - (view)	Author: Gregory P. Smith (gregory.p.smith) *	Date: 2021-03-20 23:53
oh, I missed that a notification happens to the other process(es) in a common case, a bit of retrying with backoff would actually make sense there. But I wouldn't let a retry run for longer than a second or three, as code tends to assume that rmtree is pretty quick and if something is going to close asynchronously, it should close quickly.
msg389203 - (view)	Author: Eryk Sun (eryksun) *	Date: 2021-03-21 00:29
> oh, I missed that a notification happens to the other process(es) in a > common case, a bit of retrying with backoff would actually make sense The other common problem with deleting an empty directory is when it's opened as the working directory of a process. This case fails as a sharing violation because the open doesn't share delete access. There's nothing reasonable to do about it without user interaction, which would be a complicated bit of code: find the process that has the directory open, display a message to the user in the desktop session of the process, and wait for a response. That's not a good candidate for the standard library.