Issue 21775: shutil.copytree() crashes copying to VFAT on Linux: AttributeError: 'PermissionError' object has no attribute 'winerror' (original) (raw)
Created on 2014-06-15 21:37 by gward, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (10)
Author: Greg Ward (gward) 
Date: 2014-06-15 21:37
When using shutil.copytree() on Linux to copy to a VFAT filesystem, it crashes like this:
Traceback (most recent call last): File "/data/src/cpython/3.4/Lib/shutil.py", line 336, in copytree copystat(src, dst) File "/data/src/cpython/3.4/Lib/shutil.py", line 190, in copystat lookup("chmod")(dst, mode, follow_symlinks=follow) PermissionError: [Errno 1] Operation not permitted: '/mnt/example_nt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "copytree-crash.py", line 14, in shutil.copytree('PC/example_nt', '/mnt/example_nt') File "/data/src/cpython/3.4/Lib/shutil.py", line 339, in copytree if why.winerror is None: AttributeError: 'PermissionError' object has no attribute 'winerror'
I am not complaining about the PermissionError. That has been . Rather, I'm complaining about the crash that happens while attempting to handle the PermissionError.
Reproducing this is fairly easy, although it requires root privilege.
- dd if=/dev/zero of=dummy bs=1024 count=1024
- mkfs.vfat -v dummy
- sudo mount -o loop /tmp/dummy /mnt
Then create the reproduction script in the root of Python's source dir:
cat > copytree-crash.py <<EOF import os import shutil
assumptions:
1. /mnt is a directory writeable by current user
2. /mnt is a VFAT filesystem
3. current OS is Linux-ish
if os.path.exists('/mnt/example_nt'): print('cleaning up') shutil.rmtree('/mnt/example_nt')
print('copying') shutil.copytree('PC/example_nt', '/mnt/example_nt') EOF
and run it:
sudo ./python copytree-crash.py
Crash happens as documented above.
Author: Greg Ward (gward)
Date: 2014-06-15 21:40
In 3.3 and earlier, copytree() crashes roughly as described in , with shutil.Error that wraps the underlying "Operation not permitted" error from trying to chmod() something in a VFAT filesystem. Since this appears to accurately reflect what's coming from the kernel, I don't think this is a bug. Only the AttributeError, which is new in 3.4, is clearly a bug.
Author: Greg Ward (gward)
Date: 2014-06-15 21:44
Bad news: because reproducing this requires sudo (to mount an arbitrary filesystem), I'm not sure it's possible/desirable to add test code for it.
Good news: the fix is trivial, and it passes my manual test. Here's a patch:
--- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -336,7 +336,7 @@ copystat(src, dst) except OSError as why: # Copying file access times may fail on Windows
if why.winerror is None:
if errors: raise Error(errors)if getattr(why, 'winerror', None) is None: errors.append((src, dst, str(why)))
Running test suite now to make sure this doesn't break anything else...
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2014-11-18 20:19
LGTM. Go ahead Greg.
Author: STINNER Victor (vstinner) *
Date: 2014-11-18 20:42
Would it be possible to write a unit test, maybe using unittest.mock to mock most parts?
Author: Greg Ward (gward)
Date: 2014-11-21 01:37
Would it be possible to write a unit test, maybe using unittest.mock to mock most parts?
Good idea! Turns out this was quite straightforward. The test patch is:
--- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1,6 +1,7 @@
Copyright (C) 2003 Python Software Foundation
import unittest +import unittest.mock import shutil import tempfile import sys @@ -758,6 +759,20 @@ self.assertEqual(os.stat(restrictive_subdir).st_mode, os.stat(restrictive_subdir_dst).st_mode)
- @unittest.mock.patch('os.chmod')
- def test_copytree_winerror(self, mock_patch):
# When copying to VFAT, copystat() raises OSError. On Windows, the# exception object has a meaningful 'winerror' attribute, but not# on other operating systems. Do not assume 'winerror' is set.src_dir = tempfile.mkdtemp()dst_dir = os.path.join(tempfile.mkdtemp(), 'destination')self.addCleanup(shutil.rmtree, src_dir)self.addCleanup(shutil.rmtree, os.path.dirname(dst_dir))mock_patch.side_effect = PermissionError('ka-boom')with self.assertRaises(shutil.Error):shutil.copytree(src_dir, dst_dir)@unittest.skipIf(os.name == 'nt', 'temporarily disabled on Windows') @unittest.skipUnless(hasattr(os, 'link'), 'requires os.link') def test_dont_copy_file_onto_link_to_itself(self):
When run without the bug fix, this reproduces my original failure nicely:
$ ./python Lib/test/test_shutil.py ................s........................s.s........E..s..............................s..
ERROR: test_copytree_winerror (main.TestShutil)
Traceback (most recent call last): File "/data/src/cpython/3.4/Lib/shutil.py", line 337, in copytree copystat(src, dst) File "/data/src/cpython/3.4/Lib/shutil.py", line 191, in copystat lookup("chmod")(dst, mode, follow_symlinks=follow) File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 896, in call return _mock_self._mock_call(*args, **kwargs) File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 952, in _mock_call raise effect PermissionError: ka-boom
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 1136, in patched return func(*args, **keywargs) File "Lib/test/test_shutil.py", line 774, in test_copytree_winerror shutil.copytree(src_dir, dst_dir) File "/data/src/cpython/3.4/Lib/shutil.py", line 340, in copytree if why.winerror is None: AttributeError: 'PermissionError' object has no attribute 'winerror'
Ran 89 tests in 0.095s
FAILED (errors=1, skipped=5)
Excellent! No need for root privs, and the bug is quite obvious.
Apply my getattr() fix, and the test passes.
I'll commit on branch 3.4 and merge to default.
Author: Greg Ward (gward)
Date: 2014-11-21 01:55
I'll commit on branch 3.4 and merge to default.
Whoops, never mind. Looks like I don't have push permission to hg.python.org after all. It's been 8 years since my last commit, so I shouldn't complain.
So... can someone with commit privs please
hg pull -r 3a1fc6452340 http://hg.gerg.ca/cpython
then merge to trunk and push?
Thanks!
Author: Roundup Robot (python-dev) 
Date: 2014-12-10 00:51
New changeset 50517a4d7cce by Berker Peksag in branch '3.4': Issue #21775: shutil.copytree(): fix crash when copying to VFAT https://hg.python.org/cpython/rev/50517a4d7cce
New changeset 7d5754af95a9 by Berker Peksag in branch 'default': Issue #21775: shutil.copytree(): fix crash when copying to VFAT https://hg.python.org/cpython/rev/7d5754af95a9
Author: Berker Peksag (berker.peksag) *
Date: 2014-12-10 00:55
Committed the patch with a NEWS entry. Thanks Greg.
(You can send your SSH key to hgaccounts@python.org. See also https://docs.python.org/devguide/coredev.html#ssh)
Author: STINNER Victor (vstinner) *
Date: 2014-12-10 13:11
Note: Python 2.7 is not affected, I cannot find "winerror" in shutil.py.
History
Date
User
Action
Args
2022-04-11 14:58:05
admin
set
github: 65974
2014-12-10 13:11:01
vstinner
set
messages: +
2014-12-10 00:55:36
berker.peksag
set
status: open -> closed
nosy: + berker.peksag
messages: +
resolution: fixed
stage: commit review -> resolved
2014-12-10 00:51:29
python-dev
set
nosy: + python-dev
messages: +
2014-11-21 01:55:48
gward
set
messages: +
2014-11-21 01:37:30
gward
set
messages: +
2014-11-18 20:42:12
vstinner
set
nosy: + vstinner
messages: +
2014-11-18 20:20:06
serhiy.storchaka
set
type: behavior
2014-11-18 20:19:50
serhiy.storchaka
set
versions: + Python 3.5
nosy: + serhiy.storchaka
messages: +
assignee: gward
stage: commit review
2014-06-15 21:44:19
gward
set
messages: +
2014-06-15 21:40:21
gward
set
keywords: + 3.4regression
messages: +
2014-06-15 21:37:40
gward
create