Issue 13978: OSError exception in multiprocessing module when using os.remove() on NFS (original) (raw)

Issue13978

Created on 2012-02-09 18:24 by jjardon, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (3)
msg152972 - (view)	Author: Javier Jardón (jjardon)	Date: 2012-02-09 18:24
I have this little test case: import multiprocessing manager = multiprocessing.Manager() del manager and I get this: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/util.py", line 261, in _run_finalizers finalizer() File "/usr/lib/python2.7/multiprocessing/util.py", line 200, in __call__ res = self._callback(*self._args, **self._kwargs) File "/usr/lib/python2.7/shutil.py", line 249, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/usr/lib/python2.7/shutil.py", line 247, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: '/nfs/tmp/pymp-f7R9S6/.nfs00000000e039692f00000236' (the TMDIR directory is in a nfs server)
msg153071 - (view)	Author: Charles-François Natali (neologix) *	Date: 2012-02-10 18:42
""" OSError: [Errno 16] Device or resource busy: '/nfs/tmp/pymp-f7R9S6/.nfs00000000e039692f00000236' """ That's because the temporary directory is removed while a file inside is still open. And that's really likely the unix socket used by the server's listener. What happens if you do this instead: """ import multiprocessing manager = multiprocessing.Manager() manager.shutdown() del manager """ You should shutdown the manager before it gets garbage collected. > (the TMDIR directory is in a nfs server) Bad idea, for the following reasons: - you're actually lucky Linux allows binding unix sockets over NFS filesystems, some Unix flavors don't - you're likely to run into similar problems, because code which removes a temporary directory while having still an open FD is quite common, and will break with a tmp dir over NFS - mkstemp() and friends use O_EXCL to create a temporary file securely, and some NFS implementations of O_EXCL are unsafe (should be OK with NFSv3 and later though) - it's not a good idea performance wise
msg153640 - (view)	Author: Charles-François Natali (neologix) *	Date: 2012-02-18 11:56
I'm closing, since the manager should be shutdown (and TMPDIR on NFS isn't a good idea).