Issue 25717: tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount (original) (raw)
Created on 2015-11-24 04:23 by Hans Lawrenz, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (27)
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 04:23
Inside a virtualbox vm, calling tempfile.TemporaryFile(dir=foo) where foo is a directory which resides on a volume mounted from the host OS, a FileNotFoundError exception is thrown.
In the following code sample, the second block will print "Path 2: ERROR" on Python 3.5 but not on 3.4 or 2.7 (assuming the vagrant environment provided below):
import tempfile
try: with tempfile.TemporaryFile() as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Path 1: worked") except FileNotFoundError as e: print("Path 1: ERROR")
try: with tempfile.TemporaryFile(dir="/vagrant") as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Path 2: worked") except FileNotFoundError as e: print("Path 2: ERROR")
A runnable test case can be found here: https://github.com/hlawrenz/py35tempfiletest
Author: Martin Panter (martin.panter) * 
Date: 2015-11-24 04:42
I am unable to produce this on native Linux so far with various kinds of mount points I normally have. What is your guest OS?
If you can provide any more details of the failure, such as a traceback, that would be helpful. What is the underlying OS call that is causing the exception?
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 06:54
Host OS: Mac OS 10.11.1 Guest OS: Ubuntu Trusty 64 Virtualbox: 4.3.30 Vagrant: 1.7.4
Example with trace thrown:
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ cat tt.py import tempfile
with tempfile.TemporaryFile(dir="/vagrant") as tf: tf.write("testing testing testing\n".encode('utf-8'))
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python3.5 tt.py Traceback (most recent call last): File "tt.py", line 4, in with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python3.5/tempfile.py", line 753, in TemporaryFile newline=newline, encoding=encoding) FileNotFoundError: [Errno 2] No such file or directory
I think the underlying system call that's causing the error (I attached the full strace in case that's helpful): open("/vagrant", O_RDWR|O_EXCL|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC|0x400000) = -1 EOPNOTSUPP (Operation not supported)
Finally, the mount point looks like this: vagrant on /vagrant type vboxsf (uid=1000,gid=1000,rw)
Author: Martin Panter (martin.panter) *
Date: 2015-11-24 08:39
Thanks for the strace output. I think the actual error is the fstat() call a bit after that first open() call. FileNotFoundError is ENOENT:
open("/vagrant/tmpy5ioznh4", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 4 unlink("/vagrant/tmpy5ioznh4") = 0 fstat(4, 0x7ffc0b326520) = -1 ENOENT (No such file or directory) close(4) = 0 write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
My theory is that the fstat() call at <https://hg.python.org/cpython/annotate/3.5/Modules/_io/fileio.c#l441> is failing, probably because the file entry has been removed from its directory. This call was added by revision 3b5279b5bfd1 (Issue 21679). Before this change, fstat() was called twice, but in each case an error was tolerated.
Posix does not mention fstat() returning this ENOENT error, so maybe this could be a bug with the Virtual Box or Vagrant driver for the mounted filesystem. But it might be nice to make Python more permissive if fstat() fails, like it was in 3.4.
As a workaround, you may be able to use NamedTemporaryFile, if you are happy for the file to have a name and a directory entry.
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 15:38
Unfortunately changing the tempfile call isn't an easy option for me. The situation in which I'm encountering the error is running tox on our project which is mounted in the /vagrant directory in the VM (this is standard for vagrant). Tox makes its own temp directory--presumably for test isolation--in the .tox directory in the project root. The actual call to tempfile.TemporaryFile() which is triggering the error is in a third party library that is called during a test run.
I was able to work around the issue by changing the tox workdir setting to outside the mount. However, I'd like to mention that developing in vagrant in this fashion isn't uncommon and once 3.5 gains adoption I would guess this issue may affect a good number of people.
I'm happy to take a stab at writing a patch but looking at the code it's somewhat out of my comfort zone and I worry I'd make a hash of it.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2015-11-24 15:51
What is exact version of you Python? I can't identify the line in the traceback. There are few lines "newline=newline, encoding=encoding)" in tempfile.py, but no one is closer to line 753.
Author: Anilyka Barry (abarry) * 
Date: 2015-11-24 16:07
I'm with Serhiy, the line number is inane. Could you put the exact contents of your /usr/lib/python3.5/tempfile.py file somewhere for us to see? Output of sys.version would be nice, too.
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 17:43
Serhiy and Emanuel, I'll paste below the surrounding code and attach the exact tempfile.py. It is the version distributed with the 3.5.0 release. If you take a look at the github repo I linked in the first comment you can also try it out for yourself if you've got vagrant and virtualbox installed. I was able to recreate the error with the provisioning compiling python and with it installing from a ppa. You can see the details in the Vagrantfile. I also had a coworker test in a nearly identical environment and he had the same result.
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
try:
_os.unlink(name)
return _io.open(fd, mode, buffering=buffering,
newline=newline, encoding=encoding)
except:
_os.close(fd)
raiseOne final note. I decided to try this out with a windows host and the only clarity this test adds is that the problem doesn't show up there but it fails for its own reasons on all the python versions I tried. See below for reference:
Last login: Tue Nov 24 17:16:12 2015 from 10.0.2.2 vagrant@vagrant-ubuntu-trusty-64:~$ cat /vagrant/foo.py import tempfile
with tempfile.TemporaryFile(dir="/vagrant") as tf:
tf.write("testing testing testing\n".encode('utf-8'))
print("Path 2: worked")
vagrant@vagrant-ubuntu-trusty-64:$ python /vagrant/foo.py
Traceback (most recent call last):
File "/vagrant/foo.py", line 4, in
with tempfile.TemporaryFile(dir="/vagrant") as tf:
File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
_os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpvx7Mie'
vagrant@vagrant-ubuntu-trusty-64:$ python2.7 /vagrant/foo.py
Traceback (most recent call last):
File "/vagrant/foo.py", line 4, in
with tempfile.TemporaryFile(dir="/vagrant") as tf:
File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
_os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpNXQ6Cf'
vagrant@vagrant-ubuntu-trusty-64:$ python3.4 /vagrant/foo.py
Traceback (most recent call last):
File "/vagrant/foo.py", line 4, in
with tempfile.TemporaryFile(dir="/vagrant") as tf:
File "/usr/lib/python3.4/tempfile.py", line 638, in TemporaryFile
_os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpfwhj7ip4'
vagrant@vagrant-ubuntu-trusty-64:$ python3.5 /vagrant/foo.py
Traceback (most recent call last):
File "/vagrant/foo.py", line 4, in
with tempfile.TemporaryFile(dir="/vagrant") as tf:
File "/usr/lib/python3.5/tempfile.py", line 751, in TemporaryFile
_os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmp02s19r_a'
vagrant@vagrant-ubuntu-trusty-64:$ python2.7 --version
Python 2.7.6
vagrant@vagrant-ubuntu-trusty-64:$ python3.4 --version
Python 3.4.3
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 --version
Python 3.5.0
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2015-11-24 19:06
Is your file system NFS?
Author: Anilyka Barry (abarry) *
Date: 2015-11-24 19:38
Your file has a lot of shenanigans that are triggered if 'shutil' fails to be imported, adding an extra 139 lines at the top of the file, which is exactly how offset your traceback is compared to our lines. The rest of the file is virtually identical. This makes me wonder, however, why you have these differences (and that this might not be the only different file).
I would still like the output of sys.version, which, unlike 'python --version' contains the build date and time, among other things. Bonus points for the 'sys.implementation' output. It's unlikely to be of relevance, but I would rather explore every possibility.
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 19:40
The file system causing the problem is of type vboxsf which is the Virtualbox shared folder file system type.
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-24 19:58
Emanuel, sorry, I missed the request for sys.version earlier.
The tempfile.py I attached earlier is from the python 3.5 pulled from a ppa. I wouldn't be surprised if it has some patches applied by debian/ubuntu. To be clear though the problem also presents itself with a python 3.5 I compiled. The Vagrantfile in the github repository shows exactly how I compiled it.
Here it is from the version from the ppa:
Python 3.5.0 (default, Sep 17 2015, 00:00:00) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.
import sys sys.version '3.5.0 (default, Sep 17 2015, 00:00:00) \n[GCC 4.8.4]' sys.implementation namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))
Here are the same for the version I compiled: vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 Python 3.5.0 (default, Nov 24 2015, 19:50:42) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.
import sys sys.version '3.5.0 (default, Nov 24 2015, 19:50:42) \n[GCC 4.8.4]' sys.implementation namespace(cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))
Author: Martin Panter (martin.panter) *
Date: 2015-11-24 22:10
This patch restores the previous behaviour of tolerating fstat() failures other than EBADF. Hans, if you are able to apply it to your compiled version of Python, it might prove that my fstat() theory is correct.
Author: Hans Lawrenz (Hans Lawrenz) *
Date: 2015-11-25 13:57
Martin, I applied your patch and it proved your hypothesis. See below for how I tested. I also updated the github repo for others to reproduce if they wish.
cd wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz
mkdir ~/dist
cd ~/dist
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
./configure --prefix=/home/vagrant/py35/dist &&
make && make install
mkdir ~/patch
cd ~/patch
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
patch -p1 < /vagrant/fstat-failure.patch
./configure --prefix=/home/vagrant/py35/patch &&
make && make install
vagrant@vagrant-ubuntu-trusty-64:$ ./py35/dist/bin/python3.5 /vagrant/temptest.py
Path 1: worked
Path 2: ERROR
vagrant@vagrant-ubuntu-trusty-64:$ ./py35/patch/bin/python3.5 /vagrant/temptest.py
Path 1: worked
Path 2: worked
Author: Martin Panter (martin.panter) *
Date: 2015-11-28 22:23
Antoine or Bohuslav, since you were involved in Issue 21679, would you be able to comment on fstat-failure.patch, which reverts back to ignoring most fstat() errors? The problem here is that fstat() seems to fail if the directory entry has been unlinked, and the file is on a Virtual Box shared folder filesystem.
Author: Bohuslav "Slavek" Kabrda (bkabrda) *
Date: 2015-12-01 11:00
The patch looks good to me.
Author: Roundup Robot (python-dev)
Date: 2015-12-06 03:51
New changeset 20ea12222b0e by Martin Panter in branch '3.5': Issue #25717: Tolerate fstat() failures in the FileIO constructor https://hg.python.org/cpython/rev/20ea12222b0e
New changeset 8c978cbe057c by Martin Panter in branch 'default': Issue #25717: Merge fstat() fix from 3.5 https://hg.python.org/cpython/rev/8c978cbe057c
Author: Martin Panter (martin.panter) *
Date: 2015-12-06 05:01
Thanks Bohuslav, and also to Hans for helping track this down.
Author: STINNER Victor (vstinner) *
Date: 2015-12-06 08:33
Martin, please add a comment to explain the rationale on ignoring errors other than EBADF. At least, mention this issue number.
Author: Martin Panter (martin.panter) *
Date: 2015-12-06 11:15
Okay will do
Author: Roundup Robot (python-dev)
Date: 2015-12-06 11:20
New changeset e9bf5803b716 by Martin Panter in branch '3.5': Issue #25717: Add comment explaining why errors are ignored https://hg.python.org/cpython/rev/e9bf5803b716
New changeset 8bf69413ec01 by Martin Panter in branch 'default': Issue #25717: Merge comment from 3.5 https://hg.python.org/cpython/rev/8bf69413ec01
Author: STINNER Victor (vstinner) *
Date: 2015-12-06 15:30
Thank you.
Author: Oleg Babintsev (Oleg Babintsev)
Date: 2016-06-18 09:40
Problem is still actual for Python 2.7 Can you provide the fix for 2.7 too?
Author: Martin Panter (martin.panter) *
Date: 2016-06-18 09:59
Are you sure Oleg? As far as I understand, Python 2 by default wraps C stdio file objects, and also has Python 3’s file objects in the “io” module. But I expect TemporaryFile() would use the default stdio files, and the cause of this bug, Issue 21679, should only have affected the “io” module in 3.5+.
Author: Oleg Babintsev (Oleg Babintsev)
Date: 2016-06-18 10:35
My environment: Windows as host OS, VirtualBox VM, Linux as guest OS
When temporary directory is a directory which mounted from the host OS, a "OSError: [Errno 26] Text file busy" exception is thrown.
Test file:
import tempfile
with tempfile.TemporaryFile(dir="/tmp") as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Temp file: worked")
"/tmp" directory - is a shared folder (vboxsf)
Result:
0e87f2561643">root@0e87f2561643:/# python /etc/odoo/tempfile-test.py Traceback (most recent call last): File "/etc/odoo/tempfile-test.py", line 3, in with tempfile.TemporaryFile(dir="/tmp") as tf: File "/usr/lib/python2.7/tempfile.py", line 513, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/tmp/tmpsl0JQI'
0e87f2561643">root@0e87f2561643:/# python --version Python 2.7.11+
0e87f2561643">root@0e87f2561643:/# python Python 2.7.11+ (default, Jun 2 2016, 19:34:15) [GCC 5.3.1 20160528] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.version '2.7.11+ (default, Jun 2 2016, 19:34:15) \n[GCC 5.3.1 20160528]'
Author: Martin Panter (martin.panter) *
Date: 2016-06-18 11:36
This original bug was about fstat() failing for an anonymous file (after the directory entry was removed). But in your situation the file system refuses to unlink the directory entry.
If you think there is something that can be fixed in Python, I suggest open a new bug. But it seems this is working as documented: “Under Unix, the directory entry for the file is removed immediately after the file is created.” In your case, this cannot happen, so you get the error from the OS.
Author: Oleg Babintsev (Oleg Babintsev)
Date: 2016-06-18 22:37
Thanks for explanation.
History
Date
User
Action
Args
2022-04-11 14:58:24
admin
set
github: 69903
2016-06-18 22:37:07
Oleg Babintsev
set
messages: +
2016-06-18 11:36:33
martin.panter
set
messages: +
versions: + Python 3.5, Python 3.6, - Python 2.7
2016-06-18 10:35:39
Oleg Babintsev
set
messages: +
2016-06-18 09:59:53
martin.panter
set
messages: +
2016-06-18 09:40:25
Oleg Babintsev
set
nosy: + Oleg Babintsev
messages: +
versions: + Python 2.7, - Python 3.5, Python 3.6
2016-02-11 01:43:11
eryksun
link
2015-12-06 15:30:20
vstinner
set
messages: +
2015-12-06 11:20:34
python-dev
set
messages: +
2015-12-06 11:15:29
martin.panter
set
messages: +
2015-12-06 08:33:34
vstinner
set
messages: +
2015-12-06 05:01:41
martin.panter
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2015-12-06 03:51:58
python-dev
set
nosy: + python-dev
messages: +
2015-12-01 11:00:16
bkabrda
set
messages: +
2015-11-28 22:23:47
martin.panter
set
versions: + Python 3.6
nosy: + pitrou, bkabrda
messages: +
stage: patch review
2015-11-25 13:57:02
Hans Lawrenz
set
messages: +
2015-11-24 22:16:03
serhiy.storchaka
set
nosy: + vstinner
2015-11-24 22:10:46
martin.panter
set
files: + fstat-failure.patch
keywords: + patch
messages: +
2015-11-24 19:58:27
Hans Lawrenz
set
messages: +
2015-11-24 19:40:55
Hans Lawrenz
set
messages: +
2015-11-24 19:38:21
abarry
set
messages: +
2015-11-24 19:06:27
serhiy.storchaka
set
messages: +
2015-11-24 17:43:56
Hans Lawrenz
set
files: + tempfile.py
messages: +
2015-11-24 16:07:13
abarry
set
nosy: + abarry
messages: +
2015-11-24 15:51:19
serhiy.storchaka
set
nosy: + serhiy.storchaka
messages: +
2015-11-24 15:38:38
Hans Lawrenz
set
messages: +
2015-11-24 08:39:42
martin.panter
set
messages: +
2015-11-24 06:54:31
Hans Lawrenz
set
files: + tempfile-strace.txt
messages: +
2015-11-24 04:42:49
martin.panter
set
keywords: + 3.5regression
nosy: + martin.panter
messages: +
2015-11-24 04:23:44
Hans Lawrenz
create