Issue 12760: Add create mode to open() (original) (raw)
Created on 2011-08-16 13:49 by David.Townshend, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (44)
Author: David Townshend (David.Townshend)
Date: 2011-08-16 13:49
Currently, opening a file with open(file, 'w') overwrites existing files. It would be useful for open() to raise an error when the file exists. This proposal is to add a 'c' mode to open, which has the effect to creating a file and opening it for writing, but raising an IOError if it already exists (i.e. the same as os.open(file, os.O_EXCL|os.O_CREAT).
The attached patch implements this, including tests and documentation.
Author: STINNER Victor (vstinner) * 
Date: 2011-08-16 13:57
I'm not sure that O_EXCL is portable (exist on all platforms) because Python source code uses "#ifdef O_EXCL".
Author: Benjamin Peterson (benjamin.peterson) *
Date: 2011-08-16 14:53
I think this should be brought up on python-ideas or python-dev.
Author: Charles-François Natali (neologix) *
Date: 2011-08-16 15:37
See also issue 12105. A couple downsides:
- O_EXCL is not necessarily portable (doesn't work well with NFS, maybe not on Windows?)
- O_EXCL is useful, as is O_CLOEXEC: to be consistent, we should also end up adding all other commonly-used flags, which are really OS-specific
Furthermore, you can achieve the same thing with: os.fdopen(os.open('/etc/fstab', os.O_WRONLY|os.O_CLOEXEC|os.O_CREAT)) it's more verbose, though.
Author: David Townshend (David.Townshend)
Date: 2011-08-16 15:43
It was discussed on python-ideas, but the subject of the thread was actually on shutils.move so it was not really discussed much. I will repost this idea separately.
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-18 11:33
The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:
O_EXCL is a POSIX standard. It is also supported under Windows by the _open/_wopen compatibility functions (which we use for file I/O).
Probably there are very old systems which don't support it, and perhaps new systems that don't implement it correctly (meaning not atomically); for the former I'd say we just don't care (who's gonna run Python 3 on a 1995 system?) and for the latter, well, if the OS designers think it's fine, let's just expose it as it is.
As for NFS, there's an interesting comment from 2007 here: http://lwn.net/Articles/251971/
“My NFS tester shows that it at least appears to work with Linux, Solaris and FreeBSD: http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at Linux 2.6 sources it doesn't look like it tries to implement a racy O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so the test's results should be correct. I don't know if other OSes do that. I guess it would be nice to have a better O_EXCL tester which tries to catch race conditions.”
Author: David Townshend (David.Townshend)
Date: 2011-08-18 11:56
My aim isn't to add all the commonly used flags, that would be pointless since its already possible using os.open. The aim is to add a missing feature to the builtin open(), i.e. file creation. At the moment open() implements read, write, and append, and creation is only implied by write. But in many cases, an explicit creation is desired (i.e, specifically create a new file, with an exception on failure). It is true that this is possible with os.open, but it is somewhat obscure, especially for beginners, and is not as easy to read as "open(file, 'c')"
Author: Charles-François Natali (neologix) *
Date: 2011-08-18 13:24
The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:
O_EXCL is a POSIX standard. It is also supported under Windows by the _open/_wopen compatibility functions (which we use for file I/O).
If it's supported by Windows then I'm OK (not that I personaly care about Windows :-).
and is not as easy to read as "open(file, 'c')"
Well, I'd rather have this flag called 'x', to be consistent with glibc's fopen():
""" c (since glibc 2.3.3) Do not make the open operation, or subsequent read and write operations, thread cancellation points.
x Open the file exclusively (like the O_EXCL flag ofopen(2)). If the file already exists, fopen() fails, and sets errno to EEXIST. This flag is ignored for fdopen(). """
By the way, could you submit your patch as a mercurial diff ("hg diff")? It makes it easier to review under Rietveld.
Author: David Townshend (David.Townshend)
Date: 2011-08-18 13:28
Changing form 'c' to 'x' is easy enough, and if there is already a convention it makes sense to stick to it.
I thought I had done a mercurial diff! I'll try again and resubmit.
Author: Julian Berman (Julian) *
Date: 2011-08-18 14:15
A minor documentation error in io.rst line 475 which was changed to:
- The mode can be
'c','r','w'or'a'for reading - (default), writing, or appending.
and should have "creating" at the front I assume, like you have it later.
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-18 14:19
Well, I'd rather have this flag called 'x', to be consistent with glibc's fopen():
""" c (since glibc 2.3.3) Do not make the open operation, or subsequent read and write operations, thread cancellation points.
x Open the file exclusively (like the O_EXCL flag ofopen(2)). If the file already exists, fopen() fails, and sets errno to EEXIST. This flag is ignored for fdopen().
Yeah, but I think "exclusively" is quite misleading since it does not perform any locking of any kind. Also, I don't think we'll ever integrate the glibc's "c" option in io.open().
Author: Charles-François Natali (neologix) *
Date: 2011-08-18 15:14
Yeah, but I think "exclusively" is quite misleading since it does not perform any locking of any kind.
It might be misleading, but I find it clear enough, and this name has been endorsed by POSIX.
Furthermore, there's an added bonus: actually, with the old I/O layer, one can already pass an 'x' flag to open, since it just calls fopen:
"""
cf@neobox:$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/tmp/foo", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
cf@neobox:$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/usr/lib/pymodules/python2.6/", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
IOError: [Errno 17] File exists: '/tmp/foo'
"""
I don't know if it's documented behavior, but the OP in issue 12105 was using it with python 2. Changing it to 'x' would make such code backward-compatible.
Finally, when I read open('/tmp/foo', 'wx'), it's immediately clear to me what's going on, while I'd have to look at open()'s documentation to find out what the 'c' flag does.
Author: STINNER Victor (vstinner) *
Date: 2011-08-18 15:16
I don't know if it's documented behavior
See #12103: it is not documented.
Author: David Townshend (David.Townshend)
Date: 2011-08-18 16:06
I hope this patch suits you better :-)
I've updated the documentation typo (thanks for pointing that out). I've also changed 'c' to 'x', since I think that if there is a convention we should stick to it. I don't think that it matters if the glibc docs say 'exclusive', as long it its clear in the python docs what it does, which I think it is. Having said that, I don't really have a strong opinion either way, so I'll happily change it back to 'c' if its preferred.
Author: Charles-François Natali (neologix) *
Date: 2011-08-24 19:18
So, what was the conclusion of the discussion brought up on python-dev? I had a feeling some core devs were opposed to this (I'm personally -0).
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-24 20:12
I haven't seen any discussion on python-dev. Have I missed something?
Author: Charles-François Natali (neologix) *
Date: 2011-08-24 20:17
I haven't seen any discussion on python-dev. Have I missed something?
It was on Python-ideas actually: http://mail.python.org/pipermail/python-ideas/2011-August/011183.html
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-24 20:35
Ah, right. Well I think all arguments against it were quite weak (or even wrong). Let's see:
it's not cross-platform: actually it is (OS_EXCL has been POSIX since at least 1997 (*), and Windows also has it)
os.open followed by os.fdopen is easy: it isn't that easy to get the incantation right (the pure Python open() in _pyio is 70 lines of code), especially if you want the file object to have the right "name" attribute
it doesn't fill a use case: actually, avoiding race conditions is an important use case, even though many people may never encounter it (I must admit I myself never really cared about this)
So this looks like a reasonable feature request IMHO.
(*) http://pubs.opengroup.org/onlinepubs/007908799/xsh/open.html
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *
Date: 2011-08-24 21:19
- os.open followed by os.fdopen is easy: it isn't that easy to get the incantation right (the pure Python open() in _pyio is 70 lines of code), especially if you want the file object to have the right "name" attribute
What if we can override the inner call to os.open()? For example, io.open() could grow an additional argument "fd_opener" which defaults to the equivalent of os.open.
Then creation mode can be expressed like this: open(filename, 'w', fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT) Another use case with openat (see #12797): open(filename, fd_opener=lambda path, mode: os.openat(fd, path, mode)
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-24 21:27
What if we can override the inner call to os.open()? For example, io.open() could grow an additional argument "fd_opener" which defaults to the equivalent of os.open.
I agree it would be a much better situation than what we have now.
Author: STINNER Victor (vstinner) *
Date: 2011-08-24 21:34
open(filename, 'w', fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT)
I prefer open(name, "c").
it doesn't fill a use case: actually, avoiding race conditions is an important use case, ...
It may help the issue #8604 (and maybe #8828).
Author: David Townshend (David.Townshend)
Date: 2011-10-31 10:22
I see this has been marked as a duplicate of http://bugs.python.org/issue12797. Please explain how this is, since that proposal does not appear to provide the functionality discussed here.
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *
Date: 2011-10-31 10:32
would allow things like:
def create_exclusive_file(filename): return open(filename, "w", opener=lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
Author: David Townshend (David.Townshend)
Date: 2011-11-07 06:53
It is already possible to write a wrapper function that does it:
def create(file): fd = os.open(file, os.O_EXCL | os.O_CREAT | os.O_WRONLY) return os.fdopen(fd)
The point it not that it can't be done, but that it is not straight forward. The docs say this about os.open(): "This function is intended for low-level I/O. For normal usage, use the built-in function open()"
I wouldn't call creating a new file low-level I/O, but it is normal usage.
Author: Éric Araujo (eric.araujo) *
Date: 2011-11-18 09:55
See #13424 for a doc request about this.
Author: Devin Jeanpierre (Devin Jeanpierre) *
Date: 2011-12-26 15:42
C11 uses 'x' for this, for what it's worth.
This is not a "duplicate issue". The openat solution is no easier than the os.open solution.
Author: Antoine Pitrou (pitrou) *
Date: 2011-12-27 17:11
C11 uses 'x' for this, for what it's worth.
This is not a "duplicate issue". The openat solution is no easier than the os.open solution.
Ok, let's re-open then. I'm not sold on the feature, but the fact C11 adds a dedicated letter mode for it could be a good excuse for us to mimick it :)
Author: Éric Araujo (eric.araujo) *
Date: 2011-12-30 20:59
This is not a "duplicate issue". The openat solution is no easier than the os.open solution.
Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:
...
open_exclusive = lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
...
fp = open(filename, 'w', opener=open_exclusive)
...That’s why this bug was initially closed as duplicate.
Author: Devin Jeanpierre (Devin Jeanpierre) *
Date: 2011-12-30 22:07
Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:
Sorry, yes. Wrong words, same thought. We can implement this using opener, but we could implement this with os.open before. What's changed, except that there's more ways to do it? (There is slightly more versatility with the opener method, but no more obviousness and no less typing).
My understanding from reading the other thread is that this is not the primary use-case of the new parameter for open(). In fact, this ticket was not really mentioned at all there.
Author: Éric Araujo (eric.araujo) *
Date: 2011-12-31 00:37
[...] There is slightly more versatility with the opener method, but no more obviousness and no less typing.
I agree with your opinion. I re-read this report:
- Antoine thinks this fills an important use case, namely avoiding race conditions
- Amaury then suggested the opener argument idea, which was implemented in the openat bug
- Antoine judged it would be better than what we had before
I don’t have a strong opinion on “opener is generic and good enough” vs. “not as ice as it could be”. Antoine seems to agree with you, so your patch will get reviewed and eventually accepted. Cheers
Author: Charles-François Natali (neologix) *
Date: 2012-01-05 17:47
I've done a small review.
Author: Charles-François Natali (neologix) *
Date: 2012-01-08 11:59
I intend to commit this patch within a couple days (unless anyone objects of course).
Author: Antoine Pitrou (pitrou) *
Date: 2012-01-08 14:36
I don't think the "created()" method has to be exposed. People can inspect the "mode" attribute if they want to have that information. (besides, the semantics are misleading since a new file opened with "w" has also be created, but created() would return False)
There's some bogus indentation in the patch (it uses tab characters in some places).
Also:
- if not (creating + reading or writing or appending):
Not sure why the "+". Shouldn't it be "or" instead?
Author: Charles-François Natali (neologix) *
Date: 2012-01-08 17:25
Here's a new version of the patch that should address all the comments.
Author: Antoine Pitrou (pitrou) *
Date: 2012-01-08 17:39
Here's a new version of the patch that should address all the comments.
Just a small note: FileExistsError is raised, not exactly OSError, when the file exists.
Author: Charles-François Natali (neologix) *
Date: 2012-01-08 19:43
Just a small note: FileExistsError is raised, not exactly OSError, when the file exists.
I've updated the doc accordingly.
Author: Roundup Robot (python-dev) 
Date: 2012-01-09 21:40
New changeset bf609baff4d3 by Charles-François Natali in branch 'default': Issue #12760: Add a create mode to open(). Patch by David Townshend. http://hg.python.org/cpython/rev/bf609baff4d3
Author: Charles-François Natali (neologix) *
Date: 2012-01-10 07:57
Committed. David, thanks for the patch!
Author: Charles-François Natali (neologix) *
Date: 2012-01-10 19:37
Nick suggested to call the new flag "exclusive create" in the doc (and explain in whatsnew that it's based C11 new 'x' flag). Could someone please check the attached patch? My wording sounds really clumsy, so I'd prefer if a native speaker could review it.
Author: David Townshend (David.Townshend)
Date: 2012-01-12 06:04
I've done a bit or rewording in the io docs, but I honestly don't know if this is any better that what you already had!
Author: Roundup Robot (python-dev)
Date: 2012-01-14 10:53
New changeset 8bcbe2dc3835 by Charles-François Natali in branch 'default': Issue #12760: Refer to the new 'x' open mode as "exclusive creation" mode. http://hg.python.org/cpython/rev/8bcbe2dc3835
Author: Charles-François Natali (neologix) *
Date: 2012-01-14 12:30
Thanks, I've committed your version.
Author: Petri Lehtinen (petri.lehtinen) *
Date: 2012-05-19 18:49
Shouldn't the documentation of builtin open() (in Doc/library/functions.rst) be updated too?
Author: Roundup Robot (python-dev)
Date: 2012-05-20 09:43
New changeset ef406c4c6463 by Charles-François Natali in branch 'default':
Issue #12760: Add some mising documentation about the new x exclusive
http://hg.python.org/cpython/rev/ef406c4c6463
History
Date
User
Action
Args
2022-04-11 14:57:20
admin
set
github: 56969
2012-05-20 09:43:36
python-dev
set
messages: +
2012-05-19 18:49:55
petri.lehtinen
set
nosy: + petri.lehtinen
messages: +
2012-01-14 12:30:00
neologix
set
messages: +
2012-01-14 10:53:45
python-dev
set
messages: +
2012-01-12 06:04:53
David.Townshend
set
files: + x_diff2.patch
messages: +
2012-01-10 19:37:32
neologix
set
files: + x_flag.diff
nosy: + ncoghlan
messages: +
2012-01-10 07:57:38
neologix
set
status: open -> closed
resolution: fixed
messages: +
stage: commit review -> resolved
2012-01-09 21:40:39
python-dev
set
nosy: + python-dev
messages: +
2012-01-08 19:43:17
neologix
set
messages: +
stage: patch review -> commit review
2012-01-08 17:39:12
pitrou
set
messages: +
2012-01-08 17:25:08
neologix
set
files: + open_create_x-3.patch
messages: +
2012-01-08 14:36:57
pitrou
set
messages: +
2012-01-08 11:59:58
neologix
set
messages: +
stage: patch review
2012-01-05 17:47:46
neologix
set
messages: +
2011-12-31 00:37:42
eric.araujo
set
messages: +
2011-12-30 22:07:18
Devin Jeanpierre
set
messages: +
2011-12-30 20:59:04
eric.araujo
set
messages: +
2011-12-27 19:04:30
cvrebert
set
nosy: + cvrebert
2011-12-27 17:11:34
pitrou
set
status: closed -> open
resolution: duplicate -> (no value)
superseder: io.FileIO and io.open should support openat ->
messages: +
2011-12-26 15:42:38
Devin Jeanpierre
set
nosy: + Devin Jeanpierre
messages: +
2011-11-18 09:55:28
eric.araujo
set
nosy: + eric.araujo
messages: +
2011-11-07 06:53:29
David.Townshend
set
messages: +
2011-10-31 10:32:11
amaury.forgeotdarc
set
messages: +
2011-10-31 10:22:03
David.Townshend
set
messages: +
2011-10-29 15:58:21
neologix
set
status: open -> closed
resolution: duplicate
superseder: io.FileIO and io.open should support openat
2011-09-08 23🔞43
Arfrever
set
nosy: + Arfrever
2011-08-24 21:34:40
vstinner
set
messages: +
2011-08-24 21:27:00
pitrou
set
messages: +
2011-08-24 21:19:02
amaury.forgeotdarc
set
nosy: + amaury.forgeotdarc
messages: +
2011-08-24 20:35:54
pitrou
set
messages: +
2011-08-24 20:17:21
neologix
set
messages: +
2011-08-24 20:12:39
pitrou
set
messages: +
2011-08-24 19🔞33
neologix
set
messages: +
2011-08-18 16:06:34
David.Townshend
set
files: + open_create_x-2.patch
messages: +
2011-08-18 15:16:33
vstinner
set
messages: +
2011-08-18 15:14:16
neologix
set
messages: +
2011-08-18 14:19:17
pitrou
set
messages: +
2011-08-18 14:15:58
Julian
set
nosy: + Julian
messages: +
2011-08-18 13:28:44
David.Townshend
set
messages: +
2011-08-18 13:24:52
neologix
set
messages: +
2011-08-18 11:56:53
David.Townshend
set
messages: +
2011-08-18 11:33:55
pitrou
set
nosy: + pitrou
messages: +
assignee: docs@python ->
components: - Documentation, Tests
2011-08-16 15:43:20
David.Townshend
set
messages: +
2011-08-16 15:37:05
neologix
set
nosy: + neologix
messages: +
2011-08-16 14:53:24
benjamin.peterson
set
nosy: + benjamin.peterson
messages: +
2011-08-16 13:57:33
vstinner
set
nosy: + vstinner
messages: +
2011-08-16 13:49:18
David.Townshend
create