Issue 17797: Visual C++ 11.0 reports fileno(stdin) == 0 for non-console program (original) (raw)
Created on 2013-04-19 12:18 by mloskot, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (41)
Author: Mateusz Loskot (mloskot) *
Date: 2013-04-19 12:18
In pythonrun.c, there is function initstdio() with the following test:
static int initstdio(void) { ... /* Set sys.stdin / fd = fileno(stdin); / Under some conditions stdin, stdout and stderr may not be connected * and fileno() may point to an invalid file descriptor. For example * GUI apps don't have valid standard streams by default. */ if (fd < 0) { #ifdef MS_WINDOWS std = Py_None; Py_INCREF(std); #else goto error; #endif } else { ... }
This function is fails for non-console applications (i.e. MFC) built using Visual C++ 11.0 (Visual Studio 2012), becasue strangely, fileno(stdin) == 0, so this test results in false and Python initialisation routines attempt to setup streams.
Apparently, fileno(stdin) return value has changed between Visual C++ 10.0 (VS 2010) and 11.0. The VC++ 10.0 reports fileno(stdin) == -2.
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * 
Date: 2013-04-19 12:24
And does it cause an issue later? How?
Author: Mateusz Loskot (mloskot) *
Date: 2013-04-19 12:32
Yes, it does. In file Modulfileio.c, in function fileio_init, there is this code:
if (fd >= 0) {
if (check_fd(fd))
goto error;
self->fd = fd;
self->closefd = closefd;
}The check_fd tests:
if (!_PyVerify_fd(fd) || (fstat(fd, &buf) < 0 && errno == EBADF)) {
The _PyVerify_fd(fd) == 1, but errno is "Bad file descriptor".
This eventually leads to Py_InitializeEx failure at:
if (initstdio() < 0)
Py_FatalError(
"Py_Initialize: can't initialize sys standard streams");Author: Mateusz Loskot (mloskot) *
Date: 2013-04-19 12:34
In file Modulfileio.c,
I messed the path and filename above I meant: In file Modules/_io/fileio.c,
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *
Date: 2013-04-19 12:49
Maybe check_fd(fd) could be used in initstdio as well.
Can you check whether it's the same for the other files? What are the values for fileno(stdout) and fileno(stderr)?
Author: Mateusz Loskot (mloskot) *
Date: 2013-04-19 14:09
Replacing if the current test in Python 3.2
if (fd < 0) with if (check_fd(fd) < 0)
Seems to be a working solution.
I just noticed, that in current Python/pythonrun.c in the repo, there the fd < 0 tests have been replaced with new function is_valid_fd(). But, its semantic is equivalent to fd < 0, so it does not check anything really. Perhaps is_valid_fd could be redefined as check_fd.
Here are Visual C++ 11.0 (1700) vs 10.0 differences of fileno return value for all the standard streams
int fdi, fdo, fde;
fdi = fileno(stdin);
fdo = fileno(stdout);
fde = fileno(stderr);#if _MSC_VER < 1700 assert(fdi == -2); assert(fdo == -2); assert(fde == -2); #else assert(fdi == 0); assert(fdo == 1); assert(fde == 2); #endif
By the way, I assume such sudden change in fileno(std*) behaviour between Visual C++ versions is suspicious, so I also submitted bug report to Visual Studio: https://connect.microsoft.com/VisualStudio/feedback/details/785119/
Author: Mateusz Loskot (mloskot) *
Date: 2013-04-29 23:21
I've just got an update on the bug report [1] I submitted to Microsoft. The Visual C++ team confirmed "It does appear to be a regression from Visual Studio 2010."
So, it's not a bug in Python, but I think it may be important for Python to consider applying some reliable workaround (i.e. use of check_fd() function)
[1] http://connect.microsoft.com/VisualStudio/feedback/details/785119/
Author: V.E.O (V.E.O)
Date: 2013-07-06 21:30
Tested on MSVCRT110.DLL (11.0.51106.1 released in 2012/11/05). Previous one 11.0.50727.1 released in 2012/07/26 does not have this problem. This problem better be fixed in Python for MS does not have clear document.
Author: Christian Heimes (christian.heimes) *
Date: 2013-07-06 22:54
Python 3.2 is in security fix mode. Bug fixes such as this issue don't count as security fix. Are you able to update to Python 3.3? Python 2.7 has a different implementation and uses FILE* instead of file descriptors.
Author: V.E.O (V.E.O)
Date: 2013-07-07 09:35
Hi Christian,
The latest runtime Microsoft provided is buggy. Tried fix the PyVerify_fd with more GetFileType verification. But a lot more problems came for isatty returns true in non-console program. The fix in Python side shall be large. Details is reported to Microsoft, we can only hope they make it right.
Author: Mateusz Loskot (mloskot) *
Date: 2013-07-08 08:45
This is still an issue in VS 2012 Version 11.0.60610.01 Update 3
Author: Daniel (m_python)
Date: 2013-09-10 13:37
After contacting Microsoft they answered that they cannot fix that within the VS2012 or VS2013 cycle. Very bad. Any ideas?
Author: Daniel (m_python)
Date: 2013-09-11 06:56
Here are some solutions which might help you until MS fixed the bug.
Even if the subsystem is Windows one solution is to call AllocConsole() before Py_Initialize() to create a console.
Second Solution: If you don't want to open a console and your application has its own output window you could redirect stdout/stderr/stdin with freopen to a temp file first. (e.g: freopen("file.txt","w",stdout);) Call freopen for all std handles before you call Py_Initialize(). You can keep the redirection to the file or after Py_Initialize() succeeded you change the redirection now in Python by redirect sys.stdout to the place you want.
Author: Antoine Pitrou (pitrou) *
Date: 2013-09-11 09:26
Which 3.2 version does this break with? As far as I can tell, it should be fixed in 3.2.3: see changeset f15943505db0, issue #7111.
Author: Mateusz Loskot (mloskot) *
Date: 2013-09-11 10:04
@Antoine (Re )
I'm not sure which minor version of Python 3.2 it was, but in my comment in , I confirmed that I have tested the later version with added is_valid_fd function. As far as I understand, the changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce that is_valid_fd function, which does not seem to solve the problem.
Author: Daniel (m_python)
Date: 2013-09-11 10:36
Sorry, I used the latest Python3.4 branch, I haven't tested this with prior versions.
Author: Antoine Pitrou (pitrou) *
Date: 2013-09-11 11:34
I'm not sure which minor version of Python 3.2 it was, but in my comment in , I confirmed that I have tested the later version with added is_valid_fd function. As far as I understand, the changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce that is_valid_fd function, which does not seem to solve the problem.
Could you try to investigate this a bit further? Does is_valid_fd() return true, and why?
Author: Mateusz Loskot (mloskot) *
Date: 2013-09-11 16:01
On 11 September 2013 12:34, Antoine Pitrou <report@bugs.python.org> wrote:
I'm not sure which minor version of Python 3.2 it was, but in my comment in , I confirmed that I have tested the later version with added is_valid_fd function. As far as I understand, the changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce that is_valid_fd function, which does not seem to solve the problem.
Could you try to investigate this a bit further? Does is_valid_fd() return true, and why?
I have already checked that (see )
Shortly, is_valid_fd always returns true because fd < 0 is always false as my tests with Visual Studio 2012 (Visual C++ 11.0 (1700) confirmed (see also bug report linked in )
assert(fdi == 0);
assert(fdo == 1);
assert(fde == 2);IOW, fd is always larger than 0.
Author: Antoine Pitrou (pitrou) *
Date: 2013-09-11 23:06
Shortly, is_valid_fd always returns true because fd < 0 is always false as my tests with Visual Studio 2012
Well, that's not all there is, is_valid_fd() does other checks before returning true.
Author: Mateusz Loskot (mloskot) *
Date: 2013-09-11 23:49
On 12 September 2013 00:06, Antoine Pitrou <report@bugs.python.org> wrote:
Shortly, is_valid_fd always returns true because fd < 0 is always false as my tests with Visual Studio 2012
Well, that's not all there is, is_valid_fd() does other checks before returning true.
Given the function:
is_valid_fd(int fd) { int dummy_fd; if (fd < 0 || !_PyVerify_fd(fd)) return 0; dummy_fd = dup(fd); if (dummy_fd < 0) return 0; close(dummy_fd); return 1; }
for fd values of 0, 1 or 2
- fd < 0 is always false
- _PyVerify_fd(fd) is always true. Given the current definition:
#define _PyVerify_fd(fd) (_get_osfhandle(fd) >= 0) for those values of fd _get_osfhandle(fd) >= 0, always. 3. for those fd values, dup() never returns fd < 0
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *
Date: 2013-09-12 18:35
- _PyVerify_fd(fd) is always true. Given the current definition:
#define _PyVerify_fd(fd) (_get_osfhandle(fd) >= 0) for those values of fd _get_osfhandle(fd) >= 0, always.
Hum, are you sure this is the selected implementation?
- this code is only in 2.7 http://hg.python.org/cpython/file/v2.7/Include/fileobject.h#l73
- it's protected by #if statements. In your case the first case should be selected, can you verify if it's the case, and if not, why?
Author: Mateusz Loskot (mloskot) *
Date: 2013-09-30 10:57
I have just tested Windows GUI application built against Python 3.2.1 with is_valid_fd patch according to http://hg.python.org/cpython/rev/f15943505db0/
All built using VS2012.
Again, I can confirm that is_valid_fd does NOT solve the problem. Here is extract of execution flow in initstdio function:
- fd = 0, despite it is GUI app, see https://connect.microsoft.com/VisualStudio/feedback/details/785119/
fd = fileno(stdin);
- is_valid_fd will return true, so it moves to calling create_stdio()
if (!is_valid_fd(fd)) { ... } else { std = create_stdio(iomod, fd, 0, "", encoding, errors); if (std == NULL) goto error; }
- The create_stdio() call fails though, causing error followed by abort
Still, the only solution that solves this problem in Windows GUI applications buitl using VS2012 is to use check_fd() function to check fd, instead of is_valid_fd(). The check_fd() is more reliable as it will return -1 due to errno == EBADF.
My previous attempt to analyse _PyVerify_fd() is not relevant for the problem, let's forget it.
Author: (eaducac)
Date: 2013-11-02 05:32
Hi all,
I ran into this problem myself today with 3.3.2 and I thought I'd provide some more detail on what's going on. As has been described, prior to VC11, fileno(stdin) returned -2 in applications with no console, so is_valid_fd() was properly rejecting it and the standard streams were being set to None. With VC11, it now returns 0, and the NT handle that underlies the FILE object associated with file descriptor 0 is -2. The problem is that -2 is a pseudo-handle representing the current thread. dup() is simply a wrapper around the Windows API function DuplicateHandle(), so when dup() is called on file descriptor 0, it returns success because DuplicateHandle() is successfully creating a new handle referring to the current thread. It's not until the call to fstat() in check_fd() much later that the CRT realizes it's not dealing with a file.
I see in Mercurial that 3.3.4 and 3.4 that the is_valid_fd() code hasn't been changed. This is definitely a Microsoft bug, but if (as it sounds) they're not going to be able to fix it anytime soon, I think it would be ideal to have a change for this in Python. Replacing this with check_fd() would fix it, or if you don't want to rely on fstat(), some Windows-specific code using GetStdHandle():
#ifdef MS_WINDOWS if (!is_valid_fd(fd) || GetStdHandle(STD_INPUT_HANDLE) == NULL) { #else if (!is_valid_fd(fd)) { #endif
That would have to be repeated for stdout and stderr using GetStdHandle(STD_OUTPUT_HANDLE) and GetStdHandle(STD_ERROR_HANDLE), as those descriptors have the same problem.
Author: Mateusz Loskot (mloskot) *
Date: 2014-06-21 11:33
FYI, I've got it confirmed fix for the bug in VS has been released [1]
""" We have fixed this behavior for the next major release, Visual Studio "14." The fix is present in the Visual Studio "14" CTP that was released earlier this month. """
Author: Tomasz Wasilczyk (twasilczyk@spoon) *
Date: 2015-01-29 13:15
It seems that bug remains not fixed.
Just like eaducac pointed out, there are cases when underlying file handle is -2, which is not a valid file handle, but is a valid handle (which is GetCurrentThread).
Thus, provided dup solution accepts it, while it shouldn't. However, suggested GetStdHandle check still doesn't work for me. While working on the related Spoon issue, I've worked out a working solution (see the patch attached).
Tomek, Spoon.net dev
Author: STINNER Victor (vstinner) *
Date: 2015-01-29 13:16
@steve.dower: Do you have info on this issue?
Author: Steve Dower (steve.dower) *
Date: 2015-01-29 14:30
This is fixed in VS2015 last time I tried it. I've proposed workarounds for this before, but it's never affected any official build do there's never been any traction. The fstat patch is normally the check that I use.
I don't know what Tomasz means that it's not fixed. Fixes like this don't magically appear in earlier VS versions, and while VS2015 does have the fix, it's not quite ready for production work.
Author: Tomasz Wasilczyk (twasilczyk@spoon) *
Date: 2015-01-29 19:18
I don't mean fixing VS, but providing a workaround in Python code, when compiled with VS2012.
I know newer VS versions are fixed. Do you mean, VS2012 is not supported? If it's not - then LibreOffice team have a problem, because their official release is built with this suite.
If it is supported, and you don't want to alter behavior for other versions, then a patch may also contain an ifdef for MSVC 11.0.
Author: Steve Dower (steve.dower) *
Date: 2015-01-29 21:07
It's not supported for building Python, is what I meant. I don't know what the status is for Microsoft support of VS2012, but I'd expect it's security fix only by now.
Being able to support building Python with alternative compilers is a task that we don't have the volunteers to support, especially on Windows. That said, I'm not against taking patches like this to improve it, but I'd expect they'll be treated as an enhancement and won't make it into any version before 3.4.
Author: Mark Lawrence (BreamoreBoy) *
Date: 2015-01-29 21:34
An enhancement can only go into 3.5, not 3.4. This has to happen before beta 1 which is currently scheduled for May 24 2015. I'm against the idea as it would create too many compatibility issues.
Author: Tomasz Wasilczyk (twasilczyk@spoon) *
Date: 2015-01-29 22:06
In such case, I agree it's an "enhancement", not a "bugfix". Could go into 3.5 branch.
How about changing the ifdef to the following, to avoid any issues on non-MSVS2012:
#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER == 1700
Author: Jean-Charles Lefebvre (polyvertex) *
Date: 2015-03-12 22:12
Hi all, a small update to confirm this issue with version 3.5.0a2 embedded in a native C++ GUI application, having everything built with VS2013 SP4.
Same execution flow as described by Mateusz in .
I've modified Python/pylifecycle.c file according to Tomasz's attached patch. It just "works for me" and I've just slightly modified the #ifdef line:
#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER >= 1700 && _MSC_VER < 2000
It would definitely be nice to have this workaround applied to the 3.5 branch since MS' fix is lost in release cycle. Would it be too intrusive?
Author: Jean-Charles Lefebvre (polyvertex) *
Date: 2015-03-12 22:14
Oops, mistyping, #ifdef test would be:
#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER >= 1700 && _MSC_VER < 1900
Author: Steve Dower (steve.dower) *
Date: 2015-03-12 23:13
There's not much point putting this into 3.5 anymore, as we are going to break pre-VS2015 builds completely sooner or later (too many changes to the CRT that we can't just work around with #ifdefs - and too much opposition to having code purely to support unsupported compilers).
It's probably only worth putting this into 3.4 at this stage, which I'm okay with but I'm not going to push hard to make it happen. Also worth noting that the code here in 3.5 has diverged from 3.4, and so the attached patch doesn't apply.
Any of the other core devs strongly opposed to me updating 3.4 so people embedding that version can have the fix?
Author: Jean-Charles Lefebvre (polyvertex) *
Date: 2015-03-13 09:17
Well, just in case, I've attached the patch to apply against 3.5.0a2.
Author: Mateusz Loskot (mloskot) *
Date: 2015-05-20 14:25
Re , I confirm python-3.5.0a2-fdvalidation.patch fixes the problem for Python 3.5.0a4 and VS2013.
The only issue I encountered was with HAVE_FSTAT which is missing from PC/pyconifg.h, so I edited the patch and removed the check if defined(HAVE_FSTAT). Does PC/pyconifg.h need update?
I also confirm vanilla Python 3.5.0a4 with VS2015RC does not require this patch, because the fileno regression/bug has been fixed in VS2015RC (details at https://connect.microsoft.com/VisualStudio/feedback/details/785119/).
Author: Ilya Kulakov (Ilya.Kulakov) *
Date: 2015-08-15 16:01
I see issue to be fixed but patch wasn't applied yet. Is it still supposed to be included in 3.5 or there is something wrong?
Author: Steve Dower (steve.dower) *
Date: 2015-08-15 18:50
Rereading the discussion, there seems to be agreement that this is an enhancement. It does not apply for Python 3.5 (which requires a compiler without the bug that causes this), and Python 3.4 is no longer eligible for enhancements. I'm closing this as out of date.
The patches will be here, so anyone who is rebuilding Python 3.4 or earlier for their own use with VS 2012 is welcome to apply the patch themselves.
Author: Ilya Kulakov (Ilya.Kulakov) *
Date: 2015-08-23 19:44
Steve,
What's going to be the required msvc compiler for 3.5 on Windows?
Author: Eryk Sun (eryksun) * 
Date: 2015-08-23 21:42
The 3.5 build uses MSVC 14 (VS 2015):
https://docs.python.org/3.5/using/windows.html#compiling-python-on-windows https://hg.python.org/cpython/file/3.5/PCbuild/readme.txt
Author: Sophie (sophieSurmont)
Date: 2016-06-06 14:01
In case someone else need it, here is a patch for python 3.5.0b1
History
Date
User
Action
Args
2022-04-11 14:57:44
admin
set
github: 61997
2016-06-08 05:23:31
BreamoreBoy
set
nosy: - BreamoreBoy
2016-06-06 14:01:30
sophieSurmont
set
files: + python-3.5.0b1-fdvalidation.patch
versions: + Python 3.5, - Python 3.2, Python 3.3, Python 3.4
nosy: + sophieSurmont
messages: +
2015-08-23 21:42:55
eryksun
set
nosy: + eryksun
messages: +
2015-08-23 19:44:17
Ilya.Kulakov
set
messages: +
2015-08-15 18:50:01
steve.dower
set
status: open -> closed
versions: - Python 3.5
type: enhancement
messages: +
resolution: fixed -> out of date
2015-08-15 16:01:21
Ilya.Kulakov
set
nosy: + Ilya.Kulakov
messages: +
2015-07-03 19:10:38
steve.dower
link
2015-05-20 14:25:40
mloskot
set
messages: +
2015-03-13 09:17:04
polyvertex
set
files: + python-3.5.0a2-fdvalidation.patch
messages: +
2015-03-12 23:13:11
steve.dower
set
messages: +
2015-03-12 22:14:10
polyvertex
set
messages: +
2015-03-12 22:12:07
polyvertex
set
nosy: + polyvertex
messages: +
versions: + Python 3.5
2015-01-29 22:06:48
twasilczyk@spoon
set
messages: +
2015-01-29 21:34:18
BreamoreBoy
set
nosy: + BreamoreBoy
messages: +
2015-01-29 21:07:58
steve.dower
set
messages: +
2015-01-29 19🔞52
twasilczyk@spoon
set
messages: +
2015-01-29 14:30:32
steve.dower
set
messages: +
2015-01-29 13:16:56
vstinner
set
nosy: + steve.dower
messages: +
2015-01-29 13:15:32
twasilczyk@spoon
set
files: + python-3.3.5-fdvalidation.patch
nosy: + twasilczyk@spoon
messages: +
keywords: + patch
2014-08-18 14:16:42
brian.curtin
set
nosy: - brian.curtin
2014-08-18 14:14:15
mont29
set
nosy: + mont29
2014-06-21 11:33:52
mloskot
set
messages: +
2014-05-06 02🔞56
Luke.Dunstan
set
nosy: + Luke.Dunstan
versions: + Python 3.4
2013-11-02 05:32:24
eaducac
set
nosy: + eaducac
messages: +
versions: + Python 3.3
2013-09-30 10:57:10
mloskot
set
messages: +
2013-09-12 18:35:51
amaury.forgeotdarc
set
messages: +
2013-09-11 23:49:27
mloskot
set
messages: +
2013-09-11 23:06:19
pitrou
set
messages: +
2013-09-11 16:01:54
mloskot
set
messages: +
2013-09-11 11:34:50
pitrou
set
messages: +
2013-09-11 10:36:56
m_python
set
messages: +
2013-09-11 10:04:29
mloskot
set
status: pending -> open
messages: +
2013-09-11 09:26:17
pitrou
set
status: open -> pending
versions: + Python 3.2, - Python 3.3, Python 3.4
nosy: + pitrou
messages: +
resolution: fixed
stage: needs patch ->
2013-09-11 09:21:20
pitrou
set
priority: normal -> high
stage: needs patch
resolution: out of date -> (no value)
versions: + Python 3.3, Python 3.4, - Python 3.2
2013-09-11 09:20:52
pitrou
set
nosy: + tim.peters, tim.golden, brian.curtin
2013-09-11 06:56:08
m_python
set
messages: +
2013-09-10 13:37:48
m_python
set
nosy: + m_python
messages: +
2013-07-08 08:45:40
mloskot
set
messages: +
2013-07-07 09:35:10
V.E.O
set
status: pending -> open
messages: +
2013-07-06 22:54:04
christian.heimes
set
status: open -> pending
nosy: + christian.heimes
messages: +
components: + Windows
resolution: out of date
2013-07-06 21:30:20
V.E.O
set
nosy: + V.E.O
messages: +
2013-04-29 23:37:27
vstinner
set
nosy: + vstinner
2013-04-29 23:21:30
mloskot
set
messages: +
2013-04-19 14:09:25
mloskot
set
messages: +
2013-04-19 12:49:12
amaury.forgeotdarc
set
messages: +
2013-04-19 12:34:09
mloskot
set
messages: +
2013-04-19 12:32:45
mloskot
set
messages: +
2013-04-19 12:24:20
amaury.forgeotdarc
set
nosy: + amaury.forgeotdarc
messages: +
2013-04-19 12🔞32
mloskot
create