| msg78102 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2008-12-20 13:47 |
| I like and I need an "unbuffered" standard output which was provided by -u command line option (or PYTHONUNBUFFERED environment variable). Current status of -u option in Python3: the option exists and change the buffer size (disable buffering) of the stdin, stdout and stderr file descriptors. The problem is in initstdio() which creates files with buffering=-1 (default buffer) instead of buffering=0 (no buffering) or buffering=1 (line buffer). But open() enable line buffering of TextIOWrapper is buffering=-1 and the raw file is a tty. Example with py3k trunk: ------------ $ ./python >>> import sys; sys.stdout.line_buffering True $ ./python |cat >>> import sys; sys.stdout.line_buffering False ------------ I would like line buffering when stdout is redirected to a pipe and -u option is used. initstdio() have to be changed to choose buffering option. So it's something like: Index: Python/pythonrun.c =================================================================== --- Python/pythonrun.c (révision 67870) +++ Python/pythonrun.c (copie de travail) @@ -810,7 +810,12 @@ #endif } else { - if (!(std = PyFile_FromFd(fd, "", "w", -1, encoding, + int buffering; + if (1) + buffering = 1; /* line */ + else + buffering = -1; /* default */ + if (!(std = PyFile_FromFd(fd, "", "w", buffering, encoding, errors, "\n", 0))) { goto error; } But "if (1)" have to be replaced "if -u option is used" :-) See unbuffered variable of Modules/main.c. |
|
|
| msg78127 - (view) |
Author: Fabio Zadrozny (fabioz) * |
Date: 2008-12-20 21:51 |
| Just as a note, Pydev needs the unbuffered output (or it cannot get it). This has been brought up in the python-dev list: http://mail.python.org/pipermail/python-dev/2008-December/084436.html As a workaround for now I'm using: sys.stdout._line_buffering = True, but that doesn't seem right as it's accessing an internal attribute. |
|
|
| msg78387 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-12-27 22:49 |
| Here is a patch. |
|
|
| msg78393 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2008-12-28 01:23 |
| pitrou's patch changes PyFile_FromFd() behaviour for a text file opened with buffering=0: /* As a convenience, when buffering == 0 on a text file, we open the underlying binary stream in unbuffered mode and wrap it with a text stream in line-buffered mode. */ Why changing PyFile_FromFd() and not io.open() directly? Note: I prefer Py_UnbufferedStdoutFlag=1 instead of Py_UnbufferedStdoutFlag++ (for -u command line option). Except the minor comments, I like the patch (and it has unit tests!) ;-) |
|
|
| msg78395 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-12-28 10:27 |
| > pitrou's patch changes PyFile_FromFd() behaviour for a text file > opened with buffering=0: > /* As a convenience, when buffering == 0 on a text file, we > open the underlying binary stream in unbuffered mode and > wrap it with a text stream in line-buffered mode. */ > > Why changing PyFile_FromFd() and not io.open() directly? I must admit I'm a bit lazy, and changing io.open() means changing a fundamental public API, as Guido said on python-dev, so more discussion and some parts of the patches delayed to 3.1. If someone else wants to do it, please don't hesitate... > Note: I prefer Py_UnbufferedStdoutFlag=1 instead of > Py_UnbufferedStdoutFlag++ (for -u command line option). Well, I minimally changed the existing code. |
|
|
| msg78400 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2008-12-28 12:19 |
| > > Why changing PyFile_FromFd() and not io.open() directly? > > I must admit I'm a bit lazy, and changing io.open() means changing > a fundamental public API, as Guido said on python-dev, so > more discussion and some parts of the patches delayed to 3.1. You're right, and PyFile_FromFd() is also a fundamental "public" API. Since TextIOWrapper doesn't support real unbuffered buffer (only pseudo line buffer: unbuffered raw buffer and line buffering for TextIOWrapper), I prefer to change only stdout/stderr instead of PyFile_FromFd(). My new patch only changes initstdio() using pitrou's code. Should we also change stdin? |
|
|
| msg78403 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-12-28 13:54 |
| Le dimanche 28 décembre 2008 à 12:19 +0000, STINNER Victor a écrit : > STINNER Victor <victor.stinner@haypocalc.com> added the comment: > > > > Why changing PyFile_FromFd() and not io.open() directly? > > > > I must admit I'm a bit lazy, and changing io.open() means changing > > a fundamental public API, as Guido said on python-dev, so > > more discussion and some parts of the patches delayed to 3.1. > > You're right, and PyFile_FromFd() is also a fundamental "public" API. Well, open() is fundamental as in part of the built-ins and used pervasively. PyFile_FromFd(), on the other hand, is a relic of the 2.x C file handling API. Let's see what others have to say about this. > Should we also change stdin? I don't know, but "python -h" only talks about stderr/stdout. |
|
|
| msg78408 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-12-28 15:11 |
| It seems the "name" field of the TextIOWrapper object isn't set in create_stdio() (the "char *name" parameter isn't used). Otherwise, the patch looks good. |
|
|
| msg78417 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2008-12-28 16:04 |
| >> Should we also change stdin? > I don't know, but "python -h" only talks about stderr/stdout. The manpage of Python2 is clear: -u Force stdin, stdout and stderr to be totally unbuffered. stdin is also unbuffered. > It seems the "name" field of the TextIOWrapper object isn't > set in create_stdio() It used only used for buffered output. Without the patch, sys.stdout.name == sys.stdout.buffer.name == '1' :-/ New patch: - use create_stdio() to create unbuffered sys.stdin - rename Py_UnbufferedStdoutFlag to Py_UnbufferedStdioFlag - replace "Py_UnbufferedStdioFlag++;" by "Py_UnbufferedStdioFlag = 1;" - change create_stdio(): (...) Note: there is no test for unbuffered input because I don't know how to test this (even by manual tests) :-p |
|
|
| msg78420 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2008-12-28 16:57 |
| Attached: quick and dirty test to check if the standard input is buffered or not. My short test program works with python2.5 and py3k trunk without the -u command line option. So changing sys.stdin buffer is not really important. About the wrong name, I opened a separated issue: #4762, PyFile_FromFd() doesn't set the file name. |
|
|
| msg78886 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2009-01-02 21:51 |
| Instead of importing IO each time in create_stdio, maybe you should just pass io.open to create_stdio. |
|
|
| msg78922 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2009-01-03 01:40 |
| > Instead of importing IO each time in create_stdio, > maybe you should just pass io.open to create_stdio create_stdio() uses io.open() but also io.TextIOWrapper. Since io module is already imported in initstdio(), I updated the patch to just pass the pointer to the module to create_stdio(). |
|
|
| msg79444 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-08 21:37 |
| If `PyObject_SetAttrString(raw, "_name", text)` fails, a reference to raw is leaked. Other than that, the patch looks good. |
|
|
| msg79449 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2009-01-08 23:18 |
| Updated patch: clear raw on error + if (!Py_UnbufferedStdioFlag) + Py_XDECREF(raw); Question: Should we use line_buffering in unbuffered mode? |
|
|
| msg79490 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-09 19:15 |
| Committed in r68451. Thanks! |
|
|
| msg80190 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-19 16:27 |
| Reopening, since sys.stdin is actually broken in unbuffered mode: $ ./python -u Python 3.1a0 (py3k:68756, Jan 19 2009, 01:17:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.stdin.read(1) Traceback (most recent call last): File "", line 1, in File "/home/antoine/py3k/__svn__/Lib/io.py", line 1739, in read eof = not self._read_chunk() File "/home/antoine/py3k/__svn__/Lib/io.py", line 1565, in _read_chunk input_chunk = self.buffer.read1(self._CHUNK_SIZE) AttributeError: 'FileIO' object has no attribute 'read1' >>> What I propose is that stdin be always opened in buffered mode (even with -u), since I don't see how the behaviour can differ for a read-only non-seekable stream. |
|
|
| msg80195 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-19 17:15 |
| Here is a patch. |
|
|
| msg80539 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-25 21:25 |
| > since I don't see how the behaviour can differ for a read-only > non-seekable stream. Unless I'm misunderstanding you (quite likely), I think one *can* get different results with buffered and unbuffered stdin. For example, on my machine, if I create the following script: #!/usr/bin/python -u import sys print sys.stdin.readline() and name it test.py, I get the following result in an OS X Terminal running bash: dickinsm$ ls python_source/trunk/Objects/ | (./test.py; ./test.py) abstract.c boolobject.c Whereas if I remove the '-u' from the shebang line I just get: dickinsm$ ls python_source/trunk/Objects/ |
(./test.py; ./test.py) abstract.c I'm not 100% sure that I understand exactly what's going on here, but it's something like the following: in the first (unbuffered) case, the stdin.readline call of the first ./test.py only reads one line from stdin, leaving the rest intact; so the second ./test.py also gets to output a line. In the second case some larger amount of stdin (1024 bytes?) is immediately slurped into the stdin buffer for the first Python process, so the second ./test.py doesn't get anything. |
|
| msg80540 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-25 21:33 |
| [...] I hadn't thought of such situations :-/ So the question is whether it is really useful to enforce unbuffered stdin with the '-u' option (or your example is simply too borderline). If so, the patch will have to be replaced with another one implementing read1() in the FileIO class. |
|
|
| msg80541 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-25 21:36 |
| Thinking about it, TextIOWrapper has its own input buffering (the `decoded_chars` attribute), so your use case would probably not be satisfied. (and disabling TextIOWrapper's internal buffering would be a bad idea since it would make it horribly slow) |
|
|
| msg80543 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-25 22:00 |
| > So the question is whether it is really useful to enforce unbuffered > stdin with the '-u' option (or your example is simply too borderline). Hard to say. It seems at least possible that there are Python users for whom stdin being unbuffered (with -u) matters, so if there's any reasonable way of avoiding changing this it should probably be considered. Though I have to admit that I'm not one of those users (I don't think I've *ever* used the -u option outside of testing...). |
|
|
| msg80544 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-25 22:10 |
| > Hard to say. It seems at least possible that there are Python users for > whom stdin being unbuffered (with -u) matters, so if there's any > reasonable way of avoiding changing this it should probably be considered. It's not about changing it, stdin has always been buffered in py3k. My original commit actually attempted to change it, and it failed (I hadn't noticed it at first due to mis-testing on my part). The new patch is about putting it back in buffered mode even with '-u'. |
|
|
| msg80575 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-26 16:05 |
| > It's not about changing it, stdin has always been buffered in py3k. Sorry: I should have been clearer. It's the change from 2.x to 3.x that interests me. So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not; I'm wondering: was this an intentional design change? Or was it just an accident/by-product of the rewritten io? Anyway, the patch looks good to me. |
|
|
| msg80590 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-26 20:49 |
| > So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not; > I'm wondering: was this an intentional design change? Or was it just > an accident/by-product of the rewritten io? I'm not sure (I didn't write the new io in the first place) but I'd say it was simply overlooked. Otherwise 'python3.0 -u' would have had at least unbuffered stdout/stderr, which it didn't have. |
|
|
| msg80597 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-26 22:09 |
| Committed and applied a small fix to the test so that it passes in debug mode (r68977, r68981, r68982). Thanks! |
|
|
| msg196224 - (view) |
Author: Joe Borg (Joe.Borg) |
Date: 2013-08-26 16:45 |
| Can I confirm this is still in the trunk? I have 3.3.2 and am suffering from the fact that `-u` isn't setting stdin to unbuffered. I'm have to run a flush every command, which is awful. |
|
|