| msg190458 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2013-06-01 23:50 |
| [0] nikratio@vostro:~/tmp$ cat bugme.py #!python import getpass import warnings warnings.simplefilter('default') getpass.getpass("What's up?") [0] nikratio@vostro:~/tmp$ python3 --version Python 3.3.2 [0] nikratio@vostro:~/tmp$ python3 bugme.py /usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> return io.open(fd, *args, **kwargs) What's up? |
|
|
| msg190459 - (view) |
Author: Alex Gaynor (alex) *  |
Date: 2013-06-01 23:56 |
| Attached patch should fix this issue. |
|
|
| msg190460 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2013-06-02 00:01 |
| No, it doesn't. |
|
|
| msg190461 - (view) |
Author: Alex Gaynor (alex) * |
Date: 2013-06-02 00:04 |
| Are you sure you applied it correctly? With and without: Alexanders-MacBook-Pro:cpython alex_gaynor$ ./python.exe x.py What's up? Alexanders-MacBook-Pro:cpython alex_gaynor$ hg revert --all --no-backup reverting Lib/getpass.py Alexanders-MacBook-Pro:cpython alex_gaynor$ ./python.exe x.py What's up? x.py:6: ResourceWarning: unclosed file <_io.TextIOWrapper name=3 mode='w+' encoding='UTF-8'> getpass.getpass("What's up?") |
|
|
| msg190462 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2013-06-02 00:24 |
| Yes, I'm pretty sure: [0] nikratio@vostro:~/tmp$ cp /usr/lib/python3.3/getpass.py . [0] nikratio@vostro:~/tmp$ patch -p2 < .diff patching file getpass.py Hunk #1 succeeded at 57 (offset -1 lines). [0] nikratio@vostro:~/tmp$ python3 bugme.py /usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> return io.open(fd, *args, **kwargs) What's up? # Test if we're using the patched getpass.py... [0] nikratio@vostro:~/tmp$ vim getpass.py [0] nikratio@vostro:~/tmp$ python3 bugme.py Hello /usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> return io.open(fd, *args, **kwargs) What's up? |
|
|
| msg190468 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-02 03:26 |
| This bug happens in Python 3.4 as well. [sky@localhost cpython]$ ./python --version Python 3.4.0a0 [sky@localhost cpython]$ ./python /tmp/bugme.py /home/sky/Code/python/programming_language/cpython/Lib/os.py:1025: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> return io.open(fd, *args, **kwargs) What's up? I tried to apply the patch manually (by copying, cutting and pasting) from Alex but Nikolaus is right. The patch does not work. The bug still happens |
|
|
| msg190471 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-02 07:24 |
| I isolate the bug. It happens in these lines: # Always try reading and writing directly on the tty first. fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY) tty = os.fdopen(fd, 'w+', 1) So to produce the bug more specifically, you can try this python file: # bugme2.py import os fd = os.open('/dev/tty', os.O_RDWR |
os.O_NOCTTY) os.fdopen(fd, 'w+', 1) # end of bugme2.py In Linux Fedora 18, I would get this error: /home/sky/Code/python/programming_language/cpython/Lib/os.py:1025: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> return io.open(fd, *args, **kwargs) Traceback (most recent call last): File "/tmp/bugme2.py", line 4, in os.fdopen(fd, 'w+', 1) File "/home/sky/Code/python/programming_language/cpython/Lib/os.py", line 1025, in fdopen return io.open(fd, *args, **kwargs) io.UnsupportedOperation: File or stream is not seekable. |
|
| msg190472 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-02 07:55 |
| I have investigated this problem and come up with the patch to fix the problem. This patch does the job. Caution: only for Python 3.4. But translating this patch to Python 3.3 should be straightforward. I hope this patch could be the foundation for better programmers to create better patch. Some of the issues with this patch are: I am not sure how to handle encoding and where the best place to close tty is. Reference: https://github.com/stefanholek/term/issues/1 http://stackoverflow.com/questions/5471158/typeerror-str-does-not-support-the-buffer-interface |
|
|
| msg190473 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-02 08:09 |
| Sorry, My previous patch breaks the test. This one should pass the test and fix the bug. Still, there are ugly code in the patch that I hope better programmers could fix. |
|
|
| msg190520 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2013-06-03 07:02 |
| This code is pretty broken. I don't think ttys are ever seekable, so the os.fdopen has probably been always failing since 3.0. It thus always leaks an fd to '/dev/tty' if the first os.open succeeds. The whole function should probably be rewriten to work with byte streams encoding the prompt with os.device_encoding(tty_fd) falling back on locale.getpreferredencoding(). |
|
|
| msg190603 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-04 15:09 |
| Based on the input from Benjamin Peterson, I grab encoding from the os module in the getpass module. I put some test as well. Until the whole function is rewritten, I hope this patch will suffice and do the job properly. |
|
|
| msg190706 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2013-06-06 08:50 |
| >>> getpass.getpass('Password: ', sys.stdout) Traceback (most recent call last): File "", line 1, in File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass passwd = _raw_input(prompt, stream, input=input) File "/home/serhiy/py/cpython/Lib/getpass.py", line 146, in _raw_input stream.write(bytes(prompt, tty_encoding)) TypeError: must be str, not bytes >>> getpass.getpass('Password: ', io.StringIO()) Traceback (most recent call last): File "", line 1, in File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass passwd = _raw_input(prompt, stream, input=input) File "/home/serhiy/py/cpython/Lib/getpass.py", line 146, in _raw_input stream.write(bytes(prompt, tty_encoding)) TypeError: string argument expected, got 'bytes' |
|
|
| msg190710 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2013-06-06 10:25 |
| The problem is in io.open, not in getpass. Here is a test script. $ ./python buffered_open_fd_leak.py buffered_open_fd_leak.py:7: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'> tty = io.open(fd, 'w+', 1) |
|
|
| msg190711 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-06 10:40 |
| So the correct fix should be: 1. Make sure we can open /dev/tty in non-binary mode, 2. We can write string to /dev/tty, 3. Close the leak if we can not open /dev/tty. Is it? |
|
|
| msg190714 - (view) |
Author: Vajrasky Kok (vajrasky) * |
Date: 2013-06-06 13:48 |
| Fixing IO leak resource would fix this bug but leave another bug opened which I try to fix as well. These statements with Python3 under Linux will always fail because we need to open /dev/tty file in binary mode (for whatever reason). fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY) tty = os.fdopen(fd, 'w+', 1) So I guess the correct fix would be to open /dev/tty in binary mode (and set buffering off) and go on from there. Of course, one can argue whether this bug should be separated from the original bug (resource warning). I am not sure either. Anyway, here is the patch that will work with stream StringIO and stdout. Thank you for Serhiy for pointing out my mistakes. |
|
|
| msg190716 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2013-06-06 14:32 |
| >>> getpass.getpass('Password: ', open('/dev/stdout', 'w')) Traceback (most recent call last): File "", line 1, in File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass passwd = _raw_input(prompt, stream, input=input) File "/home/serhiy/py/cpython/Lib/getpass.py", line 143, in _raw_input stream.write(bytes(prompt, tty_encoding)) TypeError: must be str, not bytes It seems that you are moving in the wrong direction. No need to test explicitly for stdout/stderr/etc, the code should work with arbitrary text stream. |
|
|
| msg190740 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2013-06-07 09:26 |
| Here is a patch which fixes getpass bug: unix_getpass() always fallback to sys.stdin. As side effect it also fixes resource warning in getpass(). I'm not sure I have correctly changed tests. David, could you please review the patch? |
|
|
| msg190746 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2013-06-07 13:29 |
| The test changes look correct to me, but it sure would be nice to come up with less fragile tests. For a function like this, though, it probably isn't possible. |
|
|
| msg192836 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2013-07-10 21:10 |
| New changeset 70f55dc9d43f by R David Murray in branch 'default': #18116: getpass no longer always falls back to stdin. http://hg.python.org/cpython/rev/70f55dc9d43f |
|
|
| msg192837 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2013-07-10 21:13 |
| I played around with this for a bit, but I couldn't come up with any test improvements, or any way to test the bug that is being fixed. So I just committed it as is. Thanks, Serhiy. And thanks Vajrasky for giving it a try and figuring out some of the stuff that *doesn't* work as a fix :) I decided to only commit this to 3.4. I don't think the risk of an unexpected behavior change in a maintenance release is worth the relatively small benefit that this fix provides. |
|
|
| msg206999 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2013-12-27 16:48 |
| New changeset 100f632d4306 by R David Murray in branch '3.3': #18116: backport fix to 3.3 since real-world failure mode demonstrated. http://hg.python.org/cpython/rev/100f632d4306 New changeset 29a5a5b39dd6 by R David Murray in branch 'default': Mostly-null merge of #18116 backport (updated NEWS entry). http://hg.python.org/cpython/rev/29a5a5b39dd6 |
|
|
| msg216839 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2014-04-19 03:25 |
| I opened Issue 21310 about a ResourceWarning from open() which I suspect is the same as what was originally described here. |
|
|