Issue 20115: NUL bytes in commented lines (original) (raw)

Issue20115

process

Created on 2014-01-03 17:59 by arigo, last changed 2022-04-11 14:57 by admin.

Messages (19)
msg207232 - (view)	Author: Armin Rigo (arigo) *	Date: 2014-01-03 17:59
This is probably the smallest example of a .py file that behaves differently in CPython vs PyPy, and for once, I'd argue that the CPython behavior is unexpected: # make the file: >>> open('x.py', 'wb').write('#\x00\na') # run it: python x.py Expected: either some SyntaxError, or "NameError: global name 'a' is not defined". Got: nothing. It seems that CPython completely ignores the line that is immediately after a line with a '#' and a following '\x00'.
msg207282 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2014-01-04 13:18
Indeed. CPython parser reads first line '#\x00\n' and save it in the buffer. But because C strings are used here (result of decode_str()), the line is truncated to '#'. As far as this data is not ended by '\n', it considered incomplete and next line is read and appended: '#' + 'a' -> '#a'. And this line is commented out now.
msg207290 - (view)	Author: Benjamin Peterson (benjamin.peterson) *	Date: 2014-01-04 15:35
I guess NULL bytes should just be banned.
msg207358 - (view)	Author: Armin Rigo (arigo) *	Date: 2014-01-05 07:22
Fwiw, both exec and eval() ban NUL bytes, which means that there is a strange case in which some files can be imported, but not loaded and exec'ed. So I agree with Benjamin.
msg207872 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2014-01-10 18:22
Python should have a uniform definition of 'Python source' in both the doc and in practice in all source code processing functions. Currently, "2. Lexical analysis" in the Language Manual just says "Python reads program text as Unicode code points; the encoding of a source file can be given by an encoding declaration and defaults to UTF-8." UTF-8 encodes code point U+0000 as a null byte and this code point is nowhere excluded in the doc. (The definition of string literals uses 'source character' without any additional specification, so I take it to mean 'Unicode code point'.) If U+0000 is a legal 'source character', it, as with other control chars not given special meaning, should be a SyntaxError unless occurring in a comment or string literal. Eval and exec exclude even the latter with TypeError: source code string cannot contain null bytes If null bytes are legal, this is wrong. Simply truncating lines as done by the CPython parser is wrong whether not not U+0000 is legal. The simplest change would be to change the parser to match exec and add " other than U+000" after "Unicode code points" in the sentence quoted above.
msg207873 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2014-01-10 18:23
Armin, what is the different behavior of PyPy? We should perhaps get Guido's opinion on this issue.
msg207879 - (view)	Author: Armin Rigo (arigo) *	Date: 2014-01-10 20:14
PyPy 2.x accepts null characters in all of import, exec and eval, and complains if they occur in non-comment. PyPy 3.x refuses them in import, which is where this bug report originally comes from (someone complained that CPython 3.x "accepts" them but not PyPy 3.x, even thought this complain doesn't really make sense as CPython just gets very confused by them). I don't know about exec and eval. We need a consistent decision for 3.5. I suppose it's not really worth backporting it to CPython 2.7 - 3.3 - 3.4, but it's your choice. PyPy will just follow the lead (or keep its current behavior for 2.x if CPython 2.x is not modified).
msg207939 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2014-01-12 09:28
I'm in favor of PyPy's behavior: null bytes anywhere in the source, even in comments, usually mean there's something weird or fishy going on with either the editor or (if downloaded/copied) the source of the code.
msg208086 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2014-01-14 09:12
I'll try, but I'm not sure this is possible. Some used C functions (e.g. fgets()) returns char* and doesn't work with string containing null bytes. Some public API (e.g. PyParser_SimpleParseString()) work with null-terminated C strings.
msg208087 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2014-01-14 09:14
See also .
msg218239 - (view)	Author: (ita1024)	Date: 2014-05-10 22:42
Do not touch that please!!!! The null bytes are already rejected when forbidden by the encoding (utf-8 for example). Null byte characters in comments are perfectly valid in ISO8859-1 encoding, and a few scripts depend on them: http://ftp.waf.io/pub/release/waf-1.7.16 Parsing the commented lines is also likely to slow down the parser, so keep your hands of it please! There are too many regressions already! http://bugs.python.org/issue21086
msg395928 - (view)	Author: Irit Katriel (iritkatriel) *	Date: 2021-06-16 14:38
This is still the same in 3.11. I added another line to the example's file, which shows more clearly what's happening: >>> open('x.py', 'wb').write(b'#\x00\na\nb\n') % ./python.exe x.py Traceback (most recent call last): File "x.py", line 2, in a NameError: name 'b' is not defined
msg395932 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2021-06-16 15:16
https://docs.python.org/3/reference/toplevel_components.html#file-input says that file input and exec input (should) have the same grammar. This implies that the divergence is a bug.
msg395938 - (view)	Author: Guido van Rossum (gvanrossum) *	Date: 2021-06-16 15:38
Yeah, null bytes should just be rejected. If someone comes up with a fix for this we'll accept it.
msg395989 - (view)	Author: Irit Katriel (iritkatriel) *	Date: 2021-06-17 10:33
See also .
msg401193 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2021-09-06 23:29
The compile() doc currently says ""This function raises SyntaxError if the compiled source is invalid, and ValueError if the source contains null bytes." And indeed, in repository 3.9, 3.10, 3.11, >>> compile('\0','','exec') Traceback (most recent call last): File "", line 1, in ValueError: source code string cannot contain null bytes Ditto when run same in a file from IDLE or command line. The exception sometimes when the null is in a comment or string within the code. >>> '\0' '\x00' >>> #\0 >>> compile('#\0','','single', 0x200) Traceback (most recent call last): File "", line 1, in ValueError: source code string cannot contain null bytes >>> compile('"\0"','','single', 0x200) ValueError: source code string cannot contain null bytes I am puzzled because "\0" and #\0 in the IDLE shell are sent as strings containing the string or comment to compiled with the call above in codeop. There must be some difference in when \0 is interpreted.
msg401196 - (view)	Author: Guido van Rossum (gvanrossum) *	Date: 2021-09-06 23:34
Which part puzzles you? I see that you tried >>> #\0 This does not contain a null byte, just three characters: a hash, a backslash, and a digit zero.
msg401204 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2021-09-07 03:02
What I missed before is that duplicating the effect of the first two interactive entries (no exception) requires escaping the backslash so that the source argument for the explicit compile does not have a null. compile("'\\0'", '', 'exec') <code object at 0x00000214431CAA20, file "", line 1> compile("#\\0", '', 'exec') <code object at 0x00000214431CAC30, file "", line 1> So I did not actually see an exception to the rule. --- *On Win 10*, I experimented with a version of Armin and Irit's example, without and with b'...' and 'wb'. s = '#\x00\na\nb\n' print(len(s)) # 7 with open("f:/Python/a/nulltest.py", 'w') as f: f.write(s) import nulltest When I ran a local repository build of 3.9, 3.10, or 3.11 with f:\dev\3x>python f:/Python/a/nulltest.py I got Irit's strange NameError instead of the proper ValueError. When I ran with installed 3.9 or 3.10 with py -3.10 -m a.nulltest I got the null-byte ValueError. When I ran from IDLE's editor running on either installed or repository python, the import gave the null-byte ValueError.
msg401208 - (view)	Author: Guido van Rossum (gvanrossum) *	Date: 2021-09-07 05:32
Serhiy’s comment from 2014-01-04 gives the answer. It’s different reading from a file than from a string. And only “python x.py” still reads from a file.