[Python-Dev] Raw string syntax inconsistency (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Jun 18 01:55:13 CEST 2012

On Mon, Jun 18, 2012 at 6:41 AM, Guido van Rossum <guido at python.org> wrote:

Would it make sense to detect and reject these in 3.3 if the 2.7 syntax is used?

Possibly - I'm trying not to actually change any of the internals of the string literal processing, though. (If I recall the way we implemented the change correctly, by the time we get to processing the string contents, we've forgotten which specific prefix was used)

However, tis question did remind me of another detail I wanted to check after realising this discrepancy existed: it turns out this semantic inconsistency already arises if you use "from future import unicode_literals" to get supposedly "Python 3 style" string literals in 2.x

Python 2.7.3 (default, May 29 2012, 14:54:22)

from future import unicodeliterals print(r"\u03b3") γ print("\u03b3") γ

Python 3.2.1 (default, Jul 11 2011, 18:54:42)

print(r"\u03b3") \u03b3 print("\u03b3") γ

So, perhaps the answer is to leave this as is, and try to make 2to3 smart enough to detect such escapes and replace them with their properly encoded (according to the source code encoding) Unicode equivalent? After all, that's already the way to include such characters in a forward compatible way when using the future import:

Python 2.7.3 (default, May 29 2012, 14:54:22)

from future import unicodeliterals print("γ") γ print(r"γ\n") γ\n

Python 3.2.1 (default, Jul 11 2011, 18:54:42)

print("γ") γ print(r"γ\n") γ\n

So, rather than going ahead with reverting "ur" support as I first suggested (since it turns out that's not a new problem, but just a different way of spelling an existing problem), how about I do the following:

  1. Add a note to PEP 414 and the Py3k porting guide regarding the discrepancy in escaping semantics for raw Unicode strings between 2.x and 3.x
  2. Reject the tracker issue for reverting the ur support (the semantic problem already exists, and any solution we come up with for future.unicode_literals should handle the ur prefix as well)
  3. Create a new feature request for 2to3 to see if it can automatically handle the problem of translating "\u" and "\U" escapes into properly encoded Unicode characters

The scope of the problem is really quite small: you have to be using a raw Unicode string in 2.x (either via the string prefix, or the future import) and using a "\u" or "\U" escape within that string.

Regards, Nick.

--Guido van Rossum (sent from Android phone) On Jun 17, 2012 1:13 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

On Mon, Jun 18, 2012 at 3:54 AM, Terry Reedy <tjreedy at udel.edu> wrote: > The premise of the discussion of adding 'u', and of Guido's acceptance, > was > that "it's about as harmless as they come". I do not remember any > discussion > of 'ur' and what it really means in 2.x, and that supporting it meant > adding > back 2.x's interaction effect. Indeed, Nick's version goes on to say > "This > PEP was originally written by Armin Ronacher, and Guido's approval was > given > based on that version." Armin's original version (and subsequent edit) > only > proposed adding 'u' (and 'U') and made no mention of 'ur'. Nick's > seemingly > innocuous addition of also adding 'ur' came after Guido's approval, and > as > discovered, is not so innocuous. Right, that matches my recollection as well - we (or least I) thought mapping "ur" to the Python 3 "r" prefix was sufficient, but it turns out doing so means there are some 2.x string literals that will silently behave differently in 3.x. Martin's right that that part of the PEP should definitely be amended (along with the relevant section in What's New) > I do not think he needs to discuss adding and deleting support, but > merely > state that 'ur' support is not added because 'ur' has a special meaning > that > would require changing literal handling. The sentence about supporting > 'ur' > could be negated and moved after the sentence about not changing Unicode > handling. A possibility: > > "Combination of the unicode prefix with the raw string prefix will not > be > supported because in Python 2, the combination 'ur' has a special > meaning > that would require changing the handling of unicode literals" In addition to changing the proposal section to only cover "u" and "U", I'll actually add a new subsection along the lines of the following: Exclusion of Raw Unicode Strings ------------------------------------------------- Python 2.x includes a concept of "raw Unicode" strings. These are partially raw string literals that still support the "\u" and "\U" escape codes for Unicode character entry, but otherwise treat "" as a literal backslash character. As 3.x has no such concept of a partially raw string literal, explicit raw Unicode literals are still not supported. Such literals in Python 2 code will need to be converted to ordinary Unicode literals for forward compatibility with Python 3. Cheers, Nick. -- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list