Issue 3511: Incorrect charset range handling with ignore case flag? (original) (raw)
Created on 2008-08-06 19:41 by mrabarnett, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (7)
Author: Matthew Barnett (mrabarnett) * 
Date: 2008-08-06 19:41
While working on the regex code in sre_compile.py I came across the following code in the handling of charset ranges in _optimize_charset:
for i in range(fixup(av[0]), fixup(av[1])+1):
charmap[i] = 1The function fixup converts the ends of the range to lower case if the ignore-case flag is present. The problem with this approach is illustrated below:
import re print re.match(r'[9-A]', 'A') <sre.SRE_Match object at 0x00A78058> print re.match(r'[9-A]', 'a') None print re.match(r'[9-A]', '') None print re.match(r'[9-A]', 'A', re.IGNORECASE) <sre.SRE_Match object at 0x00D0BFA8> print re.match(r'[9-A]', 'a', re.IGNORECASE) <_sre.SRE_Match object at 0x00A78058> print re.match(r'[9-A]', '', re.IGNORECASE) <_sre.SRE_Match object at 0x00D0BFA8>
'_' doesn't lie between '9' and 'A', but it does lie between '9' and 'a'.
Surely the ignore-case flag should not affect whether non-letters are matched or not?
Author: Jeffrey C. Jacobs (timehorse)
Date: 2008-09-24 13:15
I think this is even more complicated when you consider that localization my be an issue. Consider "Á": is this grammatically before "A" or after "a"? From a character set point of view, it is typically after "a" but when Locale is taken into account, all that is done is there is a change to relative ordering, so Á appears somewhere before A and B. But when this is done, does that mean that [9-Á] is going to cover ALL uppercase and ALL lowercase and ALL characters with ord from 91 to 96 and 123 to 127 and all kinds of other UNICODE symbols? And how will this effect case-insensitivity.
In a sense, I think it may only be safe to say that character class ranges are ONLY appropriate over Alphabetic character ranges or numeric character ranges, since the order of the ASCII symbols between 0 and 47, 56 and 64, 91 adn 96 and 123 and 127, though well-defined, are none the less implementation dependent. When we bring UNICODE into this, things get even more befuddled with some Latin characters in Latin-1, some in Latin-2, Cyrillic, Hebrew, Arabic, Chinese, Japanese and Korean character sets just to name a few of the most common! And how does a total ordering of characters apply to them?
In the end, I think it's just dangerous to define character group ranges that span the gap BETWEEN numbers and alphabetics. Instead, I think a better solution is simply to implement Emacs / Perl style named character classes as in issue 2636 sub-item 8.
I do agree this is a problem, but as I see it, the solution may not be that simple, especially in a UNICODE world.
Author: Ezio Melotti (ezio.melotti) * 
Date: 2009-02-20 06:46
I'd close this as "won't fix", because (IMHO) ranges like [9-A] shouldn't be used at all, so I won't expect it to work properly.
FWIW Perl doesn't seem to match the '', even with the 'i' flag. Tested with: perl -e '$s = ("" =~ /[9-A]/); print s′andperl−e′s' and perl -e 's′andperl−e′s = ("" =~ /[9-A]/i); print $s'. It matches ":" with [9-A] and "" with [9-a] though (both without the 'i' flag).
Author: Matthew Barnett (mrabarnett) *
Date: 2009-02-20 16:31
"[9-A]" is equivalent to "[9:;<=>?@A]", or should be.
It'll be fixed in issue #2636.
Author: Ezio Melotti (ezio.melotti) *
Date: 2009-02-20 19:05
If there's already a patch, then it's fine (and useful for ranges of Unicode chars), but I wouldn't like to find a range like [9-A] around ;)
Author: Terry J. Reedy (terry.reedy) *
Date: 2010-08-03 18:56
EM and MB seemed to agree on closing this.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2014-11-08 12:14
Fixed in (which has more realistic example than [9-A]).
History
Date
User
Action
Args
2022-04-11 14:56:37
admin
set
github: 47761
2014-11-08 12:14:26
serhiy.storchaka
set
nosy: + serhiy.storchaka
messages: +
resolution: wont fix -> duplicate
superseder: IGNORECASE breaks unicode literal range matching
2010-08-03 18:56:03
terry.reedy
set
status: open -> closed
nosy: + terry.reedy
messages: +
resolution: wont fix
2009-02-20 19:05:29
ezio.melotti
set
messages: +
2009-02-20 16:31:11
mrabarnett
set
messages: +
2009-02-20 06:46:22
ezio.melotti
set
nosy: + ezio.melotti
messages: +
2008-09-24 13:15:03
timehorse
set
messages: +
2008-09-24 12:58:36
timehorse
set
nosy: + timehorse
2008-08-06 19:48:38
pitrou
set
priority: normal
nosy: + pitrou
versions: + Python 2.6, Python 3.0
2008-08-06 19:41:11
mrabarnett
create