Issue 32483: Misleading reflective behaviour due to PEP 3131 NFKC identifiers normalization. (original) (raw)

Consistent with PEP 3131 and NFKC normalization of identifiers, these two last lines yield an error, since 𝜏 (U+1D70F) is automatically converted to τ (U+03C4).

class Base(object):
    def __init__(self):
        self.𝜏 = 5 # defined with U+1D70F

a = Base()
print(a.𝜏)     # 5             # (U+1D70F) expected and intuitive
print(a.τ)     # 5 as well     # (U+03C4)  normalized version, okay.
d = a.__dict__ # {'τ':  5}     # (U+03C4)  still normalized version
print(d['τ'])  # 5             # (U+03C4)  consistent with normalization
assert hasattr(a, 'τ')         # (U+03C4)  consistent with normalization
# But if I want to retrieve it the way I entered it because I can type (U+1D70F)
print(d['𝜏'])  # KeyError: '𝜏' # (U+1D70F) counterintuitive
assert hasattr(a, '𝜏') # Fails # (U+1D70F) counterintuitive

I've described and undestood the problem in this post.

Nothing is unconsistent here. However, I am worried that:

• this behaviour might be counterintuitive and misleading, especially if it occurs that the character user can easily enter for some reason (e.g. U+1D70F) is not equivalent to its NFKC normalization (e.g. U+03C4)

• this behaviours makes it more difficult to enjoy python's reflective __dict__, hasattr and getattr features in this particular case.

Maybe it is user's responsibility to be aware of this limitation, and to keep considering utf-8 coding a bad practice. In this case, maybe this particular reflective limitation could be made explicit in PEP 3131.

Or maybe it is python's responsibility to ensure intuitive and consistent behaviour even in tricky-unicode-cases. So reflective features like __dict__.__getitem__, hasattr or getattr would NFKC-convert their arguments before searching just like a.𝜏 does, so that:

getattr(a, '𝜏') is gettatr(a, 'τ')

always yields True.

I actually have no idea of the philosophy to stick to. And the only purpose of this post is to inform the community about this particular, low-priority case.

Thank you for supporting Python anyway, cheers for your patience.. and happy 2018 to everyone :)

-- Iago-lito