[Python-Dev] PyUnicode_EncodeDecimal (original) (raw)

Victor Stinner victor.stinner at haypocalc.com
Mon Nov 21 21:39:53 CET 2011

Hi,

I'm trying to rewrite PyUnicode_EncodeDecimal() to upgrade it to the new Unicode API. The problem is that the function is not accessible in Python nor tested. Should we document and test it, leave it unchanged and deprecate it, or simply remove it?

--

Python has a PyUnicode_EncodeDecimal() function. It was used in Python 2 by int, long and complex constructors. In Python 3, the function is no more used: it has been replaced by PyUnicode_TransformDecimalToASCII() in Python <= 3.2 and _PyUnicode_TransformDecimalAndSpaceToASCII() in Python 3.3.

PyUnicode_EncodeDecimal() goes into an unlimited loop if there is more than one unencodable character. It's a known bug and there is a patch: http://bugs.python.org/issue13093

PyUnicode_EncodeDecimal() is undocumented and not tested: http://bugs.python.org/issue8646

Stefan Krah uses PyUnicode_EncodeDecimal() in its cdecimal project.

See also "Malformed error message from float()" issue: http://bugs.python.org/issue10557

Python 3.3 has now 3 encoders to decimal:

_PyUnicode_TransformDecimalAndSpaceToASCII() replaces also Unicode spaces with ASCII spaces. PyUnicode_EncodeDecimal() and PyUnicode_TransformDecimalToASCII() take Py_UNICODE* strings.

PyUnicode_EncodeDecimal() requires an output buffer and it has no argument for the size of the output buffer. It is unsafe: it leads to buffer overflow if the buffer is too small.

Victor

More information about the Python-Dev mailing list