str vs bytes confusion in ujson on Python 3 · Issue #18878 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

import numpy as np import pandas._libs.json as ujson

ujson.encode(np.array(1))

Problem description

On CPython 3.5, this raises TypeError, as expected, but the output contains garbage bytes:

Traceback (most recent call last):
  File "bug.py", line 4, in <module>
    ujson.encode(np.array(1))
TypeError: � (0d array) is not JSON serializable at the moment

On pypy3, it segfaults.

The problem comes from this code:

tmpObj = PyObject_Repr(obj);
PyErr_Format(PyExc_TypeError,
"%s (0d array) is not JSON serializable at the moment",
PyString_AS_STRING(tmpObj));
Py_DECREF(tmpObj);

It calls PyString_AS_STRING (=PyBytes_AS_STRING on py3k) on the PyUnicodeObject returned by PyObject_Repr.

I haven't looked very carefully, but there seem to be a few more similar misuses of PyString_AS_STRING in the same file.

Expected Output

Traceback (most recent call last):
  File "bug.py", line 4, in <module>
    ujson.encode(np.array(1))
TypeError: array(1) (0d array) is not JSON serializable at the moment

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-135-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: fr_FR.UTF-8