Series iteration and to_dict methods sometimes return underlying storage type vs. Python object · Issue #25969 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

import numpy as np import pandas as pd

s1 = pd.Series({"a": np.int64(64), "b": 10}) for v in s1.to_dict().values(): print(type(v)) # prints <class 'int'> 2x

s2 = pd.Series({"a": np.int64(64), "b": 10, "c": "ABC"}) for v in s2.to_dict().values(): print(type(v)) # prints <class 'numpy.int64'> for first variable "a"

for k, v in s1.items(): print(k, type(v)) # prints <class 'int'> 2x

for k, v in s2.items(): print(k, type(v)) # prints <class 'numpy.int64'> again for the first variable "a"

Problem description

pd.Series.to_dict can return different types for objects depending on the composition of the series. This also affects iteration, e.g., for k, v in series: .... This is inconsistent and, critically, leads to really weird and hard to debug issues downstream with types, especially around JSON conversion (the built-in json module and many others will blow up when it encounters numpy dtypes).

I cannot find this exact issue open in the issue tracker, though there are a number of related issues including:

An issue related to DataFrame.to_dict and inconsistent types (closed in 0.24): to_dict doesn't convert np.int64 to python integers (second try) #24908
This issue also related to scalar coercion on DataFrame.to_dict calls (also closed recently): DataFrame.to_dict returning numpy scalars in certain cases #23753
This PR fixes the issue deriving from iteration, but it looks like the above case is either an untested edge case or a regression: COMPAT: Iteration should always yield a python scalar #17491

Expected Output

Expected output is for type coercion to Python ints to occur regardless of the exact column composition in the Series. #24908 is a related issue for DataFrame coercions with irregular behavior happening as a result.

Output of `pd.show_versions()`

INSTALLED VERSIONS