BUG: AttributeError: 'BlockManager' object has no attribute 'is_mixed_type' when trying to jsonify dataframe. · Issue #39837 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example

csv_reader = pandas.read_csv( csv_file, iterator=True, chunksize=2500, sep=',', encoding=None, error_bad_lines=False ) for i, chunk in enumerate(csv_reader): rows = chunk.to_json(orient="records", lines=True)

Problem description

on pypy 7.3.3 - python 3.7.9
to_json execution raises an exception:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/opt/pypy/site-packages/pandas/core/generic.py", line 2478, in to_json
    storage_options=storage_options,
  File "/opt/pypy/site-packages/pandas/io/json/_json.py", line 94, in to_json
    indent=indent,
  File "/opt/pypy/site-packages/pandas/io/json/_json.py", line 155, in write
    indent=self.indent,
AttributeError: 'BlockManager' object has no attribute 'is_mixed_type'

I suspect this PR: https://github.com/pandas-dev/pandas/pull/36873/files
@jbrockmendel do you mind explaining why the BlockManager no longer has the is_mixed_type property?

Expected Output

When I try to manually assign is_mixed_type to the chunk._mgr object, as dropped in the aforementioned PR (a BlockManager) it seems to be working fine:

for i, chunk in enumerate(csv_reader):
     chunk._mgr._consolidate_inplace()
     chunk._mgr.is_mixed_type = len(chunk._mgr.blocks) > 1
     print(chunk.to_json(orient="records", lines=True))

{"@timestamp":10000000,"test_1":"something","test_2":"something2"}

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 7d32926
python : 3.7.9.final.0
python-bits : 64
OS : Linux
OS-release : 4.19.121-linuxkit
Version : #1 SMP Tue Dec 1 17:50:32 UTC 2020
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.2
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 20.3.4
setuptools : 53.0.0
Cython : None
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : 7.20.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.23
tables : None
tabulate : 0.8.7
xarray : None
xlrd : 1.2.0
xlwt : None
numba : None