Read_json overflow error when json contains big number strings · Issue #30320 · pandas-dev/pandas (original) (raw)

Skip to content

Navigation Menu

• • GitHub Copilot Write better code with AI
  • GitHub Advanced Security Find and fix vulnerabilities
  • Actions Automate any workflow
  • Codespaces Instant dev environments
  • Issues Plan and track work
  • Code Review Manage code changes
  • Discussions Collaborate outside of code
  • Code Search Find more, search less
• Explore
  • Learning Pathways
  • Events & Webinars
  • Ebooks & Whitepapers
  • Customer Stories
  • Partners
  • Executive Insights
• • GitHub Sponsors Fund open source developers
  • The ReadME Project GitHub community articles
• • Enterprise platform AI-powered developer platform
• Pricing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Description

Code Sample, a copy-pastable example if possible

import json import pandas as pd

test_data = [{"col": "31900441201190696999"}, {"col": "Text"}] test_json = json.dumps(test_data) pd.read_json(test_json)

Problem description

The current behaviour doesn't return a dateframe for a valid JSON. Note when the number is smaller, it works fine. It also works when only big numbers are present. It would be cool to have it work with big numbers as it works for small numbers.

Expected Output

A dataframe with a number and string

       col
0  3.190044e+19
1     Text

Output of pd.read_json()

Traceback (most recent call last): File "", line 1, in File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 592, in read_json result = json_reader.read() File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 717, in read obj = self._get_object_parser(self.data) File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 739, in _get_object_parser obj = FrameParser(json, **kwargs).parse() File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 855, in parse self._try_convert_types() File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1151, in _try_convert_types lambda col, c: self._try_convert_data(col, c, convert_dates=False) File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1131, in _process_converter new_data, result = f(col, c) File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1151, in lambda col, c: self._try_convert_data(col, c, convert_dates=False) File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 927, in _try_convert_data new_data = data.astype("int64") File ".../.venv/lib/python3.6/site-packages/pandas/core/generic.py", line 5882, in astype dtype=dtype, copy=copy, errors=errors, **kwargs File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 581, in astype return self.apply("astype", dtype=dtype, **kwargs) File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 438, in apply applied = getattr(b, f)(**kwargs) File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 559, in astype return self._astype(dtype, copy=copy, errors=errors, values=values, **kwargs) File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 643, in _astype values = astype_nansafe(vals1d, dtype, copy=True, **kwargs) File ".../.venv/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 707, in astype_nansafe return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape) File "pandas/_libs/lib.pyx", line 547, in pandas._libs.lib.astype_intsafe OverflowError: Python int too large to convert to C long