BUG: missing values encoded in pandas.Int64Dtype not supported by json table scheme · Issue #40255 · pandas-dev/pandas (original) (raw)

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd data = {'A' : [1, 2, 2, pd.NA, 4, 8, 8, 8, 8, 9], 'B': [pd.NA] * 10} data = pd.DataFrame(data) data = data.astype(pd.Int64Dtype()) # in my example I get this from data.convert_dtypes() data_json = data.to_json(orient='table', indent=4) pd.read_json(data_json, orient='table') #ValueError: Cannot convert non-finite values (NA or inf) to integer

Problem description

I am investigating different json-ouput formats for my use-case. I expect as an end-user that if I can write a json with pandas, I can also read it in without an error.

Background: I used pd.convert_dtypes on a larger DataFrame, where some columns are populated by identical values. convert_dtypes converted some of them having only missing to pd.Int64Dtype. I tried understand why this is failing und could replicate the issue using the above example having identical dtypes as in my use-case.

Expected Output

      A     B
0     1  <NA>
1     2  <NA>
2     2  <NA>
3  <NA>  <NA>
4     4  <NA>
5     8  <NA>
6     8  <NA>
7     8  <NA>
8     8  <NA>
9     9  <NA>

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f2c8480
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Danish_Denmark.1252

pandas : 1.2.3
numpy : 1.18.5
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 50.3.1.post20201107
Cython : None
pytest : None
hypothesis : None
sphinx : 3.5.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : None
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None