json_normalize gives KeyError in 0.23 · Issue #21158 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

import json from pandas import show_versions from pandas.io.json import json_normalize

print(show_versions())

with open('test.json', 'r') as infile: d = json.load(infile)

normed = json_normalize(d)

The test.json file is rather lengthy, with a structure similar to:

{ "subject": { "pairs": { "A1-A2": { "atlases": { "avg.corrected": { "region": null, "x": 49.151580810546875, "y": -33.148521423339844, "z": 27.572303771972656 } } } } } }

This minimal version is enough to show the error below.

Problem description

This problem is new in pandas 0.23. I get the following traceback:

Traceback:

Traceback (most recent call last): File "test.py", line 10, in normed = json_normalize(d) File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 203, in json_normalize data = nested_to_record(data, sep=sep) File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record new_d.update(nested_to_record(v, newkey, sep, level + 1)) File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record new_d.update(nested_to_record(v, newkey, sep, level + 1)) File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record new_d.update(nested_to_record(v, newkey, sep, level + 1)) File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 84, in nested_to_record new_d.pop(k) KeyError: 'region'

Note that running the same code on pandas 0.22 does not result in any errors. I suspect this could be related to #20399.

Expected Output

Expected output is a flattened DataFrame without any errors.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: 3.5.1
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: 0.10.3
IPython: 6.3.1
sphinx: 1.7.2
patsy: None
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None