pd.Categorical(Series, categories=..) returns broken data with out-of-bound codes · Issue #25318 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

c1 = pd.Categorical(['A', 'B', 'B', 'C'])

Works as expected:

#c2 = pd.Categorical(c1, categories=c1.categories[1:])

Generates broken data:

c2 = pd.Categorical(pd.Series(c1), categories=c1.categories[1:]) print(c1[:2]) print(c2[:2]) print(c2.codes.max(), len(c2.categories))

Prints:

[A, B] Categories (3, object): [A, B, C] [B, C] Categories (2, object): [B, C] 2 2

Problem description

pd.Categorical should not be able to construct an object where codes is >= len(categories). If it does this, then it's very likely that passing the array into other Pandas functions will trigger a segfault. For example, if you try to use this Categorical as a column in pd.concat then Python just dies:

>>> pd.concat([pd.DataFrame.from_dict({'A': c2}), pd.DataFrame.from_dict({'A': c2})], axis=0)

C:\>echo %ERRORLEVEL%
-1073741819

Expected Output

[A, B] Categories (3, object): [A, B, C] [NaN, B] Categories (2, object): [B, C] 1 2

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

pandas: 0.24.1
pytest: 3.1.2
pip: 19.0.2
setuptools: 39.0.1
Cython: 0.27.2
numpy: 1.16.1
scipy: 1.2.1
pyarrow: 0.9.0
xarray: None
IPython: 6.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: 1.3.0.dev0
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 3.8.0
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.1.11
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None
gcsfs: None