BUG: read_csv(dtype='category') raises with many categories (original) (raw)

Code Sample, a copy-pastable example if possible

import io import pandas as pd csv = io.StringIO('\n'.join(map(str, range(10**6)))) df = pd.read_csv(csv, dtype='category')

results in

File "bug.py", line 5, in df = pd.read_csv(csv, dtype='category') File "lib/python3.6/site-packages/pandas/io/parsers.py", line 705, in parser_f return _read(filepath_or_buffer, kwds) File "lib/python3.6/site-packages/pandas/io/parsers.py", line 451, in _read data = parser.read(nrows) File "lib/python3.6/site-packages/pandas/io/parsers.py", line 1065, in read ret = self._engine.read(nrows) File "lib/python3.6/site-packages/pandas/io/parsers.py", line 1828, in read data = self._reader.read(nrows) File "pandas/_libs/parsers.pyx", line 894, in pandas._libs.parsers.TextReader.read File "pandas/_libs/parsers.pyx", line 944, in pandas._libs.parsers.TextReader._read_low_memory File "pandas/_libs/parsers.pyx", line 2218, in pandas._libs.parsers._concatenate_chunks File "lib/python3.6/site-packages/numpy/core/numerictypes.py", line 1016, in find_common_type array_types = [dtype(x) for x in array_types] File "lib/python3.6/site-packages/numpy/core/numerictypes.py", line 1016, in array_types = [dtype(x) for x in array_types] TypeError: data type not understood

Problem description

read_csv now raises when reading a column with many unique values as a category. This appears to be a regression in 0.21.0, due to the introduction of CategoricalDtype.

Expected Output

No exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.11.11-300.fc26.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C
LANG: C
LOCALE: None.None

pandas: 0.21.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 4.2.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0b10
sqlalchemy: 1.1.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: 0.1.3
pandas_gbq: None
pandas_datareader: None