BUG: read_csv returns object dtype for dates in empty frame · Issue #15524 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

import io import pandas as pd

csv = lambda: io.StringIO('date') df = pd.read_csv(csv(), dtype=int) df.dtypes

date int64 dtype: object

df = pd.read_csv(csv(), parse_dates=['date']) df.dtypes

date object dtype: object

df.date.dt.tz_localize('UTC')

Traceback (most recent call last): File "", line 1, in df.date = df.date.dt.tz_localize('UTC') File "/anaconda/lib/python3.5/site-packages/pandas/core/generic.py", line 2740, in getattr return object.getattribute(self, name) File "/anaconda/lib/python3.5/site-packages/pandas/core/base.py", line 241, in get return self.construct_accessor(instance) File "~/anaconda/lib/python3.5/site-packages/pandas/core/series.py", line 2743, in _make_dt_accessor raise AttributeError("Can only use .dt accessor with datetimelike " AttributeError: Can only use .dt accessor with datetimelike values

parser = lambda x: pd.to_datetime(x, utc=True) df = pd.read_csv(csv(), parse_dates=['date'], date_parser=parser) df.dtypes

date object dtype: object

df.date.dt.tz_convert('EST')

Traceback (most recent call last): File "", line 1, in df.date.dt.tz_convert('EST') File "/anaconda/lib/python3.5/site-packages/pandas/core/generic.py", line 2740, in getattr return object.getattribute(self, name) File "/anaconda/lib/python3.5/site-packages/pandas/core/base.py", line 241, in get return self.construct_accessor(instance) File "~/anaconda/lib/python3.5/site-packages/pandas/core/series.py", line 2743, in _make_dt_accessor raise AttributeError("Can only use .dt accessor with datetimelike " AttributeError: Can only use .dt accessor with datetimelike values

Problem description

When reading CSVs with no data rows, read_csv() returns the dtype object for dates, which can raise errors on later manipulation. This is contrary to the general behaviour of read_csv(), which otherwise correctly sets dtypes for empty frames when those dtypes are explicitly passed.

I don't think it would be hard to return the correct dtype here? If date_parser is not set, we know the dtype is datetime64[ns]; otherwise, we can call the parser with empty data, and use the returned dtype.

Note that e.g. read_csv(..., dtype='datetime64[ns]') is not a solution, as this throws an error when the csv is non-empty.

Expected Output

date    int64
dtype: object

date      datetime64[ns]
dtype: object

date      datetime64[ns, UTC]
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Darwin
OS-release: 16.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
statsmodels: 0.6.1
xarray: 0.9.1
IPython: 4.2.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 1.5.3
openpyxl: 2.4.0
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.4
pymysql: None
psycopg2: None
jinja2: 2.8.1
boto: 2.45.0
pandas_datareader: None