BUG: .diff() function for UTC datetimes doesn't work · Issue #18578 · pandas-dev/pandas (original) (raw)

example code

import pandas as pd
import numpy as np
import datetime as dt

start_timestamp = int(dt.datetime(year=2017, day=1, month=1).timestamp())
dr = np.arange(start_timestamp, start_timestamp+100)

df = pd.DataFrame({'ts': dr})
df['dt'] = pd.to_datetime(df['ts'], unit='s')
df['dtUTC'] = pd.to_datetime(df['ts'], unit='s', utc=True)

# you can see that the dtypes are different between 0.20.3 and 0.21
print(df.dtypes)

# this all will work
__ = df['dt'].diff()
__ = df[['dt']].diff()
__ = df['dtUTC'].diff()

# this will raise an IndexError in pandas==0.21 but works in pandas==0.20.3
__ = df[['dtUTC']].diff()

Problem description

This worked in pandas 0.20.3. For some reason in pandas 0.21 and greater the to_datetime() is now providing a 'datetime64[ns, UTC]' which doesn't play well with diff() when called from the DataFrame. When called from the Series it works fine.

Expected Output

I don't really have an opinion on which dtype is used. In fact the dtypes seem more "correct" in 0.21 than in 0.20.3. I just want diff() to work when using a datetime64[ns, UTC] the same way it does for a datetime64[ns] when called from a DataFrame. It works when called from a Series so this seems like a bug.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.0-40-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 20.7.0
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
None