DF.stack() on single column datetime with timezone causes loss of timezone · Issue #19420 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

import pandas as pd

ts1 = list(pd.date_range(freq="D", start="20180101", end="20180103")) df1 = pd.DataFrame({"A": ts1}, index=["a", "b", "c"]) print("Works as expected; no tz") print(df1.stack()) #a A 2018-01-01 #b A 2018-01-02 #c A 2018-01-03 #dtype: datetime64[ns]

ts2 = list(pd.date_range(freq="D", start="20180101", end="20180103", tz="America/New_York")) ts3 = list(pd.date_range(freq="D", start="20180104", end="20180106", tz="America/New_York"))

df2 = pd.DataFrame({"A": ts2, "B": ts3}, index=["a", "b", "c"]) print("Works as expected; has tz") print(df2.stack()) #a A 2018-01-01 00:00:00-05:00

B 2018-01-04 00:00:00-05:00

#b A 2018-01-02 00:00:00-05:00

B 2018-01-05 00:00:00-05:00

#c A 2018-01-03 00:00:00-05:00

B 2018-01-06 00:00:00-05:00

#dtype: datetime64[ns, America/New_York]

df3 = pd.DataFrame({"A": ts2}, index=["a", "b", "c"]) print("Unexpected loss of tz, but converted to UTC") print(df3.stack()) #a A 2018-01-01 05:00:00 #b A 2018-01-02 05:00:00 #c A 2018-01-03 05:00:00 #dtype: datetime64[ns]

Problem description

Stacking a DF with multiple datetime with timezone columns seems fine but stacking a single datetime with timezone column appears to convert the time to a naive datetime.

This can be worked around using the .dt.tz and .dt.tz_localize/.dt.tz_convert functionality

Expected Output

a A 2018-01-01 00:00:00-05:00
b A 2018-01-02 00:00:00-05:00
c A 2018-01-03 00:00:00-05:00
dtype: datetime64[ns, America/New_York]

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-514.21.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.utf-8
LANG: en_US.utf-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None