Where method does not properly handle values with datetimes with TZ · Issue #21546 · pandas-dev/pandas (original) (raw)

Code Sample, a copy-pastable example if possible

Series input:

import pandas as pd
dts1 = pd.date_range('20150101', '20150105', tz='America/New_York')
df1 = pd.DataFrame({'date': dts1})
dts2 = pd.date_range('20150103', '20150107', tz='America/New_York')
df2 = pd.DataFrame({'date': dts2})
ser_result = df1.date.where(df1.date < df1.date[3], df2.date)
ser_result

Series output:

0 2015-01-01 00:00:00-05:00 1 2015-01-02 00:00:00-05:00 2 2015-01-03 00:00:00-05:00 3 1420520400000000000 4 1420606800000000000 Name: date, dtype: object

DataFrame input:

import pandas as pd
dts1 = pd.date_range('20150101', '20150105', tz='America/New_York')
df1 = pd.DataFrame({'date': dts1, 'x':np.arange(5), 'y':dts1.tz_localize(None)})
dts2 = pd.date_range('20150103', '20150107', tz='America/New_York')
df2 = pd.DataFrame({'date': dts2, 'x':np.arange(3, 8), 'y':dts2.tz_localize(None)})
mask = pd.DataFrame(True, index=df1.index, columns=df2.columns)
mask.iloc[3:] = False
df_result = df1.where(mask, df2)
df_result

DataFrame output:

                   date  x          y

0 2015-01-01 00:00:00-05:00 0 2015-01-01 1 2015-01-02 00:00:00-05:00 1 2015-01-02 2 2015-01-03 00:00:00-05:00 2 2015-01-03 3 2015-01-04 00:00:00-05:00 6 2015-01-06 4 2015-01-05 00:00:00-05:00 7 2015-01-07 2015-01-01 00:00:00-05:00 2015-01-02 00:00:00-05:00 2015-01-03 00:00:00-05:00 2015-01-04 00:00:00-05:00 2015-01-05 00:00:00-05:00 2015-01-01 00:00:00-05:00 2015-01-02 00:00:00-05:00 2015-01-03 00:00:00-05:00 2015-01-04 00:00:00-05:00 2015-01-05 00:00:00-05:00 2015-01-03 00:00:00-05:00 2015-01-04 00:00:00-05:00 2015-01-05 00:00:00-05:00 2015-01-06 00:00:00-05:00 2015-01-07 00:00:00-05:00 2015-01-03 00:00:00-05:00 2015-01-04 00:00:00-05:00 2015-01-05 00:00:00-05:00 2015-01-06 00:00:00-05:00 2015-01-07 00:00:00-05:00

Problem description

where fails on both Series and DataFrame when given values that have datetime-tz dtype. Both work fine with naive datetime values. Series ends up with a mix of datetime-tz and what appear to be i8 values; datetime-tz columns in DataFrames end up with the wrong shape (perhaps a concat along the wrong axis is occurring?).

Expected Output

Series:

0 2015-01-01 00:00:00-05:00 1 2015-01-02 00:00:00-05:00 2 2015-01-03 00:00:00-05:00 3 2015-01-06 00:00:00-05:00 4 2015-01-07 00:00:00-05:00 Name: date, dtype: datetime64[ns, America/New_York]

DataFrame:

                   date  x          y

0 2015-01-01 00:00:00-05:00 0 2015-01-01 1 2015-01-02 00:00:00-05:00 1 2015-01-02 2 2015-01-03 00:00:00-05:00 2 2015-01-03 3 2015-01-06 00:00:00-05:00 6 2015-01-06 4 2015-01-07 00:00:00-05:00 7 2015-01-07

Output of pd.show_versions()

INSTALLED VERSIONS

commit: 2f4d393
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0.dev0+113.g263386389
pytest: 3.6.0
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.14.2
scipy: 1.0.0
pyarrow: 0.8.0
xarray: 0.10.6
IPython: 6.4.0
sphinx: 1.7.5
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.5
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.8
pymysql: 0.8.1
psycopg2: None
jinja2: 2.10
s3fs: 0.1.5
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None