BUG: dt.tz_convert() on categorical[datetime] resets index (original) (raw)

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• (optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample

import itertools import pandas as pd

ts_list = pd.date_range(start='2021-01-01T02:00:00', periods=3, freq='1D').to_list() df = pd.DataFrame(data=list(itertools.islice(itertools.cycle(ts_list), 9)), index=[2, 4, 5, 6, 7, 8, 11, 14, 15], columns=['pure_datetime']) df['categorical_datetime'] = df['pure_datetime'].astype(pd.CategoricalDtype())

"""

df pure_datetime categorical_datetime 2 2021-01-01 02:00:00 2021-01-01 02:00:00 4 2021-01-02 02:00:00 2021-01-02 02:00:00 5 2021-01-03 02:00:00 2021-01-03 02:00:00 6 2021-01-01 02:00:00 2021-01-01 02:00:00 7 2021-01-02 02:00:00 2021-01-02 02:00:00 8 2021-01-03 02:00:00 2021-01-03 02:00:00 11 2021-01-01 02:00:00 2021-01-01 02:00:00 14 2021-01-02 02:00:00 2021-01-02 02:00:00 15 2021-01-03 02:00:00 2021-01-03 02:00:00

df.dtypes pure_datetime datetime64[ns] categorical_datetime category dtype: object """

Now apply some function from .dt namespace.

"""

df['pure_datetime'].dt.tz_localize('Europe/Berlin') 2 2021-01-01 02:00:00+01:00 4 2021-01-02 02:00:00+01:00 5 2021-01-03 02:00:00+01:00 6 2021-01-01 02:00:00+01:00 7 2021-01-02 02:00:00+01:00 8 2021-01-03 02:00:00+01:00 11 2021-01-01 02:00:00+01:00 14 2021-01-02 02:00:00+01:00 15 2021-01-03 02:00:00+01:00 Name: pure_datetime, dtype: datetime64[ns, Europe/Berlin] """

All is fine with pure datetime column

BUT, when applied to categorical column, the index is reset:

"""

df['categorical_datetime'].dt.tz_localize('Europe/Berlin') 0 2021-01-01 02:00:00+01:00 1 2021-01-02 02:00:00+01:00 2 2021-01-03 02:00:00+01:00 3 2021-01-01 02:00:00+01:00 4 2021-01-02 02:00:00+01:00 5 2021-01-03 02:00:00+01:00 6 2021-01-01 02:00:00+01:00 7 2021-01-02 02:00:00+01:00 8 2021-01-03 02:00:00+01:00 Name: categorical_datetime, dtype: datetime64[ns, Europe/Berlin] """

Naive construction of new columns yields convoluted results

df['localized_pure'] = df['pure_datetime'].dt.tz_localize('Europe/Berlin') df['localized_categorical'] = df['categorical_datetime'].dt.tz_localize('Europe/Berlin') """

df pure_datetime categorical_datetime localized_pure localized_categorical 2 2021-01-01 02:00:00 2021-01-01 02:00:00 2021-01-01 02:00:00+01:00 2021-01-03 02:00:00+01:00 4 2021-01-02 02:00:00 2021-01-02 02:00:00 2021-01-02 02:00:00+01:00 2021-01-02 02:00:00+01:00 5 2021-01-03 02:00:00 2021-01-03 02:00:00 2021-01-03 02:00:00+01:00 2021-01-03 02:00:00+01:00 6 2021-01-01 02:00:00 2021-01-01 02:00:00 2021-01-01 02:00:00+01:00 2021-01-01 02:00:00+01:00 7 2021-01-02 02:00:00 2021-01-02 02:00:00 2021-01-02 02:00:00+01:00 2021-01-02 02:00:00+01:00 8 2021-01-03 02:00:00 2021-01-03 02:00:00 2021-01-03 02:00:00+01:00 2021-01-03 02:00:00+01:00 11 2021-01-01 02:00:00 2021-01-01 02:00:00 2021-01-01 02:00:00+01:00 NaT 14 2021-01-02 02:00:00 2021-01-02 02:00:00 2021-01-02 02:00:00+01:00 NaT 15 2021-01-03 02:00:00 2021-01-03 02:00:00 2021-01-03 02:00:00+01:00 NaT """

Problem description

When working on categorical columns, I would expect all methods, that can be called without error or warning, to yield the same results as when applied to the pure (category-expanded) columns. Some methods from the .dt-namespace (at least tz_localize() and tz_convert()) seem to return non-consistent results.
In particular, they reset the index of the input column (a Series in fact).

Expected Output

I expect the method .dt.tz_localize() to not reset the index when applied to a Series of datetime categoricals.

Output of pd.show_versions()

Details

INSTALLED VERSIONS

commit : 5f648bf
python : 3.9.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.14393
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 0, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252

pandas : 1.3.2
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 57.4.0
Cython : None
pytest : 6.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None