Timezone info lost when broadcasting scalar datetime to DataFrame · Issue #11682 · pandas-dev/pandas (original) (raw)

I've encountered a bug in pandas 0.16.2, where when using broadcasting to assign a datetime.datetime value to a whole column of a DataFrame, the timezone info is lost. Here is an example:

In [1]: import pandas, datetime, pytz

In [2]: df = pandas.DataFrame({'a': [1,2,3]})

In [3]: dt = datetime.datetime.now(pytz.utc)

In [4]: dt.tzinfo Out[4]:

In [5]: df['b'] = dt

In [6]: df Out[6]: a b 0 1 2015-11-23 21:02:54.562175 1 2 2015-11-23 21:02:54.562175 2 3 2015-11-23 21:02:54.562175

In [7]: df['b'][0].tzinfo

Note how dt has a timezone attached, but the values in the 'b' column don't. The problem only occurs when broadcasting a scalar datetime column, not when assigning an array or series. Also, the problem only occurs when using the builtin datetime.datetime class, not pandas's Timestamp class.

I've tracked the problem down to the pandas.core.common._infer_dtype_from_scalar function, which is called during the assignment. It contains this code for handling scalar date times:

elif isinstance(val, (np.datetime64, datetime)) and getattr(val,'tz',None) is None:
    val = lib.Timestamp(val).value
    dtype = np.dtype('M8[ns]')

The problem is that the Timestamp.value property returns an integer value which doesn't contain the timezone information, so the timezone is lost. The reason this problem occurs for datetime.datetime, but not for pandas.Timestamp, is because the code is looking for the 'tz' attribute, which is specific to Timestamp. If the gettattr call was changed to look at the 'tzinfo' attribute instead, this code would work correctly for both pandas.Timestamp and datetime.datetime values. So a fix for this code which works for both datetime and Timestamp would be:

elif isinstance(val, (np.datetime64, datetime)) and getattr(val,'tzinfo',None) is None:
    val = lib.Timestamp(val).value
    dtype = np.dtype('M8[ns]')

I checked and this bug still exists in the latest version of the pandas source. Nevertheless here is the output of show_versions() on my machine:

In [8]: pandas.show_versions()

INSTALLED VERSIONS

commit: None python: 2.7.9.final.0 python-bits: 64 OS: Darwin OS-release: 14.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8

pandas: 0.16.2 nose: 1.3.7 Cython: 0.23.2 numpy: 1.9.2 scipy: 0.16.0 statsmodels: 0.6.1 IPython: 3.1.0 sphinx: 1.3.1 patsy: None dateutil: 2.4.2 pytz: 2012c bottleneck: None tables: 3.2.1.1 numexpr: 2.4.4 matplotlib: 1.4.2 openpyxl: None xlrd: 0.9.4 xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None httplib2: None apiclient: None sqlalchemy: 1.0.8 pymysql: None psycopg2: 2.6.1 (dt dec pq3 ext lo64)