pd.timedelta has smaller than expected range · Issue #12727 · pandas-dev/pandas (original) (raw)

The pandas timedelta class has less than expected minimum and maximum timedeltas. The range is [-99999, 99999] days instead of [-999999999, 999999999] days

Code Sample

expected_max_days = 999999999 # 9 nines max_days = 99999 # 5 nines

print("1. Expected minimum and maximum") print(pd.Timedelta.min, '\t', pd.Timedelta.max) print(datetime.timedelta.min, '\t', datetime.timedelta.max)

print("\n2. Expected min and max as pd.Timedelta objects") print(pd.Timedelta(pd.Timedelta.min), '\t', pd.Timedelta(pd.Timedelta.max)) # overflow, no error

print("\n3. True min and max (in days) and comparing with numpy") print(np.timedelta64(-max_days, 'D'), '\t\t', np.timedelta64(max_days, 'D')) print(pd.Timedelta(-max_days, 'D'), '\t', pd.Timedelta(max_days, 'D'))

print("\n4. Expected min and max (in days) and comparing with numpy") print(np.timedelta64(-expected_max_days, 'D'), '\t', np.timedelta64(expected_max_days, 'D')) print(pd.Timedelta(-expected_max_days, 'D'), '\t', pd.Timedelta(expected_max_days, 'D')) # overflowError

Expected Output

1. Expected minimum and maximum
-999999999 days, 0:00:00     999999999 days, 23:59:59.999999
-999999999 days, 0:00:00     999999999 days, 23:59:59.999999

2. Expected min and max as pd.Timedelta objects
52654 days 06:07:35.539769   -52654 days +17:52:24.460229

3. True min and max (in days) and comparing with numpy
-99999 days          99999 days
-99999 days +00:00:00    99999 days 00:00:00

4. Expected min and max (in days) and comparing with numpy
-999999999 days      999999999 days
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-104-9f01918ed468> in <module>()
     15 print("\n4. Expected min and max (in days) and comparing with numpy")
     16 print(np.timedelta64(-expected_max_days, 'D'), '\t', np.timedelta64(expected_max_days, 'D'))
---> 17 print(pd.Timedelta(-expected_max_days, 'D'), '\t', pd.Timedelta(expected_max_days, 'D')) # overflowError

pandas/tslib.pyx in pandas.tslib.Timedelta.__new__ (pandas/tslib.c:43558)()

pandas/tslib.pyx in pandas.tslib.convert_to_timedelta64 (pandas/tslib.c:53419)()

pandas/tslib.pyx in pandas.tslib.cast_from_unit (pandas/tslib.c:58740)()

OverflowError: Python int too large to convert to C long

The problem is illustrated at points 2 (silent overflow) and 4(overflow error)

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.1-gentoo-r1
machine: x86_64
processor: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.18.0
nose: None
pip: 8.1.1
setuptools: 20.3
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: 4.1.2
sphinx: 1.3.6
patsy: 0.4.1
dateutil: 2.5.0
pytz: 2016.1
blosc: None
bottleneck: None
tables: None
numexpr: 2.5
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None

I get similar results on python 3.5.1.