| msg229276 - (view) |
Author: Christopher Lee (veebers) |
Date: 2014-10-14 01:31 |
| I have an example script here[1]. This script creates 2 datetime objects (using a timedelta work around to deal with large timestamps). It then makes 2 assertions, that the timestamp of the created object is the same as the one that was used to create it. (when run with no arguments this script passes both assertions). However, if the argument 'breakme' is passed to the script then after the first assertion the method 'timestamp()' is called on a different (un-asserted) datetime which will now make the 2nd assertion fail. [1] http://paste.ubuntu.com/8556130/ |
|
|
| msg229279 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2014-10-14 07:58 |
| This has nothing to do with the datetime module. Attached script reduces the issue to a bug (?) in time.mktime() with the corresponding timezone. time.mktime() is a thin wrapper around the C library's mktime() function, so it is probably not a bug in Python at all. Note your script is fixed by removing ".replace(tzinfo=None)". It seems conter-productive to take the pain to create an aware timezone and then make it naive, IMHO. datetime.timestamp() falls back on time.mktime() when the datetime is naive (i.e. it asks the OS to do the computation). (I did my tests under Ubuntu 13.10, btw) |
|
|
| msg229344 - (view) |
Author: Christopher Lee (veebers) |
Date: 2014-10-14 20:13 |
| Hi Antoine, thanks for taking a look. I should explain further. This code is for an introspection tool[1] that provides an interface to write tests in python against an application. This code is a workaround for using large timestamps[2] (i.e. larger than 32bit time_t) the original suggested code (from SO, using timedelta) has an inconsistancy with the hour in some cases (this example in the NZ timezone): >>> datetime.datetime.fromtimestamp(2047570047) datetime.datetime(2034, 11, 19, 17, 27, 27) >>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047) datetime.datetime(2034, 11, 19, 18, 27, 27) The code you see here in my example, creating a timezone aware object, is a result of 'fixing' this behaviour I've seen. The reason for then removing the tzinfo is because currently our API states that dates returned from the introspected objects are naive. [1] https://launchpad.net/autopilot [2] https://bugs.launchpad.net/autopilot/+bug/1328600 |
|
|
| msg229388 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2014-10-15 00:17 |
| In general, you cannot expect datetime.fromtimestamp(0) + timedelta(seconds) to return the same value as datetime.fromtimestamp(seconds). This will only be true if you are lucky enough to live in an area where local government did not mess with timezones since 1970. In your particular case, I think what is happening is that you have DST in January, but you did not have that on January 1, 1970. $ zdump -v NZ | grep NZDT |
head NZ Sat Nov 2 14:00:00 1974 UTC = Sun Nov 3 03:00:00 1974 NZDT isdst=1 NZ Sat Feb 22 13:59:59 1975 UTC = Sun Feb 23 02:59:59 1975 NZDT isdst=1 NZ Sat Oct 25 14:00:00 1975 UTC = Sun Oct 26 03:00:00 1975 NZDT isdst=1 NZ Sat Mar 6 13:59:59 1976 UTC = Sun Mar 7 02:59:59 1976 NZDT isdst=1 NZ Sat Oct 30 14:00:00 1976 UTC = Sun Oct 31 03:00:00 1976 NZDT isdst=1 NZ Sat Mar 5 13:59:59 1977 UTC = Sun Mar 6 02:59:59 1977 NZDT isdst=1 NZ Sat Oct 29 14:00:00 1977 UTC = Sun Oct 30 03:00:00 1977 NZDT isdst=1 NZ Sat Mar 4 13:59:59 1978 UTC = Sun Mar 5 02:59:59 1978 NZDT isdst=1 NZ Sat Oct 28 14:00:00 1978 UTC = Sun Oct 29 03:00:00 1978 NZDT isdst=1 NZ Sat Mar 3 13:59:59 1979 UTC = Sun Mar 4 02:59:59 1979 NZDT isdst=1 |
|
| msg229390 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2014-10-15 01:01 |
| Antoine, I don't think the behavior that you have shown is a bug in strict sense. On my Mac, I get $ python mkbug.py breakme 1396702800.0 1396702800.0 but on Linux, $ python mkbug.py breakme 1396706400.0 1396702800.0 The problem here is that time.mktime((2014, 4, 6, 2, 0, 0, -1, -1, -1)) is undefined and both 1396706400 and 1396702800 timestamps are valid guesses for what 2014-04-06T02 was in NZ: $ TZ=NZ date -d @1396706400 Sun Apr 6 02:00:00 NZST 2014 $ TZ=NZ date -d @1396702800 Sun Apr 6 02:00:00 NZDT 2014 It is unfortunate that Linux C library (glibc?) makes different guess at different time, but I don't think this violates any applicable standards. |
|
|
| msg229391 - (view) |
Author: Christopher Lee (veebers) |
Date: 2014-10-15 01:05 |
| Alexander, Ah ok thanks for clarifying that. Am I wrong then to think that this code[1] should work as I think it should (i.e. datetime_from_large_timestamp(example_ts) == datetime.fromtimestamp(example_ts)) I'm trying to be able to handle timestamps larger than the 32bit time_t limit, which is why I'm doing these gymnastic steps. [1] http://paste.ubuntu.com/8562027/ |
|
|
| msg229393 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2014-10-15 01:45 |
| Your code as good as your timezone library, but you should realize that by discarding tzinfo you are making your "local_stamp" ambiguous. I am not familiar with dateutil.tz, but pytz I believe uses some tricks to make sure astimezone() result remembers the isdst flag. If you API requires naive local datetime objects, you need to carry isdst flag separately if you want to disambiguate between 2014-04-06 02:00 NZST and 2014-04-06 02:00 NZDT. On top of that, you will not be able to use datetime.timestamp() method and will have to use time.mktime or whatever equivalent utility your timezone library provides. Note that I was against adding datetime.timestamp() for this specific reason: it is supposed to be inverse of datetime.fromtimestamp(), but since the later is not monotonic, no such inverse exists in the strict mathematical sense. See in issue 2736. BTW, if you open a feature request to add isdst=-1 optional argument to datetime.timestamp(), you will have my +1. |
|
|
| msg229395 - (view) |
Author: Akira Li (akira) * |
Date: 2014-10-15 02:07 |
| Christopher, About your script http://paste.ubuntu.com/8562027/ dateutil may break if the local timezone had different UTC offset in the past. You could use tzlocal module to get pytz timezone that can handle such timezones. To get the correct time for 1414274400 timezone in Europe/Moscow timezone, you need the latest tz database e.g., `pytz` works but fromtimestamp, dateutil that use the local tz database fail (2:00 instead of 1:00): >>> import time >>> import os >>> os.environ['TZ'] = 'Europe/Moscow' >>> time.tzset() >>> from datetime import datetime, timezone >>> from tzlocal import get_localzone >>> datetime.fromtimestamp(1414274400, get_localzone()) datetime.datetime(2014, 10, 26, 1, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>) >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=timezone.utc).astimezone(get_localzone()) datetime.datetime(2014, 10, 26, 1, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>) >>> datetime.fromtimestamp(1414274400) # wrong datetime.datetime(2014, 10, 26, 2, 0) >>> datetime.fromtimestamp(1414274400, timezone.utc).astimezone() # wrong datetime.datetime(2014, 10, 26, 2, 0, tzinfo=datetime.timezone(datetime.timedelta(0, 14400), 'MSK')) >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=timezone.utc).astimezone() # wrong datetime.datetime(2014, 10, 26, 2, 0, tzinfo=datetime.timezone(datetime.timedelta(0, 14400), 'MSK')) >>> from dateutil.tz import gettz, tzutc >>> datetime.fromtimestamp(1414274400, gettz()) # wrong datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow')) >>> datetime.fromtimestamp(1414274400, tzutc()).astimezone(gettz()) # wrong datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow')) >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=tzutc()).astimezone(gettz()) # wrong datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow')) To avoid surprises, always use UTC time to perform date arithmetics: EPOCH = datetime(1970, 1,1, tzinfo=pytz.utc) utc_dt = EPOCH + timedelta(seconds=timestamp) should work for dates after 2038 too. To convert it to the local timezone: from tzlocal import get_localzone local_dt = utc_dt.astimezone(get_localzone()) ts = (local_dt - EPOCH) // timedelta(seconds=1) assert ts == timestamp # if timestamp is an integer Python stdlib assumes POSIX encoding for time.time() value () therefore the formulae work on all platforms where Python works. |
|
|
| msg229401 - (view) |
Author: Christopher Lee (veebers) |
Date: 2014-10-15 04:10 |
| Thanks Akira, everyone for all the info. It looks like I've highjacked this bug comments to trying to solve my first problem (i.e. datetime objects for large timestamps) instead of the bug at hand, I feel should move that conversation elsewhere. It appears from Alexander's comment that it might not be a bug. I need to figure out if I need to work around this or use some other mechanism. |
|
|
| msg229405 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-10-15 08:00 |
| Christopher, I've already pointed out a fix in another message: just remove ".replace(tzinfo=None)". Doing computations on UTC datetimes (rather than naive) should ensure you don't encounter any ambiguities. |
|
|
| msg250859 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2015-09-16 18:48 |
| I am reopening this issue as an "enhancement" because I would like to revisit it in light of PEP 495. |
|
|
| msg275684 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2016-09-10 19:39 |
| Superseded by #28067. |
|
|