Issue 19475: Add timespec optional flag to datetime isoformat() to choose the precision (original) (raw)
Created on 2013-11-01 17:05 by skip.montanaro, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (94)
Author: Skip Montanaro (skip.montanaro) * 
Date: 2013-11-01 17:05
I have a CSV file. Here are a few rows:
"2013-10-30 14:26:46.000528","1.36097023829" "2013-10-30 14:26:46.999755","1.36097023829" "2013-10-30 14:26:47.999308","1.36097023829" "2013-10-30 14:26:49.002472","1.36097023829" "2013-10-30 14:26:50","1.36097023829" "2013-10-30 14:26:51.000549","1.36097023829" "2013-10-30 14:26:51.999315","1.36097023829" "2013-10-30 14:26:52.999703","1.36097023829" "2013-10-30 14:26:53.999640","1.36097023829" "2013-10-30 14:26:54.999139","1.36097023829"
I want to parse the strings in the first column as timestamps. I can, and often do, use dateutil.parser.parse(), but in situations like this where all the timestamps are of the same format, it can be incredibly slow. OTOH, there is no single format I can pass to datetime.datetime.strptime() that will parse all the above timestamps. Using "%Y-%m-%d %H:%M:%S" I get errors about the leftover microseconds. Using "%Y-%m-%d %H:%M:%S".%f" I get errors when I try to parse a timestamp which doesn't have microseconds.
Alas, it is datetime itself which is to blame for this problem. The above timestamps were all printed from an earlier Python program which just dumps the str() of a datetime object to its output CSV file. Consider:
dt = dateutil.parser.parse("2013-10-30 14:26:50") print dt 2013-10-30 14:26:50 dt2 = dateutil.parser.parse("2013-10-30 14:26:51.000549") print dt2 2013-10-30 14:26:51.000549
The same holds for isoformat():
print dt.isoformat() 2013-10-30T14:26:50 print dt2.isoformat() 2013-10-30T14:26:51.000549
Whatever happened to "be strict in what you send, but generous in what you receive"? If strptime() is going to complain the way it does, then str() should always generate a full timestamp, including microseconds. The above is from a Python 2.7 session, but I also confirmed that Python 3.3 behaves the same.
I've checked 2.7 and 3.3 in the Versions list, but I don't think it can be fixed there. Can the str and isoformat methods of datetime (and time) objects be modified for 3.4 to always include the microseconds? Alternatively, can the %S format character be modified to consume optional decimal point and microseconds? I rate this as "easy" considering the easiest fix is to modify str and isoformat, which seems unchallenging.
Author: Ezio Melotti (ezio.melotti) * 
Date: 2013-11-01 17:08
See #7342.
Author: R. David Murray (r.david.murray) *
Date: 2013-11-01 17:47
It may be simple but as Ezio has pointed out, it has already been rejected :)
The problem with being generous in what you accept in this context is that the parsing is using a specific format string, and the semantics of that format string are based on external "standards" and are pretty inflexible.
The pythonic solution, IMO, is to have datetime's constructor accept what its str produces. And indeed, exactly this has been suggested by Alexander Belopolsky in issue 15873. So I'm going to close this one as a duplicate of that one.
Author: Skip Montanaro (skip.montanaro) *
Date: 2013-11-01 17:55
I don't accept your conclusion. I understand that making %S consume microseconds or ".%f" be "optional" would be a load. What's the problem with forcing str and isoformat to emit microseconds in all cases though? That would allow you to parse what they produce using existing code. No new constructor needed.
The issue of sometimes emitting microseconds, sometimes not, is annoying, even beyond this issue. I think for consistency's sake it makes sense for the string version of datetime and time objects to always be the same length.
Author: R. David Murray (r.david.murray) *
Date: 2013-11-01 18:01
It's not my conclusion. It's Guido's and the other developers who designed datetime. Argue with them. (I'd guess it would be better argued on python-ideas rather than python-dev, but use your own judgement.)
Author: Tim Peters (tim.peters) *
Date: 2013-11-01 18:07
The decision to omit microseconds when 0 was a Guido pronouncement, back when datetime was first written. The idea is that str() is supposed to be friendly, and for the vast number of applications that don't use microseconds at all, it's unfriendly to shove ".000000" in their face all the time. Much the same reason is behind why, e.g., str(2.0) doesn't produce "2.0000000000000000".
I doubt this will change. If you want to use a single format, you could massage the data first, like
if '.' not in dt: dt += ".000000"
Author: Skip Montanaro (skip.montanaro) *
Date: 2013-11-01 18:12
Okay, so no to str. What about isoformat?
Author: Tim Peters (tim.peters) *
Date: 2013-11-01 18:25
I don't know, Skip. Since .isoformat() and str() have always worked this way, and that was intentional, it's probably going to take a strong argument to change either.
Author: Guido van Rossum (gvanrossum) *
Date: 2013-11-01 18:40
Well, I don't know if this sways anything, but I was probably responsible, and I think my argument was something about not all timestamp sources having microseconds, and not wanting to emit the ".000000" in that case. If I could go back I'd probably do something else; after all str(1.0) doesn't return '1' either. But that's water under the bridge; "fixing" this is undoubtedly going to break a lot of code.
Maybe we can give isoformat() a flag parameter to force the inclusion or exclusion of the microseconds (with a default of None meaning the current behavior)?
Author: Skip Montanaro (skip.montanaro) *
Date: 2013-11-01 18:50
The ultimate culprit here is actually the csv module. :-) It calls str() on every element it's about to write. In my applications which write to CSV files I can special case datetime objects.
I will stop swimming upstream.
Author: R. David Murray (r.david.murray) *
Date: 2013-11-01 18:55
I suppose in an ideal world the csv module would have some sort of hookable serialization protocol, like the database modules do :)
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-11-01 22:14
As I understand Guido's message, he reopened this to consider adding a new parameter.
Given an existing csv file like that given, either Tim's solution or try: parse with microseconds except ValueError: parse without should work.
Author: Alexander Belopolsky (belopolsky) *
Date: 2013-11-05 15:48
+1 on adding an option to isoformat(). We already have an optional argument, so the symmetry with str is not complete. To make this option more useful, rather than implementing always_emit_microseconds=False flag, I would add a keyword argument 'precision' that would take ('hour'|'minute'|'second'|millisecond'|'microsecond') value.
Author: -- (elixir) *
Date: 2013-11-05 20:14
I would like to implement this feature. I already wrote the Python part. Is there anything else to decide?
Author: STINNER Victor (vstinner) *
Date: 2013-11-05 20:31
2013/11/5 Alexander Belopolsky <report@bugs.python.org>:
+1 on adding an option to isoformat(). We already have an optional argument, so the symmetry with str is not complete. To make this option more useful, rather than implementing always_emit_microseconds=False flag, I would add a keyword argument 'precision' that would take ('hour'|'minute'|'second'|millisecond'|'microsecond') value.
Hour precision is not part of the ISO 8601 standard.
"resolution" is maybe a better name for the new parameter than "precision": http://www.python.org/dev/peps/pep-0418/#glossary
The new parameter should be added to datetime.datetime.isoformat() but also datetime.time.isoformat().
Author: Alexander Belopolsky (belopolsky) *
Date: 2013-11-05 21:24
+1 on all Victor's points.
I like 'resolution' because this is the term that datetime module uses already:
from datetime import * datetime.resolution datetime.timedelta(0, 0, 1)
There is a slight chance of confusion stemming from the fact that datetime.resolution is timedelta, but proposed parameter is a string.
I believe ISO 8601 uses the word "accuracy" to describe this kind of format variations. I am leaning towards "resolution", but would like to hear from others. Here are the candidates:
- resolution
- accuracy
- precision
(Note that "accuracy" is the shortest but "resolution" is the most correct.)
Author: Marc-Andre Lemburg (lemburg) *
Date: 2013-11-05 21:32
On 05.11.2013 21:31, STINNER Victor wrote:
2013/11/5 Alexander Belopolsky <report@bugs.python.org>:
+1 on adding an option to isoformat(). We already have an optional argument, so the symmetry with str is not complete. To make this option more useful, rather than implementing always_emit_microseconds=False flag, I would add a keyword argument 'precision' that would take ('hour'|'minute'|'second'|millisecond'|'microsecond') value.
Hour precision is not part of the ISO 8601 standard.
"resolution" is maybe a better name for the new parameter than "precision": http://www.python.org/dev/peps/pep-0418/#glossary
The new parameter should be added to datetime.datetime.isoformat() but also datetime.time.isoformat().
Since this ticket is about being able to remove the seconds fraction part, I think it's better to use a name that is not already overloaded with other meanings, e.g. show_us=False or show_microseconds=False.
BTW: Have you thought about the rounding/truncation issues associated with not showing microseconds ?
A safe bet is truncation, but this can lead to inaccuracies of up to a second. Rounding is difficult, since it can lead to a "60" second value showing up for e.g. 11:00:59.95 seconds, or the need to return "12:00:00" for 11:59:59.95.
Author: Alexander Belopolsky (belopolsky) *
Date: 2013-11-06 15:51
MAL: Have you thought about the rounding/truncation issues associated with not showing microseconds ?
I believe it has to be the truncation. Rounding is better left to the user code where it can be done either using timedelta arithmetics or at the time source. I would expect that in the majority of cases where lower resolution printing is desired the times will be already at lower resolution at the source.
Author: Marc-Andre Lemburg (lemburg) *
Date: 2013-11-06 17:05
On 06.11.2013 16:51, Alexander Belopolsky wrote:
MAL: Have you thought about the rounding/truncation issues associated with not showing microseconds ?
Sure, otherwise I wouldn't have mentioned it :-)
mxDateTime always uses 2 digit fractions when displaying date/time values. This has turned out to be a good compromise between accuracy and usability. In early version, I used truncation, but that caused (too many) roundtrip problems, so I started using careful rounding in later versions:
/* Fix a second value for display as string.
Seconds are rounded to the nearest microsecond in order to avoid cases where e.g. 3.42 gets displayed as 03.41 or 3.425 is diplayed as 03.42.
Special care is taken for second values which would cause rounding to 60.00 -- these values are truncated to 59.99 to avoid the value of 60.00 due to rounding to show up even when the indicated time does not point to a leap second. The same is applied for rounding towards 61.00 (leap seconds).
The second value returned by this function should be formatted using '%05.2f' (which rounds to 2 decimal places).
*/
This approach has worked out well, though YMMV.
I believe it has to be the truncation. Rounding is better left to the user code where it can be done either using timedelta arithmetics or at the time source. I would expect that in the majority of cases where lower resolution printing is desired the times will be already at lower resolution at the source.
In practice you often don't know the resolution of the timing source. Nowadays, the reverse of what you said is usually true: the source resolution is higher than the precision you use to print it.
MS SQL Server datetime is the exception to that rule, with a resolution of 333ms and weird input "rounding":
http://msdn.microsoft.com/en-us/library/ms187819.aspx
For full seconds, truncation will add an error of +/- 1 second, whereas rounding only adds +/- 0.5 seconds. This is what convinced me to use rounding instead of truncation.
Author: Alexander Belopolsky (belopolsky) *
Date: 2013-11-06 17:20
I am afraid that the rounding issues may kill this proposal. Can we start with something simple? For example, we can start with show=None keyword argument and allow a single value 'microseconds' (or 'us'). This will solve the issue at hand with a reasonable syntax: t.isoformat(show='us'). If other resolutions will be required, we can later add more values and may even allow t.isoformat(show=2) to show 2 decimal digits.
Author: Skip Montanaro (skip.montanaro) *
Date: 2013-11-06 18:22
I am afraid that the rounding issues may kill this proposal. Can we start with something simple? For example, we can start with show=None keyword argument and allow a single value 'microseconds' (or 'us'). This will solve the issue at hand with a reasonable syntax: t.isoformat(show='us'). If other resolutions will be required, we can later add more values and may even allow t.isoformat(show=2) to show 2 decimal digits.
I don't think the meaning of this proposed show keyword argument should be overloaded as you suggest. If you show microseconds, just show all of them.
Furthermore...
If we go far enough back, my original problem was really that the inclusion of microseconds in csv module output was inconsistent, making it impossible for me to later parse those values in another script using a fixed strptime format. Since the csv module uses str() to convert input values for output, nothing you do to isoformat() will have any effect on my original problem.
In my own code (where I first noticed the problem) I acquiesced, and changed this
d["time"] = now
to this:
d["time"] = now.strftime("%Y-%m-%dT%H:%M:%S.%f")
where "now" is a datetime object. I thus guarantee that I can parse these timestamps later using the same format. I realize the inclusion of "T" means my fields changed in other ways, but that was intentional, and not germane to this discussion.
So, fiddle all you want with isoformat(), but do it right. I vote that if you want to add a show parameter it should simply include all fields down to that level, omitting any lower down. If people want to round or truncate things you can give them that option, returning a suitably adjusted, new datetime object. I don't think rounding, truncation, or other numeric operations should be an element of conversion to string form. This does not happen today:
import datetime x = datetime.datetime.now() x datetime.datetime(2013, 11, 6, 12, 19, 5, 759020) x.strftime("%Y-%m-%d %H:%M:%S") '2013-11-06 12:19:05'
(%S doesn't produce "06")
Skip
Author: Alexander Belopolsky (belopolsky) *
Date: 2014-06-30 13:26
Here is some "prior art": GNU date utility has an --iso-8601[=timespec] option defined as
‘-I[timespec]’ ‘--iso-8601[=timespec]’ Display the date using the ISO 8601 format, ‘%Y-%m-%d’. The argument timespec specifies the number of additional terms of the time to include. It can be one of the following:
‘auto’ Print just the date. This is the default if timespec is omitted. ‘hours’ Append the hour of the day to the date. ‘minutes’ Append the hours and minutes. ‘seconds’ Append the hours, minutes and seconds. ‘ns’ Append the hours, minutes, seconds and nanoseconds. If showing any time terms, then include the time zone using the format ‘%z’.
https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.html
Author: Alexander Belopolsky (belopolsky) *
Date: 2014-06-30 13:33
Based on GNU date "prior art", we can introduce timespec='auto' keyword argument with the following values:
'auto' - (default) same as current behavior 'hours' - %H 'minutes' - %H:%M 'seconds' - %H:%M:%S 'us' - %H:%M:%S.%f
Author: STINNER Victor (vstinner) *
Date: 2015-07-24 09:53
'seconds' - %H:%M:%S 'us' - %H:%M:%S.%f
'us' is not consistent with the datetime module: it should be 'microseconds.
datetime.datetime.now().second 50 datetime.timedelta(seconds=1) datetime.timedelta(0, 1)
datetime.datetime.now().microsecond 123710 datetime.timedelta(microseconds=1) datetime.timedelta(0, 0, 1)
Author: STINNER Victor (vstinner) *
Date: 2015-07-25 22:28
@acucci: Nice first try, but your patch contains multiple bugs.
Author: Alessandro Cucci (acucci) *
Date: 2015-07-25 23:11
@haypo thanks for the review and the suggestions, I'll correct the code soon
Author: Alessandro Cucci (acucci) *
Date: 2015-07-26 16:52
udloaded a new patch, hope i had all bugs fixed!
Author: Berker Peksag (berker.peksag) *
Date: 2015-07-31 01:21
Thanks, Alessandro! I left some additional comments on Rietveld: https://bugs.python.org/review/19475/#ps15278
Author: Alessandro Cucci (acucci) *
Date: 2015-08-03 19:46
uploaded a new patch!
Author: Alessandro Cucci (acucci) *
Date: 2015-12-15 13:10
Please can I have an update on this? It's already 4 months old, I'd like to see it closed. :)
Author: Guido van Rossum (gvanrossum) *
Date: 2015-12-15 17:34
I like the idea. I have one suggestion: can we add 'milliseconds' as an option too? And might a well add 'nanoseconds' too, for future-proofing. I suppose there isn't a real use case for 'hours' but it seems silly to leave it out. I expect that 'minutes' will be the most popular option, since HH:MM is the precision that most people care about for meetings and stuff.
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-15 19:33
It looks like issue19475_v3.patch uses some fancy Unicode quotes in the docstrings and .rst docs. Please change them to ASCII.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-15 19:56
I can work on that, although I'll need help from some senior dev. The problem here is that millisecond and nanosecond seems not to be attributes of the datetime object. What about open a new issue if we have to add them? Is not about adding an optional flag anymore... Anyway, as I said, any help is appreciated!
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-15 19:58
GvR> I suppose there isn't a real use case for 'hours' but it seems silly to leave it out.
Shouldn't we also have 'none' to leave out the time component entirely?
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-15 20:01
The problem here is that millisecond and nanosecond seems not to be attributes of the datetime object.
millisecond = dt.microsecond // 1000
nanosecond = 0 # until we add it to datetime.
Author: Guido van Rossum (gvanrossum) *
Date: 2015-12-15 20:11
Actually, nanosecond = dt.microsecond*1000.
I don't think we need 'none' -- you should just extract the date component and call its isoformat() method if that's what you want.
On Tue, Dec 15, 2015 at 12:01 PM, Alexander Belopolsky < report@bugs.python.org> wrote:
Alexander Belopolsky added the comment:
The problem here is that millisecond and nanosecond seems not to be attributes of the datetime object.
millisecond = dt.microsecond // 1000
nanosecond = 0 # until we add it to datetime.
Python tracker <report@bugs.python.org> <http://bugs.python.org/issue19475>
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-15 20:22
Actually, nanosecond = dt.microsecond*1000.
I was thinking in terms breaking the fractional part of say
00:00:00.123456789
into
mili = 123 micro = 456 nano = 789
but you are right, a correct analogy for dt.microsecond in this case will be nanosecond=123456789 or 123456000 until we start storing enough precision.
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-15 20:27
I think timespec= option should also be added to the time.isoformat method.
Author: Martin Panter (martin.panter) *
Date: 2015-12-15 20:39
Alessandro, did you try running the test suite? It is broken for me. One of the bugs is that it now omits zero hours, minutes, and seconds in addition to microseconds. (Kind of the reverse of what the bug was originally about :).
Other failures look like this:
====================================================================== ERROR: test_isoformat (test.datetimetester.TestDateTime)
Traceback (most recent call last): File "/media/disk/home/proj/python/cpython/Lib/test/datetimetester.py", line 1560, in test_isoformat timespec_error = self.time() AttributeError: 'TestDateTime_Pure' object has no attribute 'time'
I will leave some review comments.
Author: Martin Panter (martin.panter) *
Date: 2015-12-15 21:07
The doc string in the C module needs updating.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-16 07:35
Thanks for all the comments, here and on Rietveld. I'll keep working on it. Hope to upload a new patch soon.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-16 21:47
Uploaded a new patch (v4): DONE:
- now all tests work (I also added one more)
- removed non ascii chars
- added milliseconds and nanoseconds as multiple of microseconds
- removed code duplication in the datetime.py
- update the docstring.
TODO:
The doc string in the C module needs updating. Where? I guess "Macros to extract fields from datetime objects." in c-api/datetime.rst Tell me if I'm wrong
I think timespec= option should also be added to the time.isoformat method. I'll do it when I'm sure that the datetime.isoformat method is correct.
Author: Martin Panter (martin.panter) *
Date: 2015-12-16 22:41
With patch v4 many tests still fail for me. How are you running the test suite? Some excerpts:
File "/media/disk/home/proj/python/cpython/Lib/test/datetimetester.py", line 1567, in test_isoformat self.assertEqual(t.isoformat(timespec='milliseconds'), "0001-02-03T04:05:01.000") AssertionError: '0001-02-03T04:05:01.000000' != '0001-02-03T04:05:01.000'
File "/media/disk/home/proj/python/cpython/Lib/test/datetimetester.py", line 2310, in test_isoformat self.assertEqual(t.isoformat(), "00:00:00") AssertionError: '00:00:00.000000' != '00:00:00'
File "/media/disk/home/proj/python/cpython/Lib/test/datetimetester.py", line 2379, in test_str self.assertEqual(str(self.theclass(12, 2, 3, 0)), "12:02:03") AssertionError: '12:02:03.000000' != '12:02:03'
The C module doc string I referred to is this one; I left a review comment at the source:
print(datetime.isoformat.doc) [sep] -> string in ISO 8601 format, YYYY-MM-DDTHH:MM:SS[.mmmmmm][+HH:MM].
sep is used to separate the year from the time, and defaults to 'T'.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-16 22:50
I just did that...
~/Documenti/cpython$ ./configure --with-pydebug ~/Documenti/cpython$ make -s -j2 ~/Documenti/cpython$ ./python -m test -v datetimetester ... ...
Ran 325 tests in 1.128s
OK (skipped=1) 1 test OK.
Author: SilentGhost (SilentGhost) *
Date: 2015-12-16 23:01
Even that produces 3 failures on my setup, though the full list would be available when running: ./python -m test -v test_datetime
It is obvious, however, that the tests would fail: you've changed "if us:" to "if us is None:".
Author: Alexander Belopolsky (belopolsky) *
Date: 2015-12-16 23:03
~/Documenti/cpython$ ./python -m test -v datetimetester
You shouldn't use "datetimetester" on the command line. Use "test_datetime" instead.
Author: Martin Panter (martin.panter) *
Date: 2015-12-17 08:35
If the timespec allowed any arbitrary number of digits after the decimal point, that would remove any argument about nanoseconds not being supported. It would also mean I could use this in one case where I currently format to two decimal places (my compromise between accurate timestamps and excessive information). Just a suggestion :)
Author: Alessandro Cucci (acucci) *
Date: 2015-12-20 12:25
Uploaded a new patch.
This time I ran tests using: $ ./python -m test -v test_datetime
and got no errors.
I've written the c docstring, updated docs, and rewrote _format_time function.
I know it's boring review this another time, but I'm doing my best.
Thanks to all.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-21 22:33
Uploaded a new patch after SilentGhost review comments.
As he told me, I've left out milliseconds and nanoseconds, but refactored both python and c code so we could easily add support for them when they will be available.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-22 19:25
Sorry @SilentGhost, I didn't seen your comment at the bottom of the review. With this patch is microsecond is 0, the time will display 6 zeroes.
Author: SilentGhost (SilentGhost) *
Date: 2015-12-22 20:08
I think the patch is nearly finalised, but I'd appreciate if someone else would carefully go over the new C code. After that, I think, the patch could be committed.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-22 20:53
Thanks SilentGhost!
Author: Alessandro Cucci (acucci) *
Date: 2015-12-24 18:47
Can anyone please review the c code of the last patch?
Author: Alessandro Cucci (acucci) *
Date: 2015-12-25 14:26
I think timespec= option should also be added to the time.isoformat method.
@belopolsky I've done it in my last patch.
Author: Berker Peksag (berker.peksag) *
Date: 2015-12-29 16:08
Thanks for the patch, Alessandro. I left some comments about documentation part of the patch, but I can fix them myself if you don't have time.
Author: Alessandro Cucci (acucci) *
Date: 2015-12-29 18:16
what about the comment left by SilentGhost about versionadded?
Author: Alessandro Cucci (acucci) *
Date: 2015-12-29 18:44
Berker, thank you. In the last patch, I removed details about timespec options in Python and C docstrings, corrected the rst quotes, and checked PEP7 in the c file.
The only problem now is about versionchanged vs versionadded. I leave it as it was, as Silent and the official doc say, if you want to change it, i'll leave it to you.
Again, many many thanks to all. This was my first issue here, I learnt a lot!
Author: Martin Panter (martin.panter) *
Date: 2016-01-01 10:39
I think there is a memory leak in the C code. I left some other minor suggestions as well, but it almost looks ready.
Author: Alessandro Cucci (acucci) *
Date: 2016-01-01 12:50
Thanks @martin.panter, here is another patch made after your comments.
Author: Alessandro Cucci (acucci) *
Date: 2016-01-05 15:30
up
Author: Alessandro Cucci (acucci) *
Date: 2016-01-15 12:18
I there anything else I can do for this?
Author: SilentGhost (SilentGhost) *
Date: 2016-01-17 15:38
I there anything else I can do for this? I think you've done all you could, it's just someone needs to commit it. I don't think they're being rude on purpose - it's just that there are so many committers in the nosy list that no one is feeling like it's their job.
Author: Guido van Rossum (gvanrossum) *
Date: 2016-01-17 18:52
I think Alexander has commit rights, he did most of the review, I trust him to commit it.
Author: Terry J. Reedy (terry.reedy) *
Date: 2016-01-17 19:46
Not seeing any indication that this has been tested on Windows, I applied, compiled (32 bit), and ran test_datetime on Win10 without error. I did not rebuild the doc.
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-01-17 20:33
Is issue19475_v12.patch the final patch? I don't see it addressing Guido's suggestion in to add milli- and nanoseconds options.
Author: SilentGhost (SilentGhost) *
Date: 2016-01-17 20:46
Yes, version 12 is the final patch. It doesn't add those options. To copy my opinion from the rietveld (https://bugs.python.org/review/19475/#msg14):
I don't really think nanoseconds belong here. If they don't exist anywhere else in the module, why should they be suddenly introduced here? of all places. It would be fine to use some generic solution that would enable or ease their addition in the future, but they shall not be added at this time.
(there are altogether 27 messages there, which everyone's naturally CCed on).
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-01-17 21:23
I left some comments on Rietveld.
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-01-17 21:33
I don't really think nanoseconds belong here.
What about milliseconds? I'll leave it for Guido to make a call on nanoseconds. My vote is +0.5.
If they don't exist anywhere else in the module, why should they be suddenly introduced here?
The timespec feature is modeled after GNU date --iso-8601[=timespec] option which does support nanoseconds. It is fairly common to support nanoseconds these days and it does not cost much to implement.
Author: SilentGhost (SilentGhost) *
Date: 2016-01-17 22:07
What about milliseconds? I'll leave it for Guido to make a call on nanoseconds. My vote is +0.5. The only reason I didn't mention milliseconds because they exist in timedelta instantiation. And really, being the only place in the whole module they're as confusing there as would be nanoseconds.
If they don't exist anywhere else in the module, why should they be suddenly introduced here?
The timespec feature is modeled after GNU date --iso-8601[=timespec] option which does support nanoseconds. It is fairly common to support nanoseconds these days and it does not cost much to implement.
Yes, but the module does not support nanoseconds. And putting any such options would require a huge banner saying that the nanosecond option will just always result in three zeros at the end. My suggestion is not to pretend that we suddenly "support" nanoseconds, but rather to follow the actual implementation of the module and add the support for nanoseconds timespec when the module actually adds support for them.
Author: Guido van Rossum (gvanrossum) *
Date: 2016-01-18 02:59
You can leave out the nanoseconds but please do add the milliseconds. I'm sure they would find more use than the option to show only the hours.
Author: Alessandro Cucci (acucci) *
Date: 2016-02-21 18:20
New patch
Author: Martin Panter (martin.panter) *
Date: 2016-02-21 22:32
Left some review suggestions
Author: Alessandro Cucci (acucci) *
Date: 2016-02-22 18:15
New patch after @martin.panter comments on Rietveld. I left only this:
'milliseconds': Append the hours, minutes, seconds and milliseconds.
vadmium 2016/02/21 23:30:20 I think this should explain that fractions are truncated to zero, never rounded up. At least for fractions of milliseconds, although this could apply to the other options as well.
I think is quite obvious that a datetime.now() can't be rounded to the future if microseconds are 999500.
Author: Martin Panter (martin.panter) *
Date: 2016-02-23 10:33
About rounding: I’m not too sure what people would expect. Obviously it is much easier to implement truncating to zero. But it is different to many other rounding cases in Python; that is why I thought to make it explicit.
datetime.fromtimestamp(59.9999999).isoformat(timespec="microseconds") '1970-01-01T00:01:00.000000' datetime.fromtimestamp(59.999999).isoformat(timespec="milliseconds") '1970-01-01T00:00:59.999' format(59.999999, ".3f") '60.000'
Author: Alessandro Cucci (acucci) *
Date: 2016-02-25 18:15
Oh, now I see your point.
I've uploaded a new patch with a note for that.
Author: Guido van Rossum (gvanrossum) *
Date: 2016-02-25 18:29
Out of context here, but regarding round vs. truncate, IIUC for time truncating down is the norm. My digital clock shows "12:00" for the duration of the minute starting at noon. People look for clocks to flip to know when it is exactly a given time (if the clock is accurate enough).
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-02-25 18:49
We discussed truncation vs. rounding some time ago. See and the posts around it. The consensus was the same as Guido's current advise: do the truncation.
Author: Alessandro Cucci (acucci) *
Date: 2016-03-01 18:05
@belopolsky could you please review one of the latest two patches submitted? I think I've done all required. Now I'll wait from you if I have to do more.
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-01 20:11
Guido,
Did you consider MAL's ? I am still in favor of truncation, but would like to make sure we are not missing something that MAL knows from experience.
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-01 20:22
Another argument for truncation is that this is what GNU date does:
$ date --iso-8601=seconds --date="2016-03-01 15:00:00.999" 2016-03-01T15:00:00-0500
Author: Guido van Rossum (gvanrossum) *
Date: 2016-03-01 21:13
Given that we're talking about what to do when we're suppressing the usecs I don't think roundtripping matters. :-)
Author: STINNER Victor (vstinner) *
Date: 2016-03-01 21:16
Given that we're talking about what to do when we're suppressing the usecs I don't think roundtripping matters. :-)
I changed many times how Python rounds nanoseconds in the private PyTime API, and I got a bug report because of that! => issue #23517.
By the way, I wrote an article to explain the history the private PyTime API, especially changes on rounding ;-) https://haypo.github.io/pytime.html
Author: Guido van Rossum (gvanrossum) *
Date: 2016-03-01 21:32
But what should we do in your opinion?
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-01 21:35
I hope my prediction "I am afraid that the rounding issues may kill this proposal" (see ) will not come true.
I think the correct way to view "timespec" is a way to suppress/enforce printing of trailing digits.
Users that choose printing less than full usec format should make sure that their datetime instances are properly rounded before printing.
Unfortunately, I does not look like the datetime module makes rounding easy. The best I can think of is something like
def round_datetime(dt, delta): dt0 = datetime.combine(dt.date(), time(0)) return dt0 + round((dt - dt0) / delta) * delta
Maybe a datetime.round() method along these lines will be a worthwhile addition?
Author: STINNER Victor (vstinner) *
Date: 2016-03-01 21:47
But what should we do in your opinion?
Use ROUND_FLOOR rounding method.
time.time(), datetime.datetime.now(), etc. round the current time using the ROUND_FLOOR rounding method.
Only datetime.datetime.fromtimestamp() uses ROUND_HALF_EVEN, but it's more an exception than the rule: this function uses a float as input. To be consistent, we must use the same rounding method than other Python functions taking float as parameter, like round(), so use ROUND_HALF_EVEN.
So I suggest to also use ROUND_FLOOR for .isoformat().
Hopefully, we don't have to discuss about the exact rounding method for negative numbers, since the minimum datetime object is datetime.datetime(1, 1, 1) which is "positive" ;-)
You have a similar rounding question for file timestamps. Depending on the file system, you may have a resolution of 2 seconds (FAT), 1 second (ext3) or 1 nanosecond (ext4). But Linux syscalls accept subsecond resolution. The Linux kernel uses ROUND_FLOOR rounding method if I recall correctly. I guess that it's a requirement for makefiles. If you already experimented a system clock slew, you may understand me :-)
For full seconds, truncation will add an error of +/- 1 second, whereas rounding only adds +/- 0.5 seconds. This is what convinced me to use rounding instead of truncation.
What is truncation? Is it the ROUND_FLOOR (towards -inf) rounding method? Like math.floor(float).
Python int(float) uses ROUND_DOWN (towards zero) which is different than ROUND_FLOOR, but only different for negative numbers. int(-0.9) returns 0, whereas math.floor(-0.9) returns -1.
I guess that "rounding" means ROUND_HALF_EVEN here? The funny "Round to nearest with ties going to nearest even integer" rounding method. Like round(float).
Author: Guido van Rossum (gvanrossum) *
Date: 2016-03-01 21:48
Except for the case where you're closer than half a usec from the next value, IMO rounding makes no sense when suppressing digits. I most definitely would never want 9:59:59 to be rounded to 10:00 when suppressing seconds. If you really think there are use cases for that you could add a 'round=True' flag (as long as it defaults to False). That seems better than supporting rounding on datetime objects themselves. But I think you're just speculating.
Author: Guido van Rossum (gvanrossum) *
Date: 2016-03-01 21:50
IIUC truncation traditionally means "towards zero" -- that's why we have separate "floor" and "ceiling" operations meaning "towards [negative] infinity". Fortunately we shouldn't have to deal with negative values here so floor and truncate mean the same thing. Agreed that isoformat() should also truncate.
Author: STINNER Victor (vstinner) *
Date: 2016-03-01 21:52
Maybe a datetime.round() method along these lines will be a worthwhile addition?
Sorry, what is the use case of this method?
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-01 22:00
Personally, I don't rounding is that useful. My working assumption is that users will select say timespec='millisecond' only when they know that their time source produces datetime instances with millisecond precision and they don't want to kill more trees by printing redundant 0's.
MAL's objection this this line of arguments was that some time sources have odd resolution (he reported MS SQL's use of 333 ms) and a user may want to have a perfect round-tripping when using a sub-usec timespec and such an odd time source or destination.
Author: STINNER Victor (vstinner) *
Date: 2016-03-01 22:06
Personally, I don't rounding is that useful.
Nice, it looks like I agree with you on using ROUNDING_FLOOR :-)
I don't think that we should be prepared for theorical user requests, but rather focus on the concrete and well defined current existing user request: "Add timespec optional flag to datetime isoformat() to choose the precision".
Let's wait until users request a datetime.round() method to understand better concrete issues.
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-01 22:09
I feel odd trying to advocate a POV that I disagree with, so let me just quote MAL:
""" In practice you often don't know the resolution of the timing source. Nowadays, the reverse of what you said is usually true: the source resolution is higher than the precision you use to print it. ..
For full seconds, truncation will add an error of +/- 1 second, whereas rounding only adds +/- 0.5 seconds. This is what convinced me to use rounding instead of truncation. """
I somehow missed this argument when Marc-Andre made it, so I want to make sure that it is properly considered before we finalize this issue.
Author: Alessandro Cucci (acucci) *
Date: 2016-03-02 18:19
Meanwhile I made corrections after @belopolsky latest review
Author: Alexander Belopolsky (belopolsky) *
Date: 2016-03-02 18:25
Alessandro, thank you very much for your work and perseverance. I will do my best to commit this next weekend.
Author: Roundup Robot (python-dev)
Date: 2016-03-06 19:58
New changeset eb120f50df4a by Alexander Belopolsky in branch 'default': Closes #19475: Added timespec to the datetime.isoformat() method. https://hg.python.org/cpython/rev/eb120f50df4a
History
Date
User
Action
Args
2022-04-11 14:57:53
admin
set
github: 63674
2016-03-06 19:58:58
python-dev
set
status: open -> closed
nosy: + python-dev
messages: +
resolution: fixed
stage: commit review -> resolved
2016-03-02 18:25:44
belopolsky
set
messages: +
stage: patch review -> commit review
2016-03-02 18:24:42
SilentGhost
set
nosy: - SilentGhost
2016-03-02 18:19:23
acucci
set
files: + issue19475_v17.patch
messages: +
2016-03-01 22:09:32
belopolsky
set
messages: +
2016-03-01 22:06:38
vstinner
set
messages: +
2016-03-01 22:00:34
belopolsky
set
messages: +
2016-03-01 21:52:02
vstinner
set
messages: +
2016-03-01 21:50:55
gvanrossum
set
messages: +
2016-03-01 21:48:07
gvanrossum
set
messages: +
2016-03-01 21:47:08
vstinner
set
messages: +
2016-03-01 21:35:47
belopolsky
set
messages: +
2016-03-01 21:32:29
gvanrossum
set
messages: +
2016-03-01 21:16:26
vstinner
set
messages: +
2016-03-01 21:13:33
gvanrossum
set
messages: +
2016-03-01 20:22:38
belopolsky
set
messages: +
2016-03-01 20:11:22
belopolsky
set
messages: +
2016-03-01 19:20:57
belopolsky
set
stage: needs patch -> patch review
2016-03-01 18:05:27
acucci
set
messages: +
2016-02-25 18:49:58
belopolsky
set
messages: +
2016-02-25 18:29:22
gvanrossum
set
messages: +
2016-02-25 18:15:49
acucci
set
files: + issue19475_v16.patch
messages: +
2016-02-23 10:33:09
martin.panter
set
messages: +
2016-02-22 18:15:41
acucci
set
files: + issue19475_v15.patch
messages: +
2016-02-22 17:53:10
elixir
set
nosy: - elixir
2016-02-21 22:32:03
martin.panter
set
messages: +
2016-02-21 21:58:13
acucci
set
files: + issue19475_v14.patch
2016-02-21 18:20:29
acucci
set
files: + issue19475_v13.patch
messages: +
2016-01-18 02:59:56
gvanrossum
set
messages: +
2016-01-17 22:07:12
SilentGhost
set
messages: +
2016-01-17 21:33:15
belopolsky
set
messages: +
2016-01-17 21:23:54
belopolsky
set
messages: +
stage: commit review -> needs patch
2016-01-17 20:46:12
SilentGhost
set
messages: +
2016-01-17 20:33:35
belopolsky
set
messages: +
2016-01-17 20:25:37
belopolsky
set
assignee: belopolsky
stage: patch review -> commit review
2016-01-17 19:46:33
terry.reedy
set
messages: +
2016-01-17 18:52:20
gvanrossum
set
messages: +
2016-01-17 15:38:07
SilentGhost
set
messages: +
2016-01-15 15:09:21
r.david.murray
set
nosy: - r.david.murray
2016-01-15 12🔞09
acucci
set
messages: +
2016-01-05 15:30:32
acucci
set
messages: +
2016-01-01 12:50:31
acucci
set
files: + issue19475_v12.patch
messages: +
2016-01-01 10:39:52
martin.panter
set
messages: +
2015-12-29 18:44:40
acucci
set
files: + issue19475_v11.patch
messages: +
2015-12-29 18:16:39
acucci
set
messages: +
2015-12-29 16:08:47
berker.peksag
set
messages: +
stage: commit review -> patch review
2015-12-29 15:46:13
SilentGhost
set
stage: patch review -> commit review
2015-12-25 14:26:19
acucci
set
files: + issue19475_v10_datetime_time.patch
messages: +
2015-12-24 18:47:30
acucci
set
messages: +
2015-12-23 18🔞12
skip.montanaro
set
nosy: - skip.montanaro
2015-12-22 20:53:45
acucci
set
files: + issue19475_v9.patch
messages: +
2015-12-22 20:08:26
SilentGhost
set
messages: +
2015-12-22 19:25:41
acucci
set
files: + issue19475_v8.patch
messages: +
2015-12-22 18:44:34
acucci
set
files: + issue19475_v7.patch
2015-12-21 22:33:19
acucci
set
files: + issue19475_v6.patch
messages: +
2015-12-20 12:25:35
acucci
set
files: + issue19475_v5.patch
messages: +
2015-12-17 08:35:55
martin.panter
set
messages: +
2015-12-16 23:03:06
belopolsky
set
messages: +
2015-12-16 23:01:18
SilentGhost
set
messages: +
2015-12-16 22:50:35
acucci
set
messages: +
2015-12-16 22:41:13
martin.panter
set
messages: +
2015-12-16 21:47:21
acucci
set
files: + issue19475_v4.patch
messages: +
2015-12-16 20:11:15
SilentGhost
set
nosy: + SilentGhost
2015-12-16 07:35:50
acucci
set
messages: +
2015-12-15 21:07:50
martin.panter
set
messages: +
2015-12-15 20:39:12
martin.panter
set
nosy: + martin.panter
messages: +
2015-12-15 20:27:35
belopolsky
set
messages: +
2015-12-15 20:22:28
belopolsky
set
messages: +
2015-12-15 20:11:20
gvanrossum
set
messages: +
2015-12-15 20:01:28
belopolsky
set
messages: +
2015-12-15 19:58:25
belopolsky
set
messages: +
2015-12-15 19:56:32
acucci
set
messages: +
2015-12-15 19:33:00
belopolsky
set
messages: +
2015-12-15 17:34:46
gvanrossum
set
messages: +
2015-12-15 13:12:06
vstinner
set
nosy: + matrixise
2015-12-15 13:10:48
acucci
set
messages: +
2015-08-03 19:46:13
acucci
set
files: + issue19475_v3.patch
messages: +
2015-07-31 01:21:21
berker.peksag
set
nosy: + berker.peksag
messages: +
stage: needs patch -> patch review
2015-07-26 16:52:22
acucci
set
messages: +
2015-07-25 23:46:13
acucci
set
files: + issue19475_v2.patch
2015-07-25 23:11:19
acucci
set
nosy: + acucci
messages: +
2015-07-25 22:28:13
vstinner
set
messages: +
2015-07-25 14:12:16
acucci
set
files: + issue19475.patch
keywords: + patch
2015-07-24 09:53:38
vstinner
set
title: Add microsecond flag to datetime isoformat() -> Add timespec optional flag to datetime isoformat() to choose the precision
2015-07-24 09:53:22
vstinner
set
messages: +
2015-07-24 09:45:49
berker.peksag
set
stage: resolved -> needs patch
superseder: datetime: add ability to parse RFC 3339 dates and times ->
versions: + Python 3.6, - Python 3.5
2015-07-14 04:10:18
jerry.elmore
set
nosy: + jerry.elmore
2014-06-30 13:33:49
belopolsky
set
messages: +
2014-06-30 13:26:46
belopolsky
set
messages: +
versions: + Python 3.5, - Python 3.4
2014-04-23 06:23:00
cvrebert
set
nosy: + cvrebert
2013-11-06 18:22:31
skip.montanaro
set
messages: +
2013-11-06 17:20:54
belopolsky
set
messages: +
2013-11-06 17:05:25
lemburg
set
messages: +
2013-11-06 15:51:03
belopolsky
set
messages: +
2013-11-05 21:32:18
lemburg
set
nosy: + lemburg
messages: +
2013-11-05 21:24:47
belopolsky
set
messages: +
2013-11-05 20:31:16
vstinner
set
messages: +
2013-11-05 20:14:15
elixir
set
messages: +
2013-11-05 15:48:38
belopolsky
set
messages: +
2013-11-05 15:38:32
belopolsky
set
nosy: + belopolsky
2013-11-05 10:35:42
vstinner
set
nosy: + vstinner
2013-11-04 00:50:22
elixir
set
nosy: + elixir
2013-11-01 22:14:26
terry.reedy
set
nosy: + terry.reedy
title: Inconsistency between datetime's str()/isoformat() and its strptime() method -> Add microsecond flag to datetime isoformat()
messages: +
versions: - Python 2.7, Python 3.3
type: behavior -> enhancement
2013-11-01 18:55:55
r.david.murray
set
messages: +
2013-11-01 18:50:19
skip.montanaro
set
messages: +
2013-11-01 18:40:37
gvanrossum
set
status: closed -> open
nosy: + gvanrossum
messages: +
resolution: duplicate -> (no value)
2013-11-01 18:25:32
tim.peters
set
messages: +
2013-11-01 18:12:10
skip.montanaro
set
messages: +
2013-11-01 18:07:37
tim.peters
set
nosy: + tim.peters
messages: +
2013-11-01 18:01:09
r.david.murray
set
status: open -> closed
messages: +
2013-11-01 17:55:53
skip.montanaro
set
messages: +
2013-11-01 17:47:22
r.david.murray
set
nosy: + r.david.murray
messages: +
resolution: duplicate
superseder: datetime: add ability to parse RFC 3339 dates and times
stage: resolved
2013-11-01 17:08:38
ezio.melotti
set
nosy: + ezio.melotti
messages: +
2013-11-01 17:05:21
skip.montanaro
create