[Python-Dev] PEP 564: Add new time functions with nanosecond resolution (original) (raw)

Victor Stinner victor.stinner at gmail.com
Mon Oct 16 06:42:30 EDT 2017

Hi,

While discussions on this PEP are not over on python-ideas, I proposed this PEP directly on python-dev since I consider that my PEP already summarizes current and past proposed alternatives.

python-ideas threads:

The PEP 564 will be shortly online at: https://www.python.org/dev/peps/pep-0564/

Victor

PEP: 564 Title: Add new time functions with nanosecond resolution Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Victor Stinner <victor.stinner at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-October-2017 Python-Version: 3.7

Abstract

Add five new functions to the time module: time_ns(), perf_counter_ns(), monotonic_ns(), clock_gettime_ns() and clock_settime_ns(). They are similar to the function without the _ns suffix, but have nanosecond resolution: use a number of nanoseconds as a Python int.

The best time.time_ns() resolution measured in Python is 3 times better then time.time() resolution on Linux and Windows.

Rationale

Float type limited to 104 days

The clocks resolution of desktop and latop computers is getting closer to nanosecond resolution. More and more clocks have a frequency in MHz, up to GHz for the CPU TSC clock.

The Python time.time() function returns the current time as a floatting point number which is usually a 64-bit binary floatting number (in the IEEE 754 format).

The problem is that the float type starts to lose nanoseconds after 104 days. Conversion from nanoseconds (int) to seconds (float) and then back to nanoseconds (int) to check if conversions lose precision::

# no precision loss
>>> x = 2 ** 52 + 1; int(float(x * 1e-9) * 1e9) - x
0
# precision loss! (1 nanosecond)
>>> x = 2 ** 53 + 1; int(float(x * 1e-9) * 1e9) - x
-1
>>> print(datetime.timedelta(seconds=2 ** 53 / 1e9))
104 days, 5:59:59.254741

time.time() returns seconds elapsed since the UNIX epoch: January 1st, 1970. This function loses precision since May 1970 (47 years ago)::

>>> import datetime
>>> unix_epoch = datetime.datetime(1970, 1, 1)
>>> print(unix_epoch + datetime.timedelta(seconds=2**53 / 1e9))
1970-04-15 05:59:59.254741

Previous rejected PEP

Five years ago, the PEP 410 proposed a large and complex change in all Python functions returning time to support nanosecond resolution using the decimal.Decimal type.

The PEP was rejected for different reasons:

CPython enhancements of the last 5 years

Since the PEP 410 was rejected:

Existing Python APIs using nanoseconds as int

The os.stat_result structure has 3 fields for timestamps as nanoseconds (int): st_atime_ns, st_ctime_ns and st_mtime_ns.

The ns parameter of the os.utime() function accepts a (atime_ns: int, mtime_ns: int) tuple: nanoseconds.

Changes

New functions

This PEP adds five new functions to the time module:

These functions are similar to the version without the _ns suffix, but use nanoseconds as Python int.

For example, time.monotonic_ns() == int(time.monotonic() * 1e9) if monotonic() value is small enough to not lose precision.

Unchanged functions

This PEP only proposed to add new functions getting or setting clocks with nanosecond resolution. Clocks are likely to lose precision, especially when their reference is the UNIX epoch.

Python has other functions handling time (get time, timeout, etc.), but no nanosecond variant is proposed for them since they are less likely to lose precision.

Example of unchanged functions:

Since the time.clock() function was deprecated in Python 3.3, no time.clock_ns() is added.

Alternatives and discussion

Sub-nanosecond resolution

time.time_ns() API is not "future-proof": if clocks resolutions increase, new Python functions may be needed.

In practive, the resolution of 1 nanosecond is currently enough for all structures used by all operating systems functions.

Hardware clock with a resolution better than 1 nanosecond already exists. For example, the frequency of a CPU TSC clock is the CPU base frequency: the resolution is around 0.3 ns for a CPU running at 3 GHz. Users who have access to such hardware and really need sub-nanosecond resolution can easyly extend Python for their needs. Such rare use case don't justify to design the Python standard library to support sub-nanosecond resolution.

For the CPython implementation, nanosecond resolution is convenient: the standard and well supported int64_t type can be used to store time. It supports a time delta between -292 years and 292 years. Using the UNIX epoch as reference, this type supports time since year 1677 to year 2262::

>>> 1970 - 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)
1677.728976954687
>>> 1970 + 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)
2262.271023045313

Different types

It was proposed to modify time.time() to use float type with better precision. The PEP 410 proposed to use decimal.Decimal, but it was rejected. Apart decimal.Decimal, no portable float type with better precision is currently available in Python. Changing the builtin Python float type is out of the scope of this PEP.

Other ideas of new types were proposed to support larger or arbitrary precision: fractions, structures or 2-tuple using integers, fixed-precision floating point number, etc.

See also the PEP 410 for a previous long discussion on other types.

Adding a new type requires more effort to support it, than reusing int. The standard library, third party code and applications would have to be modified to support it.

The Python int type is well known, well supported, ease to manipulate, and supports all arithmetic operations like: dt = t2 - t1.

Moreover, using nanoseconds as integer is not new in Python, it's already used for os.stat_result and os.utime(ns=(atime_ns, mtime_ns)).

.. note:: If the Python float type becomes larger (ex: decimal128 or float128), the time.time() precision will increase as well.

Different API

The time.time(ns=False) API was proposed to avoid adding new functions. It's an uncommon (and bad?) programming practice in Python to change the result type depending on a parameter.

Different options were proposed to allow the user to choose the time resolution. If each Python module uses a different resolution, it can become difficult to handle different resolutions, instead of just seconds (time.time() returning float) and nanoseconds (time.time_ns() returning int). Moreover, as written above, there is no need for resolution better than 1 nanosecond in practive in the Python standard library.

Annex: Clocks Resolution in Python

Script ot measure the smallest difference between two time.time() and time.time_ns() reads ignoring differences of zero::

import math
import time

LOOPS = 10 ** 6

print("time.time_ns(): %s" % time.time_ns())
print("time.time(): %s" % time.time())

min_dt = [abs(time.time_ns() - time.time_ns())
          for _ in range(LOOPS)]
min_dt = min(filter(bool, min_dt))
print("min time_ns() delta: %s ns" % min_dt)

min_dt = [abs(time.time() - time.time())
          for _ in range(LOOPS)]
min_dt = min(filter(bool, min_dt))
print("min time() delta: %s ns" % math.ceil(min_dt * 1e9))

Results of time(), perf_counter() and monotonic().

Linux (kernel 4.12 on Fedora 26):

Windows 8.1:

The difference on time.time() is significant: 84 ns (2.8x better) vs 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows. The difference (presion loss) will be larger next years since every day adds 864,00,000,000,000 nanoseconds to the system clock.

The difference on time.perf_counter() and time.monotonic clock() is not visible in this quick script since the script runs less than 1 minute, and the uptime of the computer used to run the script was smaller than 1 week. A significant difference should be seen with an uptime of 104 days or greater.

.. note:: Internally, Python starts monotonic() and perf_counter() clocks at zero on some platforms which indirectly reduce the precision loss.

Copyright

This document has been placed in the public domain.

More information about the Python-Dev mailing list