| msg268205 - (view) |
Author: Christian Heimes (christian.heimes) *  |
Date: 2016-06-11 10:07 |
| In #26839 os.urandom() was made non-blocking and non-exception-raising on Linux. As a result os.urandom() is no longer a CSPRNG under some conditions as it can and will return predictable random values without any sort of warning or error flag. These conditions are (including but not limited to): * early boot state * virtualization without host-passthrough, e.g. virtio-rng * embedded devices without hardware RNG or RTC, e.g. raspberry pi Please ensure that the documentation properly warns users about these edge cases. We might also want to add that os.urandom() can block on other platforms, e.g. FreeBSD and OpenBSD. |
|
|
| msg268207 - (view) |
Author: Steven D'Aprano (steven.daprano) * |
Date: 2016-06-11 10:33 |
| Relevant: issue #27293 (I've taken the liberty of subscribing those on this issues nosy list to the new issue, I hope that's okay) |
|
|
| msg268238 - (view) |
Author: Larry Hastings (larry) * |
Date: 2016-06-11 18:07 |
| I don't think this is necessary, as the documentation for os.urandom() is already pretty good. Here's the relevant bit: This function returns random bytes from an OS-specific randomness source. The returned data should be unpredictable enough for cryptographic applications, though its exact quality depends on the OS implementation. On a Unix-like system this will query /dev/urandom, and on Windows it will use CryptGenRandom(). ISTM that the Python documentation doesn't generally indulge in warning users about specific shortcomings of particular platforms; if it did it would be littered with such warnings. Personally I'd approve of making the existing statements a little more forceful, like pulling it out into a red "warning" box and making it explicit that os.urandom() isn't any more sophisticated than the platform-specific technologies it uses. But that's as far as I'd go. I wouldn't add all the specifics you suggest. Technically I think this actually is my call, as I'm the "platform expert" for the posix module: https://docs.python.org/devguide/experts.html But really I think it's the call of the "Documentation Expert" for the relevant releases. This is a stylistic concern--should the Python docs delve into these sorts of details?--and that's really the domain of the DE. Georg Brandl is the DE for all currently-supported versions of Python. (Well, 2.7 has no official DE, but I think Georg is de facto DE for that release too.) I've nosied him here; hopefully he can tell us the standard Python doc aesthetic when it comes to these sorts of concerns. By the way, the Raspberry PI does have hardware RNG: http://scruss.com/blog/2013/06/07/well-that-was-unexpected-the-raspberry-pis-hardware-random-number-generator/ It required loading an extra driver, at least as of 2014. I concede I don't know what current crypto best-practices are on the PI. That's one good reason why I think the Python documentation doesn't indulge in these laundry lists of platform failings--such information has a tendency to become out-of-date without anyone noticing. |
|
|
| msg268239 - (view) |
Author: Larry Hastings (larry) * |
Date: 2016-06-11 18:11 |
| Oh, and, for 3.6 I would definitely support adding a mention of "Instead of using this function directly, we recommend you use the token_bytes() function in the secrets module", blah blah best practices etc. That goes for os.getrandom() too, if we add it to 3.6 (which I rather suspect we will). |
|
|
| msg268242 - (view) |
Author: Donald Stufft (dstufft) * |
Date: 2016-06-11 18:34 |
| I agree that we should add a warning to these, it's easy to see how someone might read the summary of the function "Return a string of n random bytes suitable for cryptographic use." and skip over the nuance in the rest of the body of the function. Adding a red box to ensure that they know that on popular platforms os.urandom is not going to always return bytes that are suitable for cryptographic use is pretty important in my opinion. The current wording makes it sound like it's something you only need to worry about on "weird" platforms, not on one of the (if not the) most popular platforms for running Python on. |
|
|
| msg268244 - (view) |
Author: Larry Hastings (larry) * |
Date: 2016-06-11 18:36 |
| I would suggest weakening the one-line summary. Currently the first line reads: Return a string of n random bytes suitable for cryptographic use. I'd support adding some "weasel words" to this, e.g.: Return a string of n random bytes that should be suitable for cryptographic use. |
|
|
| msg268316 - (view) |
Author: Larry Hastings (larry) * |
Date: 2016-06-12 02:58 |
| This is not a release blocker. |
|
|
| msg268649 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-06-16 08:11 |
| As far as this bug goes, 3.5 is not very different from 2.7 |
|
|
| msg268656 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-06-16 11:01 |
| Here is a possible patch for 3.5+ based on my modest understanding of the concerns about insecure results and blocking. I hope that my wording is clear, couldn’t be confused with Linux’s /dev/random blocking and running out of fresh entropy, etc. I also tried to make it clearer what APIs are used in what circumstances. It is not just Linux: we also call getrandom() on Solaris, because its getentropy() is not good enough. |
|
|
| msg268662 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-06-16 12:54 |
| Strange, I don't see the [Review] button. .. versionchanged:: 3.5.2 - On Linux, if ``getrandom()`` blocks (the urandom entropy pool is not + If ``getrandom()`` blocks (the urandom entropy pool is not initialized yet), fall back on reading ``/dev/urandom``. Please keep "On Linux", getrandom() is also used on Solaris, and my change is really restricted to Linux. |
|
|
| msg268663 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-06-16 13:08 |
| Rebased so Rietveld can work with it, earlier version was my fault. As far as I can see (looking at Python/random.c and configure.ac), the Solaris version should also use GRND_NONBLOCK: #ifdef MS_WINDOWS #elif defined(HAVE_GETENTROPY) && !defined(sun) #else #if defined(HAVE_GETRANDOM) | |
defined(HAVE_GETRANDOM_SYSCALL) const int flags = GRND_NONBLOCK; #ifdef HAVE_GETRANDOM n = getrandom(dest, n, flags); #else n = syscall(SYS_getrandom, dest, n, flags); #endif Apart from using a C function call versus syscall(), I don’t see there is much difference between the Solaris and Linux cases. Correct me if I’m wrong though. |
|
| msg268666 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-06-16 13:14 |
| > As far as I can see (looking at Python/random.c and configure.ac), the Solaris version should also use GRND_NONBLOCK: Oh, you're right: I didn't notice that GRND_NONBLOCK was also used on Solaris. The change is not deliberate, but it is good to do that :-) Solaris getrandom() is documented to fail with EAGAIN if "No entropy is available and GRND_NONBLOCK is set." https://docs.oracle.com/cd/E53394_01/html/E54765/getrandom-2.html The question is more if reading from /dev/urandom block in this case. If we don't know, I would prefer to keep the "On Linux" prefix in the doc, and don't say anything about Solaris. I'm able to check the behaviour of Solaris. You should contact the developers who get access to Solaris, you can meet them in the previous random issues specific to Solaris: issue #25003 and issue #26735. |
|
|
| msg268752 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-06-18 01:27 |
| Restored “On Linux” for the changed in 3.5.2 notice. I do think it is better to be general and future-proof, but that is a separate, less important issue to the main purpose of the patch. (I don’t know if Solaris’s version can block or not.) |
|
|
| msg274710 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2016-09-07 02:34 |
| PEP 524 has been implemented for 3.6b1 in #27776, so os.urandom() itself will now do the right thing for cryptographic use cases on Linux. Accordingly, I've removed Python 3.6 from the affected versions for this issue. I've also flagged this as purely a documentation issue, since no behavioural changes are currently proposed for older versions. |
|
|
| msg274723 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-07 03:01 |
| I dislike urandom-doc.v2.patch. There is no need to worry all users. Just be explicit and explain that the issue is specific to Linux and explain when it occurs: before the entropy pool is filled *and* if the system was not able yet to write enough entropy on disk, this thing: https://www.python.org/dev/peps/pep-0524/#load-entropy-from-disk-at-boot |
|
|
| msg275260 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-09-09 05:13 |
| Do you want to do an alternative patch Victor? Or point out all the specific bits of my patch you don’t like? I haven’t really been keeping up to date with the getrandom() changes. Though I imagine even Python 3.6’s os.urandom() will still fall back to /dev/urandom (with potential entropy problem) on older Linux versions. Is the consensus that we want to warn about insecure results on Linux >= 3.17, but don’t want to warn about older Linux versions? |
|
|
| msg275269 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-09 07:01 |
| > Though I imagine even Python 3.6’s os.urandom() will still fall back to /dev/urandom (with potential entropy problem) on older Linux versions. Right, but you should be more specific in the doc. os.urandom() is unsecure if and only if: * getrandom() is not available (ex: Linux < 3.17, or Python built without getrandom() for different reasons) or don't work (ex: blocked by a stupid SECCOMP policy, ex: issue #27955) * and /dev/urandom is not initialized yet which means that: - the currently running Python runs very early during the system initialization, - the system has no good source of entropy and slow entropy sources, - no entropy was stored on the disk on a previous boot. What I don't want: a doc warning that "oh hey, os.urandom() is unsafe, don't use it for anything serious". os.urandom() is always secure on all platforms except Linux. On Linux, it's secure is almost all cases, except on a very few very tiny corner cases. I proposed something like: "On Linux, os.urandom() can return weak entropy when /dev/urandom is used internally and the system urandom entropy pool is not initialized yet." I'm not even sure about "weak entropy" because in most cases, /dev/urandom is already partially initialized with good entropy, but just not enough to consider that it's fully initialized. Linux uses many entropy sources but don't trust them, so don't consider that these input data counts for the entropy counter. |
|
|
| msg275600 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2016-09-10 08:27 |
| On modern Intel chips, one of the entropy sources is the CPU itself, and so this problem is mostly theoretical on such systems unless you're worried about the quality of Intel's entropy generation (in which case you're well and truly into sovereign espionage and advanced persistent threat territory, and really shouldn't be blindly trusting operating systems and programming language runtimes to already be doing the right thing). So the main ways to get bad entropy from /dev/urandom on Linux are: - a misconfigured VM that has been cut off from all decent entropy sources, and doesn't have any persistent storage attached - an embedded non-Intel chip that doesn't have any decent entropy sources or persistent storage attached If you're not doing either of those things, you're probably fine. If you're worried that you might be, try running Python 3.6 and calling os.getrandom() explicitly to see what happens. |
|
|
| msg277073 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-20 21:28 |
| > Please ensure that the documentation properly warns users about these edge cases. I disagree. I don't think that the Python documentation is the right place to document the security level of system urandom. It's just a mess, there are so many corner cases and it's very hard to provide a clear explanation for end users. I suggest to keep the positive "suitable for cryptographic use". If you change this sentence, I only expect that users will use something WORSE. For example "os.urandom is not secure! we must use ssl.RAND_bytes!". No. Don't do that, ssl.RAND_bytes() has its own set of issues, like two processes with the same pid producing the same random sequence... (old known issue, very hard to fix) Python cannot workaround OS limitations, we can only do our best to use the most secure source of entropy. That's why Python 3.5 now uses getrandom() on Linux. That's why Python 3.6 now calls getrandom() in blocking mode. |
|
|
| msg277115 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2016-09-21 09:15 |
| With the 3.6 os.urandom() implementation doing the right thing consistently cross-platform, our guidance for folks that care about the quality of the CSPRNG they use should be that they either upgrade to that version, or else ensure that the kernel CSPRNG is properly seeded before they run Python. That is, I think the tone we're aiming for in the older docs now should be "You're using an older Python version, so if this problem description worries you, you need to either upgrade or else take the necessary steps to satisfy yourself that your host system's CSPRNG is properly configured", rather than the more passive "os.urandom() isn't necessarily secure" (with minimal guidance on what to do about it) that we've previously adopted. |
|
|
| msg404487 - (view) |
Author: Christian Heimes (christian.heimes) * |
Date: 2021-10-20 16:15 |
| 2.7 and 3.5 are no longer supported |
|
|