| msg125316 - (view) |
Author: Michal Čihař (nijel) * |
Date: 2011-01-04 13:42 |
| Currently if /dev/urandom does not provide any data, unradom() call is just stuck infinitely waiting for data. I actually faced this issue when /dev/urandom was empty regular file (due to bug in pbuilder, but I don't think it matters how it did happen) and urandom() call just hang. I think it would be much saner in such case to throw an error and let user know that something it wrong than waiting infinitely. |
|
|
| msg125318 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-01-04 13:49 |
| This would change semantics of the library call, though. I see two possible solutions: - do nothing and let users tackle the issue if necessary (e.g. by calling open() themselves and using select() on the resulting fd) - add an optional "blocking" parameter to os.urandom() that, if False, would return None when no data is available |
|
|
| msg125320 - (view) |
Author: Michal Čihař (nijel) * |
Date: 2011-01-04 13:59 |
| Well in this particular case (/dev/urandom is regular file), it might make sense to simply raise NotImplementedError |
|
|
| msg125321 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-01-04 14:01 |
| > Well in this particular case (/dev/urandom is regular file), it might > make sense to simply raise NotImplementedError IMO it would be quite fragile to try to detect regular files vs special files. I suppose you noticed the issue quite quickly anyway, since you had the blocking issue. |
|
|
| msg125322 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2011-01-04 14:06 |
| Is it really necessary to do something about this? /dev/urandom being a regular file is clearly a bug in your system configuration, and I don't want to know what all the other programs will do that rely on it... |
|
|
| msg125323 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2011-01-04 14:10 |
| From the documentation: "This function returns random bytes from an OS-specific randomness source." In your case, this problem shows up because of an OS misconfiguration : in that case, the behaviour is undefined (not much Python can do about it). Note that since /dev/urandom is used, with a properly configured system, this should never block (contrarily to /dev/random which might block until enough entropy has been gathered). |
|
|
| msg125324 - (view) |
Author: Michal Čihař (nijel) * |
Date: 2011-01-04 14:11 |
| Yes, it was blocking, but deep in some program (which was actually called by dpkg postinst script), so it took some time to figure out. I don't think it's that fragile to figure out whether it is regular file using os.path.isfile. Indeed it was bug in a system, but actually this was only thing which got stuck because of it. |
|
|
| msg125325 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-01-04 14:13 |
| > Note that since /dev/urandom is used, with a properly configured > system, this should never block Ok, suggesting closing then. |
|
|
| msg125362 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2011-01-04 21:08 |
| I wonder why reading from /dev/urandom has a loop in the first place, though - isn't it guaranteed that you can read as many bytes as you want in one go? This goes back to #934711, and apparently, even the original patch had the loop - for reasons that got never questioned. |
|
|
| msg125365 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2011-01-04 21:40 |
| Agreed that the original issue is invalid. So either the title should be changed so it can be used to address Martin's question, or it should be closed. |
|
|
| msg125368 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2011-01-04 21:54 |
| Closing it. It seems that people feel more safe when urandom loops until it has enough data (see http://stackoverflow.com/questions/4598507/dev-urandom-maximum-size/4598534#4598534). I might still pursue this idea, but in a different issue. |
|
|
| msg125369 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2011-01-04 21:58 |
| > Martin v. Löwis <martin@v.loewis.de> added the comment: > > I wonder why reading from /dev/urandom has a loop in the first place, though - isn't it guaranteed that you can read as many bytes as you want in one go? This goes back to #934711, and apparently, even the original patch had the loop - for reasons that got never questioned. > I found surprising that a read from /dev/urandom would be uninterruptible, so I digged a little, and found this mail from 1998: [patch] fix for urandom read(2) not interruptible http://marc.info/?l=bugtraq&m=91495921611500&w=2 "It's a bug in random.c that doesn' t check for signal pending inside the read(2) code, so you have no chance to kill the process via signals until the read(2) syscall is finished, and it could take a lot of time before return, if the buffer given to the read syscall is very big..." I've had a quick look at the source code, and indeed, read(2) from /dev/urandom can now be interrupted by a signal, so looping seems to be justified. > ---------- > nosy: +loewis > status: pending -> open > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue10824> > _______________________________________ > |
|
|
| msg125376 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2011-01-04 22:37 |
| > "It's a bug in random.c that doesn' t check for signal pending inside the > read(2) code, so you have no chance to kill the process via signals until > the read(2) syscall is finished, and it could take a lot of time before > return, if the buffer given to the read syscall is very big..." > > I've had a quick look at the source code, and indeed, read(2) from > /dev/urandom can now be interrupted by a signal, so looping seems to > be justified. No: if read(2) is interrupted, no data is returned, and exception is raised. So it won't loop in that case, but raise the exception out of urandom also (which is the right thing to do). |
|
|
| msg125519 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2011-01-06 08:18 |
| > Martin v. Löwis <martin@v.loewis.de> added the comment: > >> "It's a bug in random.c that doesn' t check for signal pending inside the >> read(2) code, so you have no chance to kill the process via signals until >> the read(2) syscall is finished, and it could take a lot of time before >> return, if the buffer given to the read syscall is very big..." >> >> I've had a quick look at the source code, and indeed, read(2) from >> /dev/urandom can now be interrupted by a signal, so looping seems to >> be justified. > > No: if read(2) is interrupted, no data is returned, and exception is > raised. So it won't loop in that case, but raise the exception out of > urandom also (which is the right thing to do). > (Sorry for being a little off-topic, but since there's not dedicated thread) Try with this: dd if=/dev/urandom of=/dev/null bs=100M count=1 Then, in another terminal: pkill -USR1 -xn dd You'll see that read returns less that 100M bytes when interrupted. You can also try with the following python code: --- import os d = os.open('/dev/urandom', os.O_RDONLY) data = os.read(d, 1 << 28) os.close(d) print('read %d bytes' % len(data)) --- and in another terminal pkill -STOP -xn python then pkill -CONT -xn python Same thing, read returns less bytes than requested. Anyway, since /dev/urandom is not part of any standard (AFAIK), it's probably better to use the common idiom while len(data) < expected: read(expected - len(data)) So we're sure it won't break under some systems/conditions. Cheers > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue10824> > _______________________________________ > |
|
|
| msg125601 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2011-01-06 22:53 |
| > You'll see that read returns less that 100M bytes when interrupted. I see. > while len(data) < expected: > read(expected - len(data)) > > So we're sure it won't break under some systems/conditions. I think this is not quite the idiom we should use if we want to deal with signals: if read() returns an empty string, we have hit end-of-file. If there is a signal before anything is read, we should catch the exception and continue reading (which the loop doesn't do, either). OTOH, if the signal was due to a user interrupt, we should raise an exception, anyway, IMO - even if we've read some data already. |
|
|