| msg22224 - (view) |
Author: David Abrahams (david_abrahams) |
Date: 2004-08-26 13:58 |
| The following script appears to be unreliable on all versions of Python we can find. The file being downloaded is approximately 34 MB. Browsers such as IE and Mozilla have no problem downloading the whole thing. ---- import urllib import os os.chdir('/tmp') urllib.urlretrieve ('http://cvs.sourceforge.net/cvstarballs/boost- cvsroot.tar.bz2', 'boost-cvsroot.tar.bz2') |
|
|
| msg22225 - (view) |
Author: Raymond Hettinger (rhettinger) *  |
Date: 2004-08-26 17:09 |
| Logged In: YES user_id=80475 Confirmed. On Py2.4 (current CVS), I got 12.7 Mb before the connection closed. |
|
|
| msg22226 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2004-08-26 18:52 |
| Logged In: YES user_id=31435 Hmm. I don't know anything about this, but thought I'd just try it. Didn't chdir(), did add a reporthook: def hook(*args): print args WinXP Pro SP1, current CVS Python, cable modem over a wireless router. Output looked like this: (0, 8192, 34520156) (1, 8192, 34520156) (2, 8192, 34520156) ... (4213, 8192, 34520156) (4214, 8192, 34520156) (4215, 8192, 34520156) Had the whole file when it ended: > wc boost-cvsroot.tar.bz2 125368 765656 34520156 boost-cvsroot.tar.bz2 *Maybe* adding the reporthook changed timing in some crucial way. Don't know. |
|
|
| msg22227 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-08-26 20:04 |
| Logged In: YES user_id=80475 Followed the same procedure (no chdir, add a hook) but bombed out at 9.1Mb: . . . (1117, 8192, 34520156) ('boost-cvsroot.tar.bz2', <httplib.HTTPMessage instance at 0x00B1E4B8>) |
|
|
| msg22228 - (view) |
Author: Irmen de Jong (irmen)  |
Date: 2004-11-07 19:47 |
| Logged In: YES user_id=129426 Confirmed here (mandrakelinux 10.0, python 2.4b2) However, I doubt it is a problem in urllib.urlretrieve, because I tried downloading the file with wget, and got the following: [irmen@isengard tmp]$ wget -S http://cvs.sourceforge.net/cvstarballs/boost-cvsroot.tar.bz2 --20:38:11-- http://cvs.sourceforge.net/cvstarballs/boost-cvsroot.tar.bz2 => `boost-cvsroot.tar.bz2.1' Resolving cvs.sourceforge.net... 66.35.250.207 Connecting to cvs.sourceforge.net[66.35.250.207]:80... connected. HTTP request sent, awaiting response... 1 HTTP/1.1 200 OK 2 Date: Sun, 07 Nov 2004 19:38:15 GMT 3 Server: Apache/2.0.40 (Red Hat Linux) 4 Last-Modified: Sat, 06 Nov 2004 15:11:39 GMT 5 ETag: "b63d5b-25c3808-687d80c0" 6 Accept-Ranges: bytes 7 Content-Length: 39598088 8 Content-Type: application/x-bzip2 9 Connection: close 31% [=======================> ] 12,665,616 60.78K/s ETA 03:55 20:40:07 (111.60 KB/s) - Connection closed at byte 12665616. Retrying. --20:40:08-- http://cvs.sourceforge.net/cvstarballs/boost-cvsroot.tar.bz2 (try: 2) => `boost-cvsroot.tar.bz2.1' Connecting to cvs.sourceforge.net[66.35.250.207]:80... connected. HTTP request sent, awaiting response... ....... so the remote server just closed the connection halfway trough! I suspect that a succesful download is sheer luck. Also, the download loop in urllib looks fine to me. It only stops when the read() returns an empty result, and that means EOF. |
|
|
| msg22229 - (view) |
Author: Irmen de Jong (irmen) |
Date: 2004-11-07 20:17 |
| Logged In: YES user_id=129426 a patch is at 1062060 (raises IOError when download is incomplete) |
|
|
| msg22230 - (view) |
Author: Irmen de Jong (irmen) |
Date: 2004-12-24 14:30 |
| Logged In: YES user_id=129426 Suggested addition to the doc of urllib (liburllib.tex, if I'm not mistaken): """ urlretrieve will raise IOError when it detects that the amount of data available was less than the expected amount (which is the size reported by a Content-Length header). This can occur, for example, when the download is interrupted. The Content-Length is treated as a lower bound (just like tools such as wget and Ffirefox appear to do): if there's more data to read, urlretrieve reads more data, but if less data is available, it raises IOError. If no Content-Length header was supplied, urlretrieve can not check the size of the data it has downloaded, and just returns it. In this case you just have to assume that the download was successful. """ |
|
|
| msg22231 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2005-08-24 18:49 |
| Logged In: YES user_id=1188172 Fixed wrt patch #1062060. |
|
|