[Python-Dev] Adding socket timeout to urllib2 (original) (raw)

Guido van Rossum guido at python.org
Mon Mar 5 23:03:05 CET 2007

On 3/5/07, Facundo Batista <facundo at taniquetil.com.ar> wrote:

I studied which modifications I need to make into urllib2 to support a socket timeout.

- I'll modify urlopen for it to accept a sockettimeout parameter, default to None

I'd call it timeout. There can't really be much ambiguity can there?

- Request will also accept a sockettimeout parameter, default to None. It will keep it in a sockettimeout attribute, so it could be added/modified from outside.

Ditto for the name.

- OpenerDirector's 'open' method will accept sockettimeout, and build the Request object with it

So, when it gets into the Handlers, how it'll be used? I checked the Handlers, and here is a resume of their 'open' or 'open'-like methods: - AbstractHTTPHandler: receives a class, and uses its .request and .getresponse methods - HTTPHandler: just calls AbstractHTTPHandler.doopen with httplib.HTTPConnection - HTTPSHandler: just calls AbstractHTTPHandler.doopen with httplib.HTTPSConnection - UnknownHandler: raises an Error - FileHandler: opens a file in a local host - FTPHandler: uses sockets in ftplib.FTP().connect, .login, .cwd, and .ntransfercmd, all through urllib's ftwrapper object - CacheFTPHandler: instantiates ftpwrapper object; does not uses it directly, though. - GopherHandler: is deprecated... So, these are the necessary further modifications in urllib2 beyond those detailed at the beggining of the mail: - Modify AbstractHTTPHandler to pass the sockettimeout parameter to the received class' methods. - Modify FTPHandler to pass the sockettimeout to ftpwrapper class. - Modify CacheFTPHandler the same way that FTPHandler Beyond that, I'll also need to: - Modify urllib.ftpwrapper to accept a sockettimeout parameter and pass it to ftplib.FTP object. - Add support to sockettimeout to ftplib's FTP object - Add support to sockettimeout to httplib's HTTPConnection and HTTPSConnection objects So, I think that in the name of clarity, I shall first modify ftplib and httplib, add sockettimeout where is necessary, and then add support to that in the more general urllib2. What do you think? Comments are very appreciated.

I say your next step should be to submit a patch, perhaps incorporating the work that Skip has already done.

You could also reduce the amount of work to be done initially by only handling http. At this point I expect that'll cover 99% of all uses.

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list