When using urllib2 to download through a HTTP proxy, which requires authorisation, a broken HTTP request is sent. The initial request might work but subsequent requests send using the same socket definitely fail. Problem occurs on Fedora Core 3 with python 2.3.4. Buggy code still exists in Python Library in 2.4.1. Found the problem using yum to download files via my companies Microsoft ISA web proxy. The proxy requires authorisation. I set the HTTP_PROXY environment variable to define the proxy like this: export HTTP_PROXY=http://username:password@proxy. example.com:8080/ Analysis from my yum bugzilla report http://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=441 , follows: Location is: File: urllib2.py Class: ProxyHandler Function: proxy_open() The basic proxy authorisation string is created using base64.encodestring() and passed to add_header() method of a Request object. However base64.encodestring() specifically adds a trailing '\n' but when the headers are sent over the socket each is followed by '\r\n'. The server sees this double new line as the end of the HTTP request and the rest of the HTTP headers as a second invalid request. The broken request looks like this: GET ... Host: ... Accept-Encoding: identity Proxy-authorization: Basic xxxxxxxxxxxxxxxx <-- Blank line which shouldn't be there User-agent: urlgrabber/2.9.2 <-- Blank line ending HTTP request The fix is just to remove the '\n' which base64.encodestring() added before calling add_header(). Just use string method strip() as is done in the only other location base64.encodestring() is used in file urllib2.py.
