Issue 1519816: urllib2 proxy does not work in 2.4.3 (original) (raw)

Issue1519816

Created on 2006-07-10 08:29 by mniklas, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (13)
msg29117 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-10 08:29
My python app had to retrieve some web pages and while our network environment requires proxy it uses urllib2 opener (source is in attachment). It worked very well on older Python interpreters: ActivePython 2.4.2 Build 248 (ActiveState Corp.) based on Python 2.4.2 (#67, Oct 30 2005, 16:11:18) [MSC v.1310 32 bit (Intel)] on win32 It works on linux with 2.3 and 2.4.1: Python 2.4.1 (#2, May 5 2005, 11:32:06) [GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2 But it does not work with newest 2.4.3 on Linux: Python 2.4.3 (#1, Jul 10 2006, 09:57:52) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Desired result: isof-mark:~# python2.3 proxy_bug.py trying http://www.python.org ... OK. We have reply from http://www.python.org. Size: 13757 [b] <me ... Yorkshire">design by pollenation Copyright Š 1990-2006, Python Software Foundation Legal Statements isof-mark:~# /usr/local/bin/python proxy_bug.py trying http://www.python.org ... Traceback (most recent call last): File "proxy_bug.py", line 37, in ? get_page() File "proxy_bug.py", line 27, in get_page f = urllib2.urlopen(request) File "/usr/local/lib/python2.4/urllib2.py", line 130, in urlopen return _opener.open(url, data) File "/usr/local/lib/python2.4/urllib2.py", line 364, in open response = meth(req, response) File "/usr/local/lib/python2.4/urllib2.py", line 471, in http_response response = self.parent.error( File "/usr/local/lib/python2.4/urllib2.py", line 402, in error return self._call_chain(*args) File "/usr/local/lib/python2.4/urllib2.py", line 337, in _call_chain result = func(*args) File "/usr/local/lib/python2.4/urllib2.py", line 480, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required I have raported it on ActiveState bug list (http:// bugs.activestate.com/show_bug.cgi?id=47018) while I first spot this bug on their destribution but it seems that bug is in others distributions too. Regards, Michal Niklas
msg29118 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-10 08:41
Logged In: YES user_id=226518 Cannot add attachment via upload so I put it here: #!/usr/bin/python # -*- coding: cp1250 -*- import urllib import urllib2 def get_page(): url = 'http://www.python.org' print "trying %s ..." % (url) # Setup proxy & authentication proxy = "poczta.heuthes:8080" usr1 = "USER" pass1 = "PASSWD" proxy_handler = urllib2.ProxyHandler({"http" : "http:/ /" + proxy}) pass_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() pass_mgr.add_password(None, "http://" + proxy, usr1, pass1) pass_mgr.add_password(None, proxy, usr1, pass1) auth_handler = urllib2.HTTPBasicAuthHandler(pass_mgr) proxy_auth_handler = urllib2.ProxyBasicAuthHandler(pass_mgr) # Now build a new URL opener and install it opener = urllib2.build_opener(proxy_handler, proxy_auth_handler, auth_handler, urllib2.HTTPHandler) urllib2.install_opener(opener) request = urllib2.Request(url) f = urllib2.urlopen(request) data = f.read() print "OK. We have reply from %s.\nSize: %d [b]" % (url, len(data)) if len(data) < 400: print data else: print data[:200] print "..." print data[-200:] get_page()
msg29119 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2006-07-10 12:49
Logged In: YES user_id=849994 Can you please try with 2.5b1? A lot of urllib2 related bugs have been fixed before this release.
msg29120 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-11 06:27
Logged In: YES user_id=226518 Tried it with 2.5 beta 1 and it is not better :( c:\tools\pyscripts\scripts>c:\python25\python2.5 Python 2.5b1 (r25b1:47027, Jun 20 2006, 09:31:33) [MSC v.1310 32 bit (Intel)] on win32 c:\tools\pyscripts\scripts>c:\python25\python2.5 proxy_bug.py trying http://www.python.org ... Traceback (most recent call last): File "proxy_bug.py", line 37, in get_page() File "proxy_bug.py", line 27, in get_page f = urllib2.urlopen(request) File "c:\python25\lib\urllib2.py", line 121, in urlopen return _opener.open(url, data) File "c:\python25\lib\urllib2.py", line 380, in open response = meth(req, response) File "c:\python25\lib\urllib2.py", line 491, in http_response 'http', request, response, code, msg, hdrs) File "c:\python25\lib\urllib2.py", line 412, in error result = self._call_chain(*args) File "c:\python25\lib\urllib2.py", line 353, in _call_chain result = func(*args) File "c:\python25\lib\urllib2.py", line 831, in http_error_407 authority, req, headers) File "c:\python25\lib\urllib2.py", line 795, in http_error_auth_reqed return self.retry_http_basic_auth(host, req, realm) File "c:\python25\lib\urllib2.py", line 805, in retry_http_basic_auth return self.parent.open(req) File "c:\python25\lib\urllib2.py", line 380, in open response = meth(req, response) File "c:\python25\lib\urllib2.py", line 491, in http_response 'http', request, response, code, msg, hdrs) File "c:\python25\lib\urllib2.py", line 418, in error return self._call_chain(*args) File "c:\python25\lib\urllib2.py", line 353, in _call_chain result = func(*args) File "c:\python25\lib\urllib2.py", line 499, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required
msg29121 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-13 10:09
Logged In: YES user_id=226518 2.5b2 does not work any better: Python 2.5b2 (r25b2:50512, Jul 11 2006, 10:16:14) [MSC v.1310 32 bit (Intel)] on win32 Result is the same as in 2.5b1 :(
msg29122 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-19 11:12
Logged In: YES user_id=226518 I have checked that the last wersion my script works with is 2.4.2 and copied that version urllib2.py to 2.4.3 Lib directory. It works again. The only change in urllib2.py is in retry_http_basic_auth(), line 723: 2.4.2 user,pw = self.passwd.find_user_password(realm, host) 2.4.3 user, pw = self.passwd.find_user_password(realm, req.get_full_url()) So "host" is replaced by "req.get_full_url()". Checked again with 2.5b2 and it works! Probably I destroyed my test environment and tested it wrong way :( So the problem is only with 2.4.3. Previous versions and 2.5b works well. Regards, Michal Niklas
msg29123 - (view)	Author: John J Lee (jjlee)	Date: 2006-07-20 18:09
Logged In: YES user_id=261020 You're sure you didn't copy over the urllib2.py from 2.5b2 also? That might make the bug appear to go away, when really it's still there. The way to be sure is to try it on a different machine. Thanks for your report.
msg29124 - (view)	Author: Michal Niklas (mniklas)	Date: 2006-07-21 07:59
Logged In: YES user_id=226518 I have just installed new virtual machine with Python 2.5b2 and my program works. It seems that only 2.4.3 is broken. Regards, Michal
msg29125 - (view)	Author: JerryKhan (jerrykhan)	Date: 2006-11-28 15:17
Hello, In my sens in a general manner there is something wrong in the urllib2 http code: But this may depends on the environment (I am not an expert in urllib) Here are my tests : using python 2.4.2 on Windows XP These simple codes failed with a 407 http error : Example E1: import urllib2 as URL a=URL.urlopen("http://lan_apache_url") print a.read() OR example E2: import urllib2 as URL r=URL.Request("http://lan_apache_url") a=URL.urlopen(r) print a.read() But succeed with urllib example E3 import urllib a=urllib.urlopen("http://lan_apache_url") print a.read() Notice that different code lines are minimal E1 and E3 are close: Notice also that I'm try to access a lan apache server which is not behind a Proxy. And I don't want to access to any Proxy (like exclusion string in IExplorer) But I found also that If I try to access to a protected link with HTTPS ... on the LAN, there is not problem. The issue is really on the HTTP interpreter or during the configuration of the URL opener. In the same time, some of my programs are able to access to Internet servers using the current Proxy server without any problem. For that, I use: import urllib2 as URL URL.install_opener(URL.build_opener( s.https_handler, s.proxy_auth_handler, s.cookie_handler)) Well, I developed a workaround in my programs ... to use urllib instead of urllib2 in the case where I try to access the LAN (in fact when I don't want to configure the Proxy server, or when the URL match my own proxy exclusion list.) I espect this will help python urllib2 experts to find the issue. Jérôme Vacher alias jerrykhan the foolish dracomorpheus of the emerald dragon dynasty.
msg29126 - (view)	Author: José Lecaros Cisterna (lecaros)	Date: 2007-01-03 16:33
i have the same issue on windows xp, python 2.4.3 but using DOMAIN\username format
msg29127 - (view)	Author: John J Lee (jjlee)	Date: 2007-01-03 19:37
lecaros and jerrykhan: Do you guys by any chance have a registry key HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyOverride? I'm guessing this setting is causing urllib to avoid using your default proxy for hosts on your local network, thereby saving you the 407 (the 407 means your proxy is complaining that you've not succeeded in authenticating). If so, the difference between urllib and urllib2's behaviours does not imply a bug, but just that urllib2 is missing support for getting proxy overrides from the Windows registry. This could easily be added.
msg29128 - (view)	Author: José Lecaros Cisterna (lecaros)	Date: 2007-01-03 20:02
I have that key set to , but I don't know what this mean :) Use for local OR don't use for local.
msg95667 - (view)	Author: Senthil Kumaran (orsenthil) *	Date: 2009-11-24 10:05
HTTP Proxy Authentication works on Windows and has been verified. Under the situation of failure, we do not enough details. Closing this bug as out-of-date as the reporter informed it worked on py2.5 and problem was with py2.4.