Issue 16109: urllib2.urlopen throws HTTP Error in python 2.7.2, 2.7.3, but not in python 2.7 (original) (raw)

Issue16109

Created on 2012-10-02 15:15 by fertig, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (10)
msg171811 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-02 15:15
wespe:/tmp/python2.7 # cat /etc/SuSE-release openSUSE 12.1 (x86_64) VERSION = 12.1 CODENAME = Asparagus wespe:/tmp/python2.7 # rpm -q python python-2.7.2-7.17.1.x86_64 wespe:/tmp/python2.7 # python Python 2.7.2 (default, Aug 19 2011, 20:41:43) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2; print urllib2.urlopen('https://172.23.6.222/') # not working Traceback (most recent call last): File "", line 1, in File "urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "urllib2.py", line 397, in open response = meth(req, response) File "urllib2.py", line 510, in http_response 'http', request, response, code, msg, hdrs) File "urllib2.py", line 435, in error return self._call_chain(*args) File "urllib2.py", line 369, in _call_chain result = func(*args) File "urllib2.py", line 518, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 405: Method Not Allowed >>> import urllib; print urllib.urlopen('https://172.23.6.222/') # working <addinfourl at 10613608 whose fp = <socket._fileobject object at 0x7f7c20a41d50>> Other machine: hornisse:/usr/lib64 # cat /etc/SuSE-release openSUSE 11.4 (x86_64) VERSION = 11.4 CODENAME = Celadon hornisse:/usr/lib64 # rpm -q python python-2.7-9.40.1.x86_64 hornisse:/usr/lib64 # python Python 2.7 (r27:82500, Aug 07 2010, 16:54:59) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2; print urllib2.urlopen('https://172.23.6.222/') # working <addinfourl at 7832424 whose fp = <socket._fileobject object at 0x756a50>> >>> import urllib; print urllib.urlopen('https://172.23.6.222/') # working <addinfourl at 10130856 whose fp = <socket._fileobject object at 0x756850>> Sometimes the HTTP Error is a 400 Bad Request, but 99% of the time a 405. I've testet openSuSE 12.2 with python 2.7.3 too, not working. I don't know, what to provide more on information, please contact me, if you need more.
msg171819 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2012-10-02 17:21
That's a non-public IP, so you'll have to provide a trace of the http traffic. It is likely the error is correct, and is reported only post 2.7.0 because a bug was fixed.
msg172058 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-05 07:53
ok, here's a run with debuglevel for the headers: python 2.7 on openSuSE 11.4: fertig@hornisse:/home/fertig > python Python 2.7 (r27:82500, Aug 07 2010, 16:54:59) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2 >>> request = urllib2.Request('https://172.23.6.222') >>> opener = urllib2.build_opener(urllib2.HTTPSHandler(debuglevel=111)) >>> feeddata = opener.open(request).read() send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Fri, 05 Oct 2012 09:46:33 GMT header: Server: Virata-EmWeb/R6_2_0 header: Connection: close header: Transfer-Encoding: chunked header: Content-Type: text/html header: Expires: Fri, 05 Oct 2012 09:46:33 GMT header: Last-Modified: Fri, 05 Oct 2012 09:46:33 GMT header: Cache-Control: no-cache >>> python 2.7.2 on openSuSE 12.1: fertig@wespe:/home/fertig > python Python 2.7.2 (default, Aug 19 2011, 20:41:43) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2 >>> request = urllib2.Request('https://172.23.6.222') >>> opener = urllib2.build_opener(urllib2.HTTPSHandler(debuglevel=111)) >>> feeddata = opener.open(request).read() <urllib2.Request instance at 0x90d878> send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 405 Method Not Allowed\r\n' header: Date: Fri, 05 Oct 2012 09:44:11 GMT header: Server: Virata-EmWeb/R6_2_0 header: Connection: close Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.7/urllib2.py", line 401, in open response = meth(req, response) File "/usr/lib64/python2.7/urllib2.py", line 514, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python2.7/urllib2.py", line 439, in error return self._call_chain(*args) File "/usr/lib64/python2.7/urllib2.py", line 372, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 522, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 405: Method Not Allowed >>>
msg172059 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-05 07:57
the IP is private and even for testing I can't put it in a public IP range, as the device is an IP Phone with a custom firmware (Siemens optiPoint 420).
msg172078 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2012-10-05 13:03
It looks like the data input is the same in both cases. (Although why does your second example show '<urllib2.Request instance at 0x90d878>'? Does urllib2 really output that as part of the debug output?) It is hard to see what could be causing the divergence in response here. Is it possible for there to be something in the SSL setup that triggers a different response from your server? Any chance you could try Python 3.2 or 3.3 just to see if there is a difference there?
msg172080 - (view)	Author: Antoine Pitrou (pitrou) *	Date: 2012-10-05 13:13
By using "openssl s_client -connect 172.23.6.222:443" you can open a telnet-like session (but using SSL) to your server, and try to type/paste an HTTP request there. That would help check whether it's a Python problem, or something else due to OpenSSL changes.
msg172081 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-05 13:18
python3 example, not working on both machines (3.2.1 on openSuSE 12.1 and 3.1.3 on openSuSE 11.4) fertig@wespe:/home/fertig > python3 Error in sitecustomize; set PYTHONVERBOSE for traceback: SyntaxError: invalid syntax (sitecustomize.py, line 89) Python 3.2.1 (default, Jul 18 2011, 16:24:40) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import http.client >>> import urllib.request >>> http.client.HTTPConnection.debuglevel = 1 >>> request = urllib.request.urlopen('https://172.23.6.222') send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/3.2\r\n\r\n' reply: 'HTTP/1.1 400 Bad Request\r\n' header: Date header: Server Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.2/urllib/request.py", line 138, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.2/urllib/request.py", line 375, in open response = meth(req, response) File "/usr/lib64/python3.2/urllib/request.py", line 487, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python3.2/urllib/request.py", line 413, in error return self._call_chain(*args) File "/usr/lib64/python3.2/urllib/request.py", line 347, in _call_chain result = func(*args) File "/usr/lib64/python3.2/urllib/request.py", line 495, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 400: Bad Request >>> request = urllib.request.urlopen('https://172.23.6.222') send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/3.2\r\n\r\n' reply: 'HTTP/1.1 405 Method Not Allowed\r\n' header: Date header: Server header: Connection Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.2/urllib/request.py", line 138, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.2/urllib/request.py", line 375, in open response = meth(req, response) File "/usr/lib64/python3.2/urllib/request.py", line 487, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python3.2/urllib/request.py", line 413, in error return self._call_chain(*args) File "/usr/lib64/python3.2/urllib/request.py", line 347, in _call_chain result = func(*args) File "/usr/lib64/python3.2/urllib/request.py", line 495, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 405: Method Not Allowed
msg172082 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-05 13:23
ok actually it isn't working either with curl or wget. So probably this is not a python problem, it seems more something in the underlying libs: Output from curl on the machine, where python 2.7 is working: fertig@hornisse:/home/fertig > curl --insecure 'https://172.23.6.222/' --verbose * About to connect() to 172.23.6.222 port 443 (#0) * Trying 172.23.6.222... connected * Connected to 172.23.6.222 (172.23.6.222) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-SHA * Server certificate: * subject: C=DE; L=Munich; O=Siemens Enterprise Communications; OU=Enterprise Systems; CN=optiPoint - WBM * start date: 2007-04-11 12:26:43 GMT * expire date: 2022-04-10 12:26:43 GMT * common name: optiPoint - WBM (does not match '172.23.6.222') * issuer: C=DE; L=Munich; O=Siemens AG; OU=Com Enterprise Systems; CN=Siemens Com ESY HD Security Office; emailAddress=hipath_security_office@siemens.com * SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway. > GET / HTTP/1.1 > User-Agent: curl/7.21.2 (x86_64-unknown-linux-gnu) libcurl/7.21.2 OpenSSL/1.0.0c zlib/1.2.5 libidn/1.15 libssh2/1.2.7 > Host: 172.23.6.222 > Accept: */* > < HTTP/1.1 405 Method Not Allowed < Date: Fri, 05 Oct 2012 11:03:02 GMT < Server: Virata-EmWeb/R6_2_0 < Connection: close < 405 Method Not Allowed * Closing connection #0 * SSLv3, TLS alert, Client hello (1): fertig@hornisse:/home/fertig > No idea anymore :/ But I fear you can close this ticket...
msg172083 - (view)	Author: Christian Fertig (fertig)	Date: 2012-10-05 13:28
for completeness: the openssl output from the machine, where python 2.7 is working: fertig@hornisse:/home/fertig > openssl s_client -connect 172.23.6.222:443 CONNECTED(00000003) depth=1 C = DE, L = Munich, O = Siemens AG, OU = Com Enterprise Systems, CN = Siemens Com ESY HD Security Office, emailAddress = hipath_security_office@siemens.com verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/C=DE/L=Munich/O=Siemens Enterprise Communications/OU=Enterprise Systems/CN=optiPoint - WBM i:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com 1 s:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com i:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com --- Server certificate -----BEGIN CERTIFICATE----- MIIEwDCCA6igAwIBAgIBFjANBgkqhkiG9w0BAQUFADCBtDELMAkGA1UEBhMCREUx [..] dEYuDjj21U+xronZ0FYVieIP1jk= -----END CERTIFICATE----- subject=/C=DE/L=Munich/O=Siemens Enterprise Communications/OU=Enterprise Systems/CN=optiPoint - WBM issuer=/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com --- No client certificate CA names sent --- SSL handshake has read 2748 bytes and written 407 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 1024 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: 506EDFB9B197A00F67A43F34B3313B660DBC64FF7353112836F813723B30071C Session-ID-ctx: Master-Key: 6928F9EF19DB7DBEEEA6D09392EF2F848613E40C1B6CE6226C342AB3D211AA1A8E04EF27BDEE4CD03AB4F6AD7317EBB3 Key-Arg : None PSK identity: None PSK identity hint: None Start Time: 1349443513 Timeout : 300 (sec) Verify return code: 19 (self signed certificate in certificate chain) --- GET / HTTP/1.1 HTTP/1.1 400 Bad Request Date: Fri, 05 Oct 2012 15:25:21 GMT Server: Virata-EmWeb/R6_2_0 Connection: close 400 Bad Request
msg172084 - (view)	Author: Antoine Pitrou (pitrou) *	Date: 2012-10-05 13:31
Ok, closing then. Thanks for the debugging, and good luck :)