cpython: 6d5336a193cc (original) (raw)

Mercurial > cpython

changeset 95555:6d5336a193cc 3.4

Issue #12955: Change the urlopen() examples to use context managers where appropriate. Patch by Martin Panter. [#12955]

Berker Peksag berker.peksag@gmail.com
date	Sun, 12 Apr 2015 13:52:49 +0300
parents	7463c06f6e87
children	08adaaf08697 c6dc1e0db7f0
files	Doc/faq/library.rst Doc/howto/urllib2.rst Doc/library/concurrent.futures.rst Doc/library/urllib.request.rst Doc/tutorial/stdlib.rst
diffstat	5 files changed, 38 insertions(+), 28 deletions(-)[+] [-] Doc/faq/library.rst 3 Doc/howto/urllib2.rst 16 Doc/library/concurrent.futures.rst 4 Doc/library/urllib.request.rst 34 Doc/tutorial/stdlib.rst 9

line wrap: on

line diff

--- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -687,7 +687,8 @@ Yes. Here's a simple example that uses u ### connect and send the server a path req = urllib.request.urlopen('http://www.some-server.out-there'[](#l1.5) '/cgi-bin/some-cgi-script', data=qs)

msg, hdrs = req.read(), req.info()

with req:

  msg, hdrs = req.read(), req.info()[](#l1.9)

Note that in general for percent-encoded POST operations, query strings must be quoted using :func:urllib.parse.urlencode. For example, to send

--- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -53,8 +53,8 @@ Fetching URLs The simplest way to use urllib.request is as follows:: import urllib.request

response = urllib.request.urlopen('http://python.org/')[](#l2.7)
html = response.read()

with urllib.request.urlopen('http://python.org/') as response:
```
  html = response.read()[](#l2.10)
```

If you wish to retrieve a resource via URL and store it in a temporary location, you can do so via the :func:~urllib.request.urlretrieve function:: @@ -79,8 +79,8 @@ response:: import urllib.request req = urllib.request.Request('http://www.voidspace.org.uk')[](#l2.17)

response = urllib.request.urlopen(req)
the_page = response.read()

with urllib.request.urlopen(req) as response:
```
  the_page = response.read()[](#l2.21)
```

Note that urllib.request makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so:: @@ -117,8 +117,8 @@ library. :: data = urllib.parse.urlencode(values) data = data.encode('utf-8') # data should be bytes req = urllib.request.Request(url, data)

response = urllib.request.urlopen(req)
the_page = response.read()

with urllib.request.urlopen(req) as response:
```
  the_page = response.read()[](#l2.32)
```

Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see `HTML Specification, Form Submission @@ -183,8 +183,8 @@ Explorer [#]_. :: data = urllib.parse.urlencode(values) data = data.encode('utf-8') req = urllib.request.Request(url, data, headers)

response = urllib.request.urlopen(req)
the_page = response.read()

with urllib.request.urlopen(req) as response:
```
  the_page = response.read()[](#l2.43)
```

The response also has two useful methods. See the section on info and geturl_ which comes after we have a look at what happens when things go wrong.

--- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -138,8 +138,8 @@ ThreadPoolExecutor Example # Retrieve a single page and report the url and contents def load_url(url, timeout):

  conn = urllib.request.urlopen(url, timeout=timeout)[](#l3.7)

```
  return conn.readall()[](#l3.8)
```

  with urllib.request.urlopen(url, timeout=timeout) as conn:[](#l3.9)

```
      return conn.read()[](#l3.10)
```

# We can use a with statement to ensure threads are cleaned up promptly with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

--- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -1048,8 +1048,9 @@ This example gets the python.org main pa it. :: >>> import urllib.request

f = urllib.request.urlopen('http://www.python.org/')[](#l4.7)
print(f.read(300))

with urllib.request.urlopen('http://www.python.org/') as f:
... print(f.read(300))
... b'\n\n\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n\n\n @@ -1091,8 +1092,9 @@ when the Python installation supports SS

import urllib.request req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi',[](#l4.17) ... data=b'This data is passed to stdin of the CGI')

f = urllib.request.urlopen(req)
print(f.read().decode('utf-8'))

with urllib.request.urlopen(req) as f:
... print(f.read().decode('utf-8'))
... Got Data: "This data is passed to stdin of the CGI"

The code for the sample CGI used in the above example is:: @@ -1107,7 +1109,8 @@ Here is an example of doing a PUT re import urllib.request DATA=b'some data' req = urllib.request.Request(url='http://localhost:8080', data=DATA,method='PUT')

f = urllib.request.urlopen(req)

with urllib.request.urlopen(req) as f:
```
   pass[](#l4.33)
```
print(f.status) print(f.reason) @@ -1173,8 +1176,10 @@ containing parameters::

import urllib.request import urllib.parse params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})

f = urllib.request.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
print(f.read().decode('utf-8'))

url = "http://www.musi-cal.com/cgi-bin/query?%s" % params
with urllib.request.urlopen(url) as f:
... print(f.read().decode('utf-8'))
...

The following example uses the POST method instead. Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data:: @@ -1186,8 +1191,9 @@ from urlencode is encoded to bytes befor >>> request = urllib.request.Request("http://requestb.in/xrbl82xr")[](#l4.51) >>> # adding charset parameter to the Content-Type header. >>> request.add_header("Content-Type","application/x-www-form-urlencoded;charset=utf-8")

f = urllib.request.urlopen(request, data)
print(f.read().decode('utf-8'))

with urllib.request.urlopen(request, data) as f:
... print(f.read().decode('utf-8'))
...

The following example uses an explicitly specified HTTP proxy, overriding environment settings:: @@ -1195,15 +1201,17 @@ environment settings:: >>> import urllib.request >>> proxies = {'http': 'http://proxy.example.com:8080/'}[](#l4.64) >>> opener = urllib.request.FancyURLopener(proxies)

f = opener.open("http://www.python.org")[](#l4.66)
f.read().decode('utf-8')

with opener.open("http://www.python.org") as f:
... f.read().decode('utf-8')
...

The following example uses no proxies at all, overriding environment settings:: >>> import urllib.request >>> opener = urllib.request.FancyURLopener({})

f = opener.open("http://www.python.org/")[](#l4.76)
f.read().decode('utf-8')

with opener.open("http://www.python.org/") as f:
... f.read().decode('utf-8')
...

Legacy interface

--- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -153,10 +153,11 @@ protocols. Two of the simplest are :mod: from URLs and :mod:smtplib for sending mail:: >>> from urllib.request import urlopen

for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):[](#l5.7)
... line = line.decode('utf-8') # Decoding the binary data to text.
... if 'EST' in line or 'EDT' in line: # look for Eastern Time
... print(line)

with urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl') as response:
... for line in response:
... line = line.decode('utf-8') # Decoding the binary data to text.
... if 'EST' in line or 'EDT' in line: # look for Eastern Time
... print(line)
Nov. 25, 09:43:32 PM EST