cpython: 720f0cf580e2 (original) (raw)

Mercurial > cpython

changeset 103203:720f0cf580e2

Issue #6135: Adds encoding and errors parameters to subprocess [#6135]

Steve Dower steve.dower@microsoft.com
date	Tue, 06 Sep 2016 20:16:17 -0700
parents	7a243a40b421
children	b38e68ff9751
files	Doc/library/subprocess.rst Doc/whatsnew/3.6.rst Lib/subprocess.py Lib/test/test_subprocess.py Misc/NEWS
diffstat	5 files changed, 154 insertions(+), 100 deletions(-)[+] [-] Doc/library/subprocess.rst 103 Doc/whatsnew/3.6.rst 3 Lib/subprocess.py 87 Lib/test/test_subprocess.py 59 Misc/NEWS 2

line wrap: on

line diff

--- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -38,7 +38,8 @@ compatibility with older versions, see t .. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,[](#l1.6)

             shell=False, timeout=None, check=False)[](#l1.7)

             shell=False, timeout=None, check=False, \[](#l1.8)

             encoding=None, errors=None)[](#l1.9)

Run the command described by args. Wait for command to complete, then return a :class:CompletedProcess instance. @@ -60,15 +61,20 @@ compatibility with older versions, see t The input argument is passed to :meth:Popen.communicate and thus to the subprocess's stdin. If used it must be a byte sequence, or a string if

universal_newlines=True. When used, the internal :class:Popen object
is automatically created with stdin=PIPE, and the stdin argument may
not be used as well.

encoding or errors is specified or universal_newlines is True. When
used, the internal :class:Popen object is automatically created with
stdin=PIPE, and the stdin argument may not be used as well. If check is True, and the process exits with a non-zero exit code, a :exc:CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.
If encoding or errors are specified, or universal_newlines is True,
file objects for stdin, stdout and stderr are opened in text mode using the
specified encoding and errors or the :class:io.TextIOWrapper default.
Otherwise, file objects are opened in binary mode. + Examples::

subprocess.run(["ls", "-l"]) # doesn't capture output @@ -85,6 +91,10 @@ compatibility with older versions, see t .. versionadded:: 3.5
.. versionchanged:: 3.6 +

 Added *encoding* and *errors* parameters[](#l1.43)

+ .. class:: CompletedProcess The return value from :func:run, representing a process that has finished. @@ -104,8 +114,8 @@ compatibility with older versions, see t .. attribute:: stdout Captured stdout from the child process. A bytes sequence, or a string if

 :func:`run` was called with ``universal_newlines=True``. None if stdout[](#l1.52)

```
 was not captured.[](#l1.53)
```

 :func:`run` was called with an encoding or errors. None if stdout was not[](#l1.54)

```
 captured.[](#l1.55)
```

If you ran the process with stderr=subprocess.STDOUT, stdout and stderr will be combined in this attribute, and :attr:stderr will be @@ -114,8 +124,8 @@ compatibility with older versions, see t .. attribute:: stderr Captured stderr from the child process. A bytes sequence, or a string if

 :func:`run` was called with ``universal_newlines=True``. None if stderr[](#l1.63)

```
 was not captured.[](#l1.64)
```

 :func:`run` was called with an encoding or errors. None if stderr was not[](#l1.65)

```
 captured.[](#l1.66)
```

.. method:: check_returncode() @@ -249,19 +259,22 @@ default values. The arguments that are m .. index:: single: universal newlines; subprocess module

If universal_newlines is False the file objects stdin, stdout and
stderr will be opened as binary streams, and no line ending conversion is
done.

If encoding or errors are specified, or universal_newlines is True,
the file objects stdin, stdout and stderr will be opened in text
mode using the encoding and errors specified in the call or the
defaults for :class:io.TextIOWrapper.

If universal_newlines is True, these file objects
will be opened as text streams in :term:universal newlines mode
using the encoding returned by :func:`locale.getpreferredencoding(False)
<locale.getpreferredencoding>`. For stdin, line ending characters
'\n' in the input will be converted to the default line separator
:data:os.linesep. For stdout and stderr, all line endings in the
output will be converted to '\n'. For more information see the
documentation of the :class:io.TextIOWrapper class when the newline
argument to its constructor is None.

For stdin, line ending characters '\n' in the input will be converted
to the default line separator :data:os.linesep. For stdout and stderr,
all line endings in the output will be converted to '\n'. For more
information see the documentation of the :class:io.TextIOWrapper class
when the newline argument to its constructor is None. +
If text mode is not used, stdin, stdout and stderr will be opened as
binary streams. No encoding or line ending conversion is performed. +
.. versionadded:: 3.6

 Added *encoding* and *errors* parameters.[](#l1.101)

.. note:: @@ -306,7 +319,8 @@ functions. stderr=None, preexec_fn=None, close_fds=True, shell=False, [](#l1.106) cwd=None, env=None, universal_newlines=False, [](#l1.107) startupinfo=None, creationflags=0, restore_signals=True, [](#l1.108)

            start_new_session=False, pass_fds=())[](#l1.109)

            start_new_session=False, pass_fds=(), *, \[](#l1.110)

            encoding=None, errors=None)[](#l1.111)

Execute a child program in a new process. On POSIX, the class uses :meth:os.execvp-like behavior to execute the child program. On Windows, @@ -482,10 +496,14 @@ functions. .. _side-by-side assembly: https://en.wikipedia.org/wiki/Side-by-Side_Assembly[](#l1.117)

If universal_newlines is True, the file objects stdin, stdout
and stderr are opened as text streams in universal newlines mode, as
described above in :ref:frequently-used-arguments, otherwise they are
opened as binary streams.

If encoding or errors are specified, the file objects stdin, stdout
and stderr are opened in text mode with the specified encoding and
errors, as described above in :ref:frequently-used-arguments. If
universal_newlines is True, they are opened in text mode with default
encoding. Otherwise, they are opened as binary streams. +
.. versionadded:: 3.6

 *encoding* and *errors* were added.[](#l1.130)

If given, startupinfo will be a :class:STARTUPINFO object, which is passed to the underlying CreateProcess function. @@ -601,11 +619,12 @@ Instances of the :class:Popen class ha Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be data to be sent to the child process, or

None, if no data should be sent to the child. The type of input
must be bytes or, if universal_newlines was True, a string.

None, if no data should be sent to the child. If streams were opened in
text mode, input must be a string. Otherwise, it must be bytes. :meth:communicate returns a tuple (stdout_data, stderr_data).

The data will be bytes or, if universal_newlines was True, strings.

The data will be strings if streams were opened in text mode; otherwise,
bytes. Note that if you want to send data to the process's stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than @@ -672,28 +691,30 @@ The following attributes are also availa .. attribute:: Popen.stdin If the stdin argument was :data:PIPE, this attribute is a writeable

stream object as returned by :func:open. If the universal_newlines
argument was True, the stream is a text stream, otherwise it is a byte
stream. If the stdin argument was not :data:PIPE, this attribute is
None.

stream object as returned by :func:open. If the encoding or errors
arguments were specified or the universal_newlines argument was True,
the stream is a text stream, otherwise it is a byte stream. If the stdin
argument was not :data:PIPE, this attribute is None.

.. attribute:: Popen.stdout If the stdout argument was :data:PIPE, this attribute is a readable stream object as returned by :func:open. Reading from the stream provides

output from the child process. If the universal_newlines argument was
True, the stream is a text stream, otherwise it is a byte stream. If the
stdout argument was not :data:PIPE, this attribute is None.

output from the child process. If the encoding or errors arguments were
specified or the universal_newlines argument was True, the stream is a
text stream, otherwise it is a byte stream. If the stdout argument was not
:data:PIPE, this attribute is None.

.. attribute:: Popen.stderr If the stderr argument was :data:PIPE, this attribute is a readable stream object as returned by :func:open. Reading from the stream provides

error output from the child process. If the universal_newlines argument was
True, the stream is a text stream, otherwise it is a byte stream. If the
stderr argument was not :data:PIPE, this attribute is None.

error output from the child process. If the encoding or errors arguments
were specified or the universal_newlines argument was True, the stream
is a text stream, otherwise it is a byte stream. If the stderr argument was
not :data:PIPE, this attribute is None.

.. warning:: @@ -886,7 +907,9 @@ calls these functions. timeout was added. -.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, [](#l1.196)

                      encoding=None, errors=None, \[](#l1.197)

                      universal_newlines=False, timeout=None)[](#l1.198)

Run command with arguments and return its output. @@ -1142,7 +1165,7 @@ handling consistency are valid for these Return (status, output) of executing cmd in a shell. Execute the string cmd in a shell with :meth:Popen.check_output and

return a 2-tuple (status, output). Universal newlines mode is used;

return a 2-tuple (status, output). The locale encoding is used; see the notes on :ref:frequently-used-arguments for more details. A trailing newline is stripped from the output.

--- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -589,6 +589,9 @@ proc: ...``) or call explicitly the :met read the exit status of the child process (Contributed by Victor Stinner in :issue:26741). +The :class:subprocess.Popen constructor and all functions that pass arguments +through to it now accept encoding and errors arguments. Specifying either +of these will enable text mode for the stdin, stdout and stderr streams. telnetlib ---------

--- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -30,7 +30,8 @@ class Popen(args, bufsize=-1, executable preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0,

       restore_signals=True, start_new_session=False, pass_fds=()):[](#l3.7)

       restore_signals=True, start_new_session=False, pass_fds=(),[](#l3.8)

       *, encoding=None, errors=None):[](#l3.9)

Arguments are: @@ -104,20 +105,13 @@ in the child process prior to executing If env is not None, it defines the environment variables for the new process. -If universal_newlines is False, the file objects stdin, stdout and stderr -are opened as binary files, and no line ending conversion is done. +If encoding or errors are specified or universal_newlines is True, the file +objects stdout and stderr are opened in text mode. See io.TextIOWrapper for +the interpretation of these parameters are used. -If universal_newlines is True, the file objects stdout and stderr are -opened as a text file, but lines may be terminated by any of '\n', -the Unix end-of-line convention, '\r', the old Macintosh convention or -'\r\n', the Windows convention. All of these external representations -are seen as '\n' by the Python program. Also, the newlines attribute -of the file objects stdout, stdin and stderr are not updated by the -communicate() method. - -In either case, the process being communicated with should start up -expecting to receive bytes on its standard input and decode them with -the same encoding they are sent in. +If no encoding is specified and universal_newlines is False, the file +objects stdin, stdout and stderr are opened as binary files, and no +line ending conversion is done. The startupinfo and creationflags, if given, will be passed to the underlying CreateProcess() function. They can specify things such as @@ -234,11 +228,8 @@ communicate(input=None) and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be data to be sent to the child process, or None, if no data should be sent to

the child. If the Popen instance was constructed with universal_newlines
set to True, the input argument should be a string and will be encoded
using the preferred system encoding (see locale.getpreferredencoding);
if universal_newlines is False, the input argument should be a
byte string.

the child. If the Popen instance was constructed in text mode, the
input argument should be a string. Otherwise, it should be bytes.

communicate() returns a tuple (stdout, stderr). @@ -808,8 +799,8 @@ def getstatusoutput(cmd): """ Return (status, output) of executing cmd in a shell. Execute the string 'cmd' in a shell with 'check_output' and

return a 2-tuple (status, output). Universal newlines mode is used,
meaning that the result with be decoded to a string.

return a 2-tuple (status, output). The locale encoding is used
to decode the output and process newlines.

A trailing newline is stripped from the output. The exit status for the command can be interpreted @@ -859,7 +850,7 @@ class Popen(object): shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False,

```
            pass_fds=()):[](#l3.69)
```

            pass_fds=(), *, encoding=None, errors=None):[](#l3.70)
   """Create new Popen instance."""[](#l3.71)
   _cleanup()[](#l3.72)
   # Held while anything is calling waitpid before returncode has been[](#l3.73)

@@ -912,6 +903,8 @@ class Popen(object): self.pid = None self.returncode = None self.universal_newlines = universal_newlines

```
   self.encoding = encoding[](#l3.78)
```
```
   self.errors = errors[](#l3.79)
```

# Input and output objects. The general principle is like # this: @@ -944,22 +937,28 @@ class Popen(object): if errread != -1: errread = msvcrt.open_osfhandle(errread.Detach(), 0)

```
   if p2cwrite != -1:[](#l3.87)
```

       self.stdin = io.open(p2cwrite, 'wb', bufsize)[](#l3.88)

```
       if universal_newlines:[](#l3.89)
```

           self.stdin = io.TextIOWrapper(self.stdin, write_through=True,[](#l3.90)

                                         line_buffering=(bufsize == 1))[](#l3.91)

```
   if c2pread != -1:[](#l3.92)
```

       self.stdout = io.open(c2pread, 'rb', bufsize)[](#l3.93)

```
       if universal_newlines:[](#l3.94)
```

           self.stdout = io.TextIOWrapper(self.stdout)[](#l3.95)

```
   if errread != -1:[](#l3.96)
```

       self.stderr = io.open(errread, 'rb', bufsize)[](#l3.97)

```
       if universal_newlines:[](#l3.98)
```

           self.stderr = io.TextIOWrapper(self.stderr)[](#l3.99)

   text_mode = encoding or errors or universal_newlines[](#l3.100)

self._closed_child_pipe_fds = False + try:

```
       if p2cwrite != -1:[](#l3.105)
```

           self.stdin = io.open(p2cwrite, 'wb', bufsize)[](#l3.106)

```
           if text_mode:[](#l3.107)
```

               self.stdin = io.TextIOWrapper(self.stdin, write_through=True,[](#l3.108)

                       line_buffering=(bufsize == 1),[](#l3.109)

                       encoding=encoding, errors=errors)[](#l3.110)

```
       if c2pread != -1:[](#l3.111)
```

           self.stdout = io.open(c2pread, 'rb', bufsize)[](#l3.112)

```
           if text_mode:[](#l3.113)
```

               self.stdout = io.TextIOWrapper(self.stdout,[](#l3.114)

                       encoding=encoding, errors=errors)[](#l3.115)

```
       if errread != -1:[](#l3.116)
```

           self.stderr = io.open(errread, 'rb', bufsize)[](#l3.117)

```
           if text_mode:[](#l3.118)
```

               self.stderr = io.TextIOWrapper(self.stderr,[](#l3.119)

                       encoding=encoding, errors=errors)[](#l3.120)

+ self._execute_child(args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, @@ -993,8 +992,8 @@ class Popen(object): raise

def _translate_newlines(self, data, encoding):

   data = data.decode(encoding)[](#l3.130)

def _translate_newlines(self, data, encoding, errors):

   data = data.decode(encoding, errors)[](#l3.132)
   return data.replace("\r\n", "\n").replace("\r", "\n")[](#l3.133)

def enter(self): @@ -1779,13 +1778,15 @@ class Popen(object): # Translate newlines, if requested. # This also turns bytes into strings.

       if self.universal_newlines:[](#l3.140)

       if self.encoding or self.errors or self.universal_newlines:[](#l3.141)
           if stdout is not None:[](#l3.142)
               stdout = self._translate_newlines(stdout,[](#l3.143)

                                                 self.stdout.encoding)[](#l3.144)

                                                 self.stdout.encoding,[](#l3.145)

                                                 self.stdout.errors)[](#l3.146)
           if stderr is not None:[](#l3.147)
               stderr = self._translate_newlines(stderr,[](#l3.148)

                                                 self.stderr.encoding)[](#l3.149)

                                                 self.stderr.encoding,[](#l3.150)

                                                 self.stderr.errors)[](#l3.151)

return (stdout, stderr) @@ -1797,8 +1798,10 @@ class Popen(object): if self.stdin and self._input is None: self._input_offset = 0 self._input = input

           if self.universal_newlines and input is not None:[](#l3.159)

               self._input = self._input.encode(self.stdin.encoding)[](#l3.160)

           if input is not None and ([](#l3.161)

               self.encoding or self.errors or self.universal_newlines):[](#l3.162)

               self._input = self._input.encode(self.stdin.encoding,[](#l3.163)

                                                self.stdin.errors)[](#l3.164)

def send_signal(self, sig):

--- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -894,31 +894,42 @@ class ProcessTestCase(BaseTestCase): # # UTF-16 and UTF-32-BE are sufficient to check both with BOM and # without, and UTF-16 and UTF-32.

   import _bootlocale[](#l4.7)
   for encoding in ['utf-16', 'utf-32-be']:[](#l4.8)

       old_getpreferredencoding = _bootlocale.getpreferredencoding[](#l4.9)

       # Indirectly via io.TextIOWrapper, Popen() defaults to[](#l4.10)

       # locale.getpreferredencoding(False) and earlier in Python 3.2 to[](#l4.11)

       # locale.getpreferredencoding().[](#l4.12)

       def getpreferredencoding(do_setlocale=True):[](#l4.13)

           return encoding[](#l4.14)
       code = ("import sys; "[](#l4.15)
               r"sys.stdout.buffer.write('1\r\n2\r3\n4'.encode('%s'))" %[](#l4.16)
               encoding)[](#l4.17)
       args = [sys.executable, '-c', code][](#l4.18)

```
       try:[](#l4.19)
```

           _bootlocale.getpreferredencoding = getpreferredencoding[](#l4.20)

           # We set stdin to be non-None because, as of this writing,[](#l4.21)

           # a different code path is used when the number of pipes is[](#l4.22)

```
           # zero or one.[](#l4.23)
```

           popen = subprocess.Popen(args, universal_newlines=True,[](#l4.24)

                                    stdin=subprocess.PIPE,[](#l4.25)

                                    stdout=subprocess.PIPE)[](#l4.26)

           stdout, stderr = popen.communicate(input='')[](#l4.27)

```
       finally:[](#l4.28)
```

           _bootlocale.getpreferredencoding = old_getpreferredencoding[](#l4.29)

       # We set stdin to be non-None because, as of this writing,[](#l4.30)

       # a different code path is used when the number of pipes is[](#l4.31)

```
       # zero or one.[](#l4.32)
```

       popen = subprocess.Popen(args,[](#l4.33)

                                stdin=subprocess.PIPE,[](#l4.34)

                                stdout=subprocess.PIPE,[](#l4.35)

                                encoding=encoding)[](#l4.36)

       stdout, stderr = popen.communicate(input='')[](#l4.37)
       self.assertEqual(stdout, '1\n2\n3\n4')[](#l4.38)

def test_communicate_errors(self):
```
   for errors, expected in [[](#l4.41)
```
```
       ('ignore', ''),[](#l4.42)
```

       ('replace', '\ufffd\ufffd'),[](#l4.43)

       ('surrogateescape', '\udc80\udc80'),[](#l4.44)

       ('backslashreplace', '\\x80\\x80'),[](#l4.45)

```
   ]:[](#l4.46)
```
```
       code = ("import sys; "[](#l4.47)
```

               r"sys.stdout.buffer.write(b'[\x80\x80]')")[](#l4.48)

       args = [sys.executable, '-c', code][](#l4.49)

       # We set stdin to be non-None because, as of this writing,[](#l4.50)

       # a different code path is used when the number of pipes is[](#l4.51)

```
       # zero or one.[](#l4.52)
```

       popen = subprocess.Popen(args,[](#l4.53)

                                stdin=subprocess.PIPE,[](#l4.54)

                                stdout=subprocess.PIPE,[](#l4.55)

                                encoding='utf-8',[](#l4.56)

                                errors=errors)[](#l4.57)

       stdout, stderr = popen.communicate(input='')[](#l4.58)

       self.assertEqual(stdout, '[{}]'.format(expected))[](#l4.59)

+ def test_no_leaking(self): # Make sure we leak no resources if not mswindows: @@ -2539,6 +2550,18 @@ class Win32ProcessTestCase(BaseTestCase) with p: self.assertIn(b"physalis", p.stdout.read())

def test_shell_encodings(self):

   # Run command through the shell (string)[](#l4.69)

   for enc in ['ansi', 'oem']:[](#l4.70)

       newenv = os.environ.copy()[](#l4.71)

       newenv["FRUIT"] = "physalis"[](#l4.72)

       p = subprocess.Popen("set", shell=1,[](#l4.73)

                            stdout=subprocess.PIPE,[](#l4.74)

                            env=newenv,[](#l4.75)

                            encoding=enc)[](#l4.76)

```
       with p:[](#l4.77)
```

           self.assertIn("physalis", p.stdout.read(), enc)[](#l4.78)

+ def test_call_string(self): # call() function with string argument on Windows rc = subprocess.call(sys.executable +

--- a/Misc/NEWS +++ b/Misc/NEWS @@ -237,6 +237,8 @@ Build Windows ------- +- Issue #6135: Adds encoding and errors parameters to subprocess. +

Issue #27959: Adds oem encoding, alias ansi to mbcs, move aliasmbcs to codec lookup.