Issue 23862: subprocess popen arguments using double quotes (original) (raw)

Using Python 2.7.9

Noticed when I run a subprocess Popen with an argument containing double quotes, it will have a different affect depending on my operating system.

In Linux, if I run ./runme.py, it will call runme.sh which will append someargs.txt correctly without escaped quotations

In Windows, if I run ./runme.py, it will call runme.bat which will append someargs.txt with escaped quotations

Also in Windows, if I run runme.bat with an argument containing quotations it will append someargs.txt correctly without escaped quotations so this problem seems to be stemming from Python.

===================================== runme.py

#!/usr/bin/python import sys import subprocess import shlex

works on Linux:

#command_line = "./runme.sh --include="check""

fails on Windows:

command_line = "runme.bat --include="check"" #command_line = "runme.bat --include="check""

arg = shlex.shlex(command_line) arg.quotes = '"' arg.whitespace_split = True arg.commenters = '' command_line_args = list(arg) print command_line_args

command_line_process = subprocess.Popen( command_line_args, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE )

line = "" while True: line = command_line_process.stdout.readline() if line: print line else: break

===================================== runme.bat

echo %* echo %* >> someargs.txt

===================================== runme.sh

#!/bin/bash echo $@ echo $@ >> someargs.txt

On Windows, subprocess.Popen has to convert an argument list to a command line (because that's what the OS expects for CreateProcess). It does so internally, using subprocess.list2cmdline, which follows the quoting rules for the MS C runtime (because that's what 99% of programs use). Unfortunately batfile argument parsing does not use these conventions, and you can find odd corner cases like this.

Specifically, it's not possible on Windows to "correctly" quote all forms of argument without knowing what program you are calling (because programs do their own argument parsing, usually but not always via the C runtime). The subprocess module has to work for unknown executables, so it chooses to do the right thing for programs using the MS C runtime argument parser (which is most programs).