[Python-Dev] PEP 389: argparse - new command line parsing module (original) (raw)

Steven Bethard steven.bethard at gmail.com
Sun Sep 27 22:59:32 CEST 2009

• Previous message: [Python-Dev] IO module precisions and exception hierarchy
• Next message: [Python-Dev] PEP 389: argparse - new command line parsing module
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Below is a new PEP based on discussions from the stdlib-sig, which proposes to add the argparse module to the standard library in Python 2.7 and 3.2.

Looking forward to your feedback!

Steve

http://www.python.org/dev/peps/pep-0389/

PEP: 389 Title: argparse - new command line parsing module Version: Revision:75097Revision: 75097 Revision:75097 Last-Modified: Date:2009−09−2712:42:40−0700(Sun,27Sep2009)Date: 2009-09-27 12:42:40 -0700 (Sun, 27 Sep 2009) Date:2009−09−2712:42:40−0700(Sun,27Sep2009) Author: Steven Bethard <steven.bethard at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-Sep-2009 Python-Version: 2.7 and 3.2 Post-History:

Abstract

This PEP proposes inclusion of the argparse [1]_ module in the Python standard library in Python 2.7 and 3.2.

Motivation

The argparse module is a command line parsing library which provides more functionality than the existing command line parsing modules in the standard library, getopt [2]_ and optparse [3]_. It includes support for positional arguments (not just options), subcommands, required options, options syntaxes like "/f" and "+rgb", zero-or-more and one-or-more style arguments, and many other features the other two lack.

The argparse module is also already a popular third-party replacement for these modules. It is used in projects like IPython (the Scipy Python shell) [4], is included in Debian testing and unstable [5], and since 2007 has had various requests for its inclusion in the standard library [6]_ [7]_ [8]_. This popularity suggests it may be a valuable addition to the Python libraries.

Why aren't getopt and optparse enough?

One argument against adding argparse is that thare are "already two different option parsing modules in the standard library" [9]_. The following is a list of features provided by argparse but not present in getopt or optparse:

• While it is true there are two option parsing libraries, there are no full command line parsing libraries -- both getopt and optparse support only options and have no support for positional arguments. The argparse module handles both, and as a result, is able to generate better help messages, avoiding redundancies like the usage= string usually required by optparse.

• The argparse module values practicality over purity. Thus, argparse allows required options and customization of which characters are used to identify options, while optparse explicitly states "the phrase 'required option' is self-contradictory" and that the option syntaxes -pf, -file, +f, +rgb, /f and /file "are not supported by optparse, and they never will be".

• The argparse module allows options to accept a variable number of arguments using nargs='?', nargs='*' or nargs='+'. The optparse module provides an untested recipe for some part of this functionality [10]_ but admits that "things get hairy when you want an option to take a variable number of arguments."

• The argparse module supports subcommands, where a main command line parser dispatches to other command line parsers depending on the command line arguments. This is a common pattern in command line interfaces, e.g. svn co and svn up.

Why isn't the functionality just being added to optparse?

Clearly all the above features offer improvements over what is available through optparse. A reasonable question then is why these features are not simply provided as patches to optparse, instead of introducing an entirely new module. In fact, the original development of argparse intended to do just that, but because of various fairly constraining design decisions of optparse, this wasn't really possible. Some of the problems included:

• The optparse module exposes the internals of its parsing algorithm. In particular, parser.largs and parser.rargs are guaranteed to be available to callbacks [11]_. This makes it extremely difficult to improve the parsing algorithm as was necessary in argparse for proper handling of positional arguments and variable length arguments. For example, nargs='+' in argparse is matched using regular expressions and thus has no notion of things like parser.largs.

• The optparse extension APIs are extremely complex. For example, just to use a simple custom string-to-object conversion function, you have to subclass Option, hack class attributes, and then specify your custom option type to the parser, like this::

  class MyOption(Option): TYPES = Option.TYPES + ("mytype",) TYPE_CHECKER = copy(Option.TYPE_CHECKER) TYPE_CHECKER["mytype"] = check_mytype parser = optparse.OptionParser(option_class=MyOption) parser.add_option("-m", type="mytype")

  For comparison, argparse simply allows conversion functions to be used as type= arguments directly, e.g.::

  parser = argparse.ArgumentParser() parser.add_option("-m", type=check_mytype)

  But given the baroque customization APIs of optparse, it is unclear how such a feature should interact with those APIs, and it is quite possible that introducing the simple argparse API would break existing custom Option code.

• Both optparse and argparse parse command line arguments and assign them as attributes to an object returned by parse_args. However, the optparse module guarantees that the take_action method of custom actions will always be passed a values object which provides an ensure_value method [12]_, while the argparse module allows attributes to be assigned to any object, e.g.::

  foo_object = ... parser.parse_args(namespace=foo_object) foo_object.some_attribute_parsed_from_command_line

  Modifying optparse to allow any object to be passed in would be difficult because simply passing the foo_object around instead of a Values instance will break existing custom actions that depend on the ensure_value method.

Because of issues like these, which made it unreasonably difficult for argparse to stay compatible with the optparse APIs, argparse was developed as an independent module. Given these issues, merging all the argparse features into optparse with no backwards incompatibilities seems unlikely.

Deprecation of getopt and optparse

There is still some debate over the best way (if at all) to encourage users to move from getopt and optparse to their replacement, argparse. The current recommendation of this PEP is the following conservative deprecation strategy:

• Python 3.2, Python 2.7 and any later Python 2.X releases -- PendingDeprecation warnings, which by default are not displayed, and documentation notes directing users of getopt and optparse to argparse.

• Python 3.3 -- Same as above

• Python 3.4 -- Deprecation warnings for getopt and optparse, which by default are displayed.

Though this is slower than the usual deprecation process, it seems prudent to avoid producing any casually visible Deprecation warnings until Python 3.X has had some additional time to attract developers.

Open Issues

The argparse module supports Python from 2.3 up through 3.2 and as a result relies on traditional %(foo)s style string formatting. It has been suggested that it might be better to use the new style {foo} string formatting [13]_. This seems like a good idea, but would break backwards compatibility for existing argparse-based scripts unless we can come up with a way to reasonably support both syntaxes.

References

.. [1] argparse (http://code.google.com/p/argparse/)

.. [2] getopt (http://docs.python.org/library/getopt.html)

.. [3] optparse (http://docs.python.org/library/optparse.html)

.. [4] argparse in IPython (http://mail.scipy.org/pipermail/ipython-dev/2009-April/005102.html)

.. [5] argparse in Debian (http://packages.debian.org/search?keywords=python-argparse)

.. [6] 2007-01-03 request for argparse in the standard library (http://mail.python.org/pipermail/python-list/2007-January/592646.html)

.. [7] 2009-06-09 request for argparse in the standard library (http://bugs.python.org/issue6247)

.. [8] 2009-09-10 request for argparse in the standard library (http://mail.python.org/pipermail/stdlib-sig/2009-September/000342.html)

.. [9] Fredrik Lundh response to [6]_ (http://mail.python.org/pipermail/python-list/2007-January/592675.html)

.. [10] optparse variable args (http://docs.python.org/library/optparse.html#callback-example-6-variable-arguments)

.. [11] parser.largs and parser.rargs (http://docs.python.org/library/optparse.html#how-callbacks-are-called)

.. [12] take_action values argument (http://docs.python.org/library/optparse.html#adding-new-actions)

.. [13] use {}-formatting instead of %-formatting (http://bugs.python.org/msg89279)

Copyright

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

• Previous message: [Python-Dev] IO module precisions and exception hierarchy
• Next message: [Python-Dev] PEP 389: argparse - new command line parsing module
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the Python-Dev mailing list