[Python-Dev] Add a developer mode to Python: -X dev command line option (original) (raw)

Victor Stinner victor.stinner at gmail.com
Mon Nov 13 11:08:06 EST 2017

Hi,

The discussion on DeprecationWarning reminded me my old idea of a "development mode" for CPython: -X dev. Since Brett likes it, I post it on python-dev. Last year, I posted this idea to python-ideas but my idea was rejected:

https://mail.python.org/pipermail/python-ideas/2016-March/039314.html

In short: python3.7 -X dev script.py behaves as: PYTHONMALLOC=debug python3.7 -Wd -b -X faulthandler script.py

The idea of -X dev is to enable "debug checks". Some of these checks are already enabled by default if CPython is compiled in debug mode. These checks help to debug bugs earlier.

The problem is that outside CPython development, Python developers use a Python binary from their Linux distributor or a binary installed from python.org: a binary compiled in release mode. Well, some Linux distributions provide a debug build, but the ABI is incompatible, and so is hard to use in practice.

Running the tests of your code in -X dev mode should help you to catch subtle bugs and prevent regressions if you upgrade Python.

The -X dev mode doesn't raise hard exceptions, but usually only emits warnings. It allows you to fix bugs one by one in your own code, without having to fix bugs in third party code.

If you control your whole code base, you may want to enable -X dev=strict which raises hard exceptions, to make sure that you are really safe.

My "-X dev" idea is not incompatible with Nick's PEP 565 "Show DeprecationWarning in main" and it's different: it's an opt-in option, while Nick wants to change the default behaviour.

Full copy of my email sent last year to python-ideas.

When I develop on CPython, I'm always building Python in debug mode using ./configure --with-pydebug. This mode enables a lot of extra checks which helps me to detect bugs earlier. The debug mode makes Python much slower and so is not the default. "python3" in Linux distributions is a binary compiled in release mode. When they also provide a binary compiled in debug mode, you will probably have issues to use your existing C extensions, since they all of them are compiled in release mode which is not compatible (you must recompile C extensions in debug mode).

I propose to add a "development mode" to Python, to get a few checks to detect bugs earlier: a new -X dev command line option. Example:

python3.6 -X dev script.py

Checks enabled by this flag must:

I propose to enable:

For example, I consider that enabling tracemalloc is too expensive (CPU & memory) and must not be enabled in -X dev.

I wrote a proof-of-concept: if -X dev is used, executable Python once again with more parameters. Basically, replace: python3.6 -X dev ... with PYTHONMALLOC=debug python3.6 -Wd -b -X faulthandler ...

The list of checks can be extended later. For example, we may enable the debug mode of asyncio: PYTHONASYNCIODEBUG=1.

The scope of the "developer mode" is unclear to me. Many modules (ex: asyncio) already have a debug mode. Would it be a bad idea to enable all debug modes of all modules? For example, http.client has a debug level: do you expect debug traces of the HTTP client when you develop your application?

IMHO the scope must be well defined: don't modify modules of the stdlib, only enable more debug flags in the Python core (warnings, unicode, memory allocators, etc.).

Maybe we even a need -X dev=strict which would be stricter:

Again, this list can be extended later.

Or maybe we need multiple level to control the quantity of debug traces, warnings, ... written into stdout/stderr?

In my experience, the problem with converting warnings to errors is that you don't control the code of your whole application. It's common that third party modules raise DeprecationWarning, ResourceWarning, etc. Even some modules of the Python stdlib raise DeprecatingWarning! For example, I recall that distutils raises a DeprecationWarning on Python 3.4 when importing the imp module.

"Similar" idea in other programming languages:

Victor

More information about the Python-Dev mailing list