[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build (original) (raw)
Nathaniel Smith njs at pobox.com
Fri Apr 12 19:43:51 EDT 2019
- Previous message (by thread): [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
- Next message (by thread): [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower at python.org> wrote:
On 10Apr2019 1917, Nathaniel Smith wrote: > It sounds like --with-pydebug has accumulated a big grab bag of > unrelated features, mostly stuff that was useful at some point for > some CPython dev trying to debug CPython itself? It's clearly not > designed with end users as the primary audience, given that no-one > knows what it actually does and that it makes third-party extensions > really awkward to run. If that's right then I think Victor's plan of > to sort through what it's actually doing makes a lot of sense, > especially if we can remove the ABI breaking stuff, since that causes > a disproportionate amount of trouble. Does it really cause a "disproportionate" amount of trouble? It's definitely not meant for anyone who isn't working on C code, whether in CPython, an extension or a host application. If you want to use third-party extensions and are not able to rebuild them, that's a very good sign that you probably shouldn't be on the debug build at all.
Well, here's what I mean by "disproportionate". Some of the costs of the ABI divergence are:
The first time I had to debug a C extension, I wasted a bunch of time trying to figure out how I was supposed to use Debian's 'python-dbg' package (the --with-pydebug build), before eventually figuring out that it was a red herring and what I actually wanted was the -dbgsym package (their equivalent of MSVC's /Zi /DEBUG files).
The extension loading machinery has extra code and complexity to track the two different ABIs. The package ecosystem does too, e.g. distutils needs to name extensions appropriately, and we need special wheel tags, and pip needs code to handle these tags: https://github.com/pypa/pip/blob/54b6a91405adc79cdb8a2954e9614d6860799ccb/src/pip/_internal/pep425tags.py#L106-L109
If you want some of the features of --with-pydebug that don't change the ABI, then you still have to rebuild third-party extensions to get at them, and that's a significant hassle. (I could do it if I had to, but my time has value.)
Everyone who uses ctypes to access a PyObject* has to include some extra hacks to handle the difference between the regular and debug ABIs. There are a few different versions that get copy/pasted around as folklore, and they're all pretty obscure. For example: https://github.com/pallets/jinja/blob/fd89fed7456e755e33ba70674c41be5ab222e193/jinja2/debug.py#L317-L334 https://github.com/johndpope/sims4-ai-engine/blob/865212e841c716dc4364e0dba286f02af8d716e8/core/framewrapper.py#L12-L41 https://github.com/python-trio/trio/blob/862ced04e1f19287e098380ed8a0635004c36dd1/trio/_core/_multierror.py#L282 And then if you want to test this code, it means you have to add a --with-pydebug build to your CI infrastructure...
I don't know how many people use Py_TRACE_REFS, but if we can't find anyone on python-dev who uses it then it must be pretty rare. If dropping Py_TRACE_REFS would let us converge the ABIs and get rid of all the stuff above, then that seems like a pretty good trade! But maybe the Windows C runtime issue will foil this...
>> The reason we ship debug Python binaries is because debug builds use a >> different C Runtime, so if you do a debug build of an extension module >> you're working on it won't actually work with a non-debug build of CPython. > > ...But this is an important point. I'd forgotten that MSVC has a habit > of changing the entire C runtime when you turn on the compiler's > debugging mode.
Technically they are separate options, but most project files are configured such that their Debug/Release switch affects both the compiler options (optimization) and the linker options (C runtime linkage).
So how do other projects handle this? I guess historically the main target audience for Visual Studio was folks building monolithic apps, where you can just rebuild everything with whatever options you want, and compared to that Python extensions are messier. But Python isn't the only project in this boat. Do ruby, nodejs, R, etc., all provide separate debug builds with incompatible ABIs on Windows, and propagate that information throughout their module/package ecosystem?
-n
-- Nathaniel J. Smith -- https://vorpus.org
- Previous message (by thread): [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
- Next message (by thread): [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]