gh-130090: Support PGO for clang-cl by chris-eibl · Pull Request #129907 · python/cpython (original) (raw)

@chris-eibl You might be up for this as a project

Oh, I think this might be a massive community effort, but yeah, someone somewhere has to start it somehow :)

we're incredibly unlikely to switch the default compiler for our releases, due to other compatibility concerns (not least that many

Yeah, I immediately agreed with you on that part on dpo - and still do.

Switching to clang-cl for the official builds is a far too high risk, since the user base is so big - and the compatibilty is too undetermined.

If e.g. @zanieb lands astral-sh/python-build-standalone#549, we'd get feedback regarding compatibility issues and a much smaller user base is affected.

But there's some possibility that we might be able to use Clang just for the interpreter loop and continue using MSVC for public interfaces

This is an interesting approach and might help us with compatibility concerns. I remember another project did something similar to get computed gotos (or the like) into an MSVC build using MingW or clang. AFAIR, they built some *.c (or just one?) files with one compiler and the rest with MSVC. But PGO and LTO are then most certainly problematic.

(which goes as deep as the entire CRT and our custom memory allocators, IMHO

clang-cl just "shims" the compiler and linker. It #includes the MSVC headers, produces compatible libs and links with the MSVC runtime. I.e., it does not bring its own allocator and the like.

What at least the pyperformance tests show: the clang-cl artifacts work with MSVC compiled extensions, since at least some of them come pre-compiled from pypi (e.g. _psutil_windows.pyd). And when pip had to compile locally for me, it used MSVC to build the extension.

unless you can somehow prove that Clang does and will always produce identical behaviour to MSVC

Most probably, the clang team would consider this as a bug and try to fix it. I can't speak for them (and maybe I am totally wrong here), but IMHO we cannot risk that dependency without having any measure about the compatibily wrt to the Python user base. There is https://clang.llvm.org/docs/MSVCCompatibility.html, all the C++ parts there have no effect on Python.

However, just because it compiles and links doesn't mean all extensions out there continue to work without issues.

I think we'll have to do baby steps here and support early adopters by fixing issues they run into - as long as this does not produce too much churn in the code base.

other than persuading MSVC to improve their optimisations, which we're already doing

Very much appreciated!

There might be an existing issue somewhere - I've only had high-level discussions about it - but if you can't find one then start a new one.

Yeah, that discussion needs to happen in a separate issue, not this PR. I did a is:issue state:open clang-cl search but it did not reveal anything suitable.

Maybe you and / or @Fidget-Spinner want to be a patron(s) of it, since he fancied to use clang-cl for the official binaries?

OTOH, maybe we'd first start on dpo to find out, what early adopters need to get started?

Building with clang-cl IMHO is now pretty well supported.

Once we have #130040, there is at least one CI schedule that will help us to not break clang-cl builds.