[Python-Dev] Use of Cython (original) (raw)

Stefan Behnel stefan_ml at behnel.de
Sat Sep 1 18:10:31 EDT 2018

Yury,

given that people are starting to quote enthusiastically the comments you made below, let me set a couple of things straight.

Yury Selivanov schrieb am 07.08.2018 um 19:34:

On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:

I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency. I do think that adding a “Cython-lite” tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython. Speaking of which, Dropbox is working on a new compiler they call "mypyc". mypyc will compile type-annotated Python code to an optimized C.

That's their plan. Saying that "it will" is a bit premature at this point. The list of failed attempts at writing static Python compilers is rather long, even if you only count those that compile the usual "easy subset" of Python.

I wish them the best of luck and endurance, but they have a long way to go.

The first goal is to compile mypy with it to make it faster, so I hope that the project will be completed.

That's not "the first goal". It's the /only/ goal. The only intention of mypyc is to be able to compile and optimise enough of Python to speed up the kind or style of code that mypy uses.

Essentially, mypyc will be similar to Cython, but mypyc is a subset of Python, not a superset.

Which is bad, right? It means that there will be many things that simply don't work, and that you need to change your code in order to make it compile at all. Cython is way beyond that point by now. Even RPython will probably continue to be way better than mypyc for quite a while, maybe forever, who knows.

Interfacing with C libraries can be easily achieved with cffi.

Except that it will be fairly slow. cffi is not designed for static analysis but for runtime operations. You can obviously also use cffi from Cython – but then, why would you, if you can get much faster code much more easily without using cffi?

That being said, if someone wants to write a static cffi optimiser for Cython, why not, I'd be happy to help with my advice. The cool thing is that this can be improved gradually, because compiling the cffi code probably already works out of the box. It's just not (much) faster than when interpreted.

Being a strict subset of Python means that mypyc code will execute just fine in PyPy.

So does normal (non-subset) Python code. You can run it in PyPy, have CPython interpret it, or compile it with Cython if you want it to run faster in CPython, all without having to limit yourself to a subset of Python. Seriously, you make this sound like requiring users to rewrite their code to make it compilable with mypyc was a good thing.

They can even apply some optimizations to it eventually, as it has a strict and static type system.

In case "they" refers to PyPy here, then I remember the PyPy project stating very clearly that they are not interested in PEP-484 typing because it is completely irrelevant for their JIT. It's really best for them to ignore it.

That's similar for Cython, simply because PEP-484 typing isn't designed for optimisation at all, definitely not for C-level optimisation. Still, Cython can make some use of PEP-484 typing, if you use it to define specific C types. That allows normal execution in CPython, static analysis with PEP-484 analyser tools (e.g. PyCharm or mypy), and efficient optimisation by Cython. The best of all worlds. See the docs on how to do that, it's been supported for about a year now (and has been around in a similar, non-PEP-484 form for years before that PEP even existed).

I'd be more willing to start using mypyc+cffi in CPython stdlib eventually, than Cython now. Cython is a relatively complex and still poorly documented language.

You are free to improve the documentation or otherwise help us find and discuss concrete problems with it. Calling Cython a "poorly documented language" could easily feel offensive towards those who have put a lot of work into the documentation, wiki, tutorials, trainings and what not that help people use the language. Even stack overflow is getting better and better in documenting Cython these days, even though responses over there that describe work-arounds tend to get outdated fairly quickly.

Besides, don't forget that it's Python, so consider reading the Python documentation first if something is unclear. And maybe some documentation of C data types as well. (.5 wink)

I'm speaking from experience after writing thousands of lines of Cython in uvloop & asyncpg. In skillful hands Cython is amazing, but I'd be cautious to advertise and use it in CPython.

Why not? You didn't actually give any reasons for that.

I'm also -1 on using Cython to test C API. While writing C tests is annoying (I wrote a fair share myself), their very purpose is to make third-party tools/extensions more stable. Using a third-party tool to test C API to track regressions that break third-party tools feels wrong.

I don't understand that argument. What's wrong about using a tool that helps you get around writing boiler plate code? The actual testing does not need to be done by Cython at all, you can write it any way you like.

Stefan

More information about the Python-Dev mailing list