[Python-Dev] Fwd: Removal of GIL through refcounting removal. (original) (raw)

Josiah Carlson josiah.carlson at gmail.com
Tue Nov 4 23🔞06 CET 2008

On Tue, Nov 4, 2008 at 5:31 AM, Leonardo Santagada <santagada at gmail.com> wrote:

On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote:

There is no shortage of algorithms (such as matrix multiplication) that are parallelizable but not particularly good candidates for an IPC-based multiprocessing paradigm.

Ahh, but those algorithms aren't going to be written in Python; they are going to be written in C, and are going to manipulate memory directly. With such things, you can use standard Python threads, call into your C runtime, and release the GIL. Alternatively, you can use the mmap module to store your data, shared across multiple processes, using the same direct-memory access as you would with multiple threads and GIL release. That's the thing, right now you can't even use python to sketch one of those algorithms and for example compare performance with a non parallel version.

Why not? The data needs to come from somewhere, it doesn't appear from the aether into your RAM. And even if it does appear from nowhere, you can always use synthetic data to compare single process vs. multiprocess - in pure Python.

Also with four cores on common intel processor you might be happy with your algo being 4x faster while being quite easy to develop (not needing to use c).

Indeed, but it's not uncommon that using psyco you can get 2-4x (if not occasionally 10-100x), with similar (though generally superior) performance gains from using Pyrex/Cython. And really, after implementing your base algorithm in Python, translating it to C/C++ is generally pretty straightforward for anything not involving complex data structures or string parsing.

I personally think this will be too much work on

And that's up to every programmer to decide for themselves. I've done a few of the things I've mentioned (shared mmaps, C code controlled by Python, etc.), and have been very pleased with the speedups I attained. With better tools (Python, multiprocessing, Pyrex/Cython, ...) now available, I can't help but say, "back in my day, we just wrote it in C and were happy that we didn't have to use Fortran". ;)

More information about the Python-Dev mailing list