[Python-Dev] [Python-ideas] Remove GIL with CAS instructions? (original) (raw)

Phillip Sitbon phillip.sitbon+python-dev at gmail.com
Thu Oct 22 19:51:01 CEST 2009

I'd just like to point out some previous discussion about implementing the GIL as a critical section or semaphore on Windows since it's come up here (although not quite the center of the OP's proposal AFAICS):

http://bugs.python.org/issue6132 http://mail.python.org/pipermail/python-dev/2009-May/089746.html

Some of this is more low-level. I did see higher performance when using non-Event objects, although I have not had time to follow up and do a deeper analysis. The GIL flashing "problem" with critical sections can very likely be rectified with a call to Sleep(0) or YieldProcessor() for those who are worried about it. On the subject of fairness, I tested various forms of the GIL on my multi-threaded ISAPI extension, where every millisecond counts when under high concurrency, and fairness wasn't an issue for single- or multi-core systems. It may be anecdotal, but it also may be that the issue is somewhat over-blown.

It seems like these discussions come up in one form or another a few times a year and don't really get anywhere - probably because many people find that it's easier to just run one instance of Python on each core/processor. IPC is cheap (cPickle rocks!), and Python's memory footprint is acceptable by today's standards. Still, it is an interesting topic to many, myself included.

Also, many people keep talking about inefficiencies due to threads waking up to a locked GIL. I'd like to see examples of this- most of the time, the OS should know that the thread is contending on the lock object and it is skipped over. Granted, a thread may wake up just to release the GIL shortly thereafter, but that's why sys.setcheckinterval() is there for us to tinker with.

Anyway, enough of my $0.02.

2009/10/21 John Arbash Meinel <john at arbash-meinel.com>:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Kristján Valur Jónsson wrote: ...

This depends entirely on the platform and primitives used to implement the GIL. I'm interested in windows.  There, I found this article: http://fonp.blogspot.com/2007/10/fairness-in-win32-lock-objects.html So, you may be on to something.  Perhaps a simple C test is in order then?

I did that.  I found, on my dual-core vista machine, that running "release", that both Mutexes and CriticalSections behaved as you describe, with no "fairness".  Using a "semaphore" seems to retain fairness, however. "fairness" was retained in debug builds too, strangely enough. Now, Python uses none of these.  On windows, it uses an "Event" object coupled with an atomically updated counter.  This also behaves fairly. The test application is attached.

I think that you ought to sustantiate your claims better, maybe with a specific platform and using some test like the above. On the other hand, it shows that we must be careful what we use.  There has been some talk of using CriticalSections for the GIL on windows.  This test ought to show the danger of that.  The GIL is different than a regular lock.  It is a reverse-lock, really, and therefore may need to be implemented in its own special way, if we want very fast mutexes for the rest of the system (cc to python-dev) Cheers, Kristján I can compile and run this, but I'm not sure I know how to interpret the results. If I understand it correctly, then everything but "Critical Sections" are fair on my Windows Vista machine. To run, I changed the line "#define EVENT" to EVENT, MUTEX, SEMAPHORE and CRIT. I then built and ran in "Release" environment (using VS 2008 Express) For all but CRIT, I saw things like: thread       5532 reclaims GIL thread       5532 working 51234 units thread       5532 worked 51234 units: 1312435761 thread       5532 flashing GIL thread       5876 reclaims GIL thread       5876 working 51234 units thread       5876 worked 51234 units: 1312435761 thread       5876 flashing GIL Where there would be 4 lines for one thread, then 4 lines for the other thread. for CRIT, I saw something more like 50 lines for one thread, and then 50 lines for the other thread. This is Vista Home Basic, and VS 2008 Express Edition, with a 2-core machine. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrfKFAACgkQJdeBCYSNAANbuQCgudU0IChylofTwvUk/JglChBd 9gsAoIJHj63/CagKpduUsd68HV8pP3QX =CuUj -----END PGP SIGNATURE-----

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/phillip.sitbon%2Bpython-dev%40gmail.com

More information about the Python-Dev mailing list