[Python-Dev] Reworking the GIL (original) (raw)
Baptiste Lepilleur baptiste.lepilleur at gmail.com
Sat Nov 7 20:37:30 CET 2009
- Previous message: [Python-Dev] Reworking the GIL
- Next message: [Python-Dev] Reworking the GIL
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
2009/11/7 Antoine Pitrou <solipsis at pitrou.net>
[...] So, to sum it up, the way the current GIL manages to have good latencies is by issueing an unreasonable number of system calls on a contended lock, and potentially killing throughput performance (this depends on the OS too, because numbers under Linux are not so catastrophic). Ah, I remember reading this in the analysis that was published now!
I made another benchmark using one of my script I ported to python 3 (and simplified a bit). I only test the total execution time. Tests done on Windows XP SP3. Processor is an Intel Core 2 Quad Q9300 (4 cores).
You can get the script from: http://gaiacrtn.free.fr/py/benchmark-kwd-newgil/purekeyword-py26-3k.py Script + test doc (940KB): http://gaiacrtn.free.fr/py/benchmark-kwd-newgil/benchmark-kwd-newgil.tar.bz2
The threaded loop is: for match in self.punctuation_pattern.finditer( document ): word = document[last_start_index:match.start()] if len(word) > 1 and len(word) < MAX_KEYWORD_LENGTH: words[word] = words.get(word, 0) + 1 last_start_index = match.end() if word: word_count += 1
Here are the results:
-j0 (main thread only) 2.5.2 : 17.991s, 17.947s, 17.780s 2.6.2 : 19.071s, 19.023s, 19.054s 3.1.1 : 46.384s, 46.321s, 46.425s newgil: 47.483s, 47.605s, 47.512s
-j4 (4 consumer threads, main thread producing/waiting) 2.5.2 : 31.105s, 30.568s 2.6.2 : 31.550s, 30.599s 3.1.1 : 85.114s, 85.185s newgil: 48.428s, 49.217s
It shows that, on my platform for this specific benchmark:
- newgil manage to leverage a significant amount of parallelism (1.7) where python 3.1 does not (3.1 is 80% slower)
- newgil as a small impact on non multi-threaded execution (~1-2%) [may be worth investigating]
- 3.1 is more than 2 times slower than python 2.6 on this benchmark
- 2.6 is "only" 65% slower when run with multiple threads compared to the 80% slower of 3.1.
Newgil is a vast improvement as it manages to leverage the short time the GIL is released by finditer [if I understood correctly in 3.x regex release the GIL].
What's worry me is the single threaded performance degradation between 2.6 and 3.1 on this test. Could the added GIL release/acquire on each finditer call explain this? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20091107/68e38500/attachment-0001.htm>
- Previous message: [Python-Dev] Reworking the GIL
- Next message: [Python-Dev] Reworking the GIL
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]