| msg127699 - (view) |
Author: Vasilis (vlachoudis) |
Date: 2011-02-01 14:04 |
| The ConfigParser class in 2.7 is almost >50 times slower than in the 2.6 which for large files it renders it almost unusable. Actually the speed decrease depends on the amount of the stored data Results from test program: Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) on 3.5GHz Fedora14 64bit machine ConfigParser 166.307140827 RawConfigParser 0.1887819767 Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:31) on 3.0GHz Fedora13 64bit machine ConfigParser 4.24494099617 RawConfigParser 0.172905921936 |
|
|
| msg127710 - (view) |
Author: Stefan Krah (skrah) *  |
Date: 2011-02-01 22:06 |
| If OrderedDict is used, the test case quickly uses 8GB of memory. With this change (I'm not suggesting this as a fix!), the timings are normal: Index: Lib/ConfigParser.py =================================================================== --- Lib/ConfigParser.py (revision 88298) +++ Lib/ConfigParser.py (working copy) @@ -92,6 +92,7 @@ except ImportError: # fallback for setup.py which hasn't yet built _collections _default_dict = dict +_default_dict = dict import re |
|
|
| msg127713 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-02 01:46 |
| Commenting-out the ``c.set(section,"item#%d"%(i),str(i))`` calls shows that that is not where the problem lies for the ConfigParser() class. The issue seems confined to ConfigParser.get(). The RawConfigParser() class doesn't seem to have the same issue. Looking at the 2.7 code for ConfigParser.get() shows that it is doing a lot more than just getting. For example, it does a full copy of defaults dictionary on every call !? I'll look at it more shortly. |
|
|
| msg127715 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-02 02:43 |
| Attaching a patch that fixes the algorithmic atrocities by using the Chainmap recipe: http://code.activestate.com/recipes/305268-chained-map-lookups |
|
|
| msg127726 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-02 08:39 |
| Fixed for 2.7 in r88318. Will make a similar fix for 3.1.4 and for 3.2.1. |
|
|
| msg127761 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-02 21:19 |
| Attaching patch for Python 3.2. Georg, I was think of waiting for 3.2.1 for this one, but it can go into 3.2.0 RC2 if you prefer. |
|
|
| msg127764 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2011-02-02 21:29 |
| 3.2.1 should be fine. |
|
|
| msg127766 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-02 21:37 |
| Fixed 3.1 in r88323. |
|
|
| msg128986 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2011-02-21 19:43 |
| See r88469 and r88470. |
|
|