| msg203270 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2013-11-18 09:42 |
| The definition of a new nametuple creates a large Python script to create the new type. The code stores the code in a private attribute: namespace = dict(__name__='namedtuple_%s' % typename) exec(class_definition, namespace) result = namespace[typename] result._source = class_definition This attribute wastes memory, I don't understand the purpose of the attribute. It was not discussed in an issue, so I guess that there is no real use case: changeset: 68879:bffdd7e9265c user: Raymond Hettinger <python@rcn.com> date: Wed Mar 23 12:52:23 2011 -0700 files: Doc/library/collections.rst Lib/collections/__init__.py Lib/test/test_collections.py description: Expose the namedtuple source with a _source attribute. Can we just drop this attribute to reduce the Python memory footprint? |
|
|
| msg203271 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2013-11-18 09:43 |
| I found this issue while using my tracemalloc module to analyze the memory consumption of Python. On the Python test suite, the _source attribute is the 5th line allocating the memory memory: /usr/lib/python3.4/collections/__init__.py: 676.2 kB |
|
|
| msg203304 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2013-11-18 15:51 |
| > the 5th line allocating the memory memory oops, the 5th line allocating the *most* memory |
|
|
| msg203707 - (view) |
Author: Eric Snow (eric.snow) * |
Date: 2013-11-22 00:42 |
| As an alternative, how about turning _source into a property? |
|
|
| msg203870 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2013-11-22 20:55 |
| In a first version namedtuple had an argument (named echo or verbose) that would cause the source code to be printed out, for use at the interactive prompt. Raymond later changed it to a _source attribute, more easy to work with than printed output. About the other question you asked on the ML (why isn’t there a base NamedTuple class to inherit): this has been discussed on python-ideas IIRC, and people have written ActiveState recipes for that idea. It should be easy to find the ML archive links from the ActiveState posts. |
|
|
| msg203878 - (view) |
Author: Eric Snow (eric.snow) * |
Date: 2013-11-22 21:06 |
| A while back, because of those python-ideas discussions, Raymond added a link at the bottom of the namedtuple section of the docs at http://docs.python.org/3.4/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields. The link points to a nice recipe by Jan Kaliszewski. |
|
|
| msg213904 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-03-17 22:01 |
| > As an alternative, how about turning _source into a property? A class or an instance property? A class property requires a metaclass. I guess that each namedtuple type requires its own metaclass, right? |
|
|
| msg213946 - (view) |
Author: Eric Snow (eric.snow) * |
Date: 2014-03-18 07:33 |
| It does not necessarily require a metaclass. You can accomplish it using a custom descriptor: class classattr: def __init__(self, getter): self.getter = getter def __get__(self, obj, cls): return self.getter(cls) FWIW, this is a descriptor that may be worth adding somewhere regardless. |
|
|
| msg213949 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-03-18 08:57 |
| namedtuple_source.patch: Replace _source attribute wasting memory with a property generating the source on demand. The patch adds also unit test for the verbose attribute (which is public and documented, even it is said to be "outdated"). The patch removes also repr_fmt and num_fields parameters of the class definition template, compute these values using the list of fields. I suggested to change Python 3.4.1 and 3.5. Test script: --- import email import http.client import pickle import test.regrtest import test.test_os import tracemalloc import xmlrpc.server snap = tracemalloc.take_snapshot() with open("dump.pickle", "wb") as fp: pickle.dump(snap, fp, 2) --- With the patch, the memory footprint is reduced by 176 kB. |
|
|
| msg214003 - (view) |
Author: Eric Snow (eric.snow) * |
Date: 2014-03-18 18:40 |
| Also be sure the have Raymond's sign-off before committing anything for this. :) |
|
|
| msg214212 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2014-03-20 12:32 |
| FWIW, the "verbose" option is mentioned as outdated because the "_source" attribute was added. Also, there are real use cases, people are using the _source as writing it to a .py file so that the dynamic namedtuple generation step can be skipped on subsequent imports. This is useful when people want to avoid the use of eval or want to run cython on the code. The attribute can be "dropped". It is part of the API. Sorry, the memory use bugs you. It is bigger than typical docstrings but is not a significant memory consumer in most applications. I like the idea of dynamically generating the source upon lookup, but want to think about whether there are any unintended consequences to that space saving hack. |
|
|
| msg214219 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2014-03-20 13:02 |
| The size of the _source attribute is about 2k per namedtuple class: >>> from collections import namedtuple >>> Response = namedtuple('Response', ['code', 'msg', 'compressed', 'written']) >>> len(Response._source) 2174 |
|
|
| msg215398 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2014-04-02 20:28 |
| Victor, I don't think the added complexity is worth 2k per named tuple class. Every time I've gone down the path of lazy evaluation, I've paid an unexpected price for it down the road. If the savings were huge, it might be worth it, but that isn't the case here. This isn't really different than proposing that all docstring be in a separate module to be lazily loaded only when people look at help. |
|
|