[Python-Dev] What is the precise problem? [was: Reference cycles in Exception.traceback] (original) (raw)
Jim J. Jewett jimjjewett at gmail.com
Fri Mar 7 22:20:49 CET 2014
- Previous message: [Python-Dev] Reference cycles in Exception.__traceback__
- Next message: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Wed Mar 5 17:37:12 CET 2014, Victor Stinner wrote:
Python 3 now stores the traceback object in Exception.traceback and exceptions can be chained through Exception.context. It's convenient but it introduced tricky reference cycles if the exception object is used out of the except block.
... see Future.setexception() of the ayncio module.
... frame.clear() raises an RuntimeError if the frame is still running. And it doesn't break all reference cycles.
An obvious workaround is to store the traceback as text, but this operation is "expensive" especially if the traceback is only needed in rare cases.
I tried to write "views" of the traceback (and frames), but Exception.traceback rejects types other than traceback and traceback instances cannot be created. It's possible to store the traceback somewhere else and set Exception.traceback to None, but there is still the problem with chained exceptions.
Any idea for a generic fix to such problem?
Could you clarify what the problem actually is? I can imagine any of the following:
(A) Exceptions take a lot of memory, because of all the related details.
- But sometimes the details are needed, so there is no good solution.
(B) Exceptions take a lot of memory, because of all the related details. There is a common use case that knows it will never need certain types of details, and releasing just those details would save a lot of memory. But frame.clear() picks the wrong details to release, at least for this case.
So write another function (or even a method) that does work, and have your framework call it. (Also see (F))
Instead of saving the original exception, could you instead create and store a new (copied?) one, which obviously won't (yet) be referenced by the traceback you assign to it?
(C) Exceptions take a lot of memory, because of all the related details. There is a common use case that knows it can make do with a summary of certain types of details, and releasing just those details would save a lot of memory. But generating the summary is expensive.
- It would help to have the summarize method available.
- It would help to have feedback from gc saying when there is enough memory pressure to make this call worthwhile.
(D) Exceptions are not released until cyclic gc, and so they eat a lot of memory for a long time prior to that.
- This may be like case B
- Are there references that can be replaced by weak references?
- Are there references that you can replace with weak references when your framework stores the exception? (Also see (F))
(E) Exceptions are not released even during cyclic gc, because of ambiguity over which del to run first.
- This may be like case B or case E
- This may be a concrete use case for the close protocol.
close is similar to del, except that it promises not to care about order of finalization, and it is run eagerly. As soon as an instance is known to be in a garbage cycle, close should be run without worrying about whether other objects also have close or del methods. Hopefully, this will break the cycle, or at least reduce the number of objects with del methods. (Whether to require that close be idempotent, or to guarantee that it is run only once/instance -- that would be part of the bikeshedding.)
(F) You know what to delete (or turn into weakrefs), but can't actually do it without changing a type.
(F1) Why does Exception.traceback reject other objects which are neither tracebacks nor None?
- Can that restriction be relaxed?
- Can you create a mixin subtype of Exception, which relaxes the constraint, and gets used by your framework?
- Can the restriction on creating tracebacks be relaxed?
- Can traceback's restriction on frames' types be relaxed?
(F2) Do you need the original Exception? (see (B))
(F3) Do you care about frame.clear() raising a runtime exception? Could you suppress it (or, better, get clear() to raise something more specific, and suppress that)? It would still have released what memory it reasonably could.
-jJ
--
If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
- Previous message: [Python-Dev] Reference cycles in Exception.__traceback__
- Next message: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]