https://pythonhosted.org/futures/#concurrent.futures.wait

Yeah, and asyncio.gather is another example in the stdlib. Or there's twisted's DeferredList. Trio is unusual in effectively forcing all tasks to be run under a gather(), but the basic concept isn't unique at all.
 
Figuring out how to *display* an exception tree coherently is going to
be a pain (it's already problematic with just the linked list), but if
we can at least model exception trees consistently, then we'd be able
to share that display logic, even if the scenarios resulting in
MultiErrors varied.

It's true, it was a pain :-). And I'm sure it can be refined. But I think this is a reasonable first cut: https://github.com/python-trio/trio/blob/9e0df6159e55fe5e389ae5e24f9bbe51e9b77943/trio/_core/_multierror.py#L341-L388

Basically the algorithm there is:

if there's a __cause__ or __context__:
  print it (recursively using this algorithm)
  print the description line ("this exception was the direct cause" or whatever is appropriate)
print the traceback and str() attached to this exception itself
for each embedded exception:
  print "Details of embedded exception  {i}:"
  with extra indentation:
     print the embedded exception (recursively using this algorithm)
(+ some memoization to avoid infinite recursion on loopy structures)

Of course really complicated trainwrecks that send exception shrapnel flying everywhere can still be complicated to read, but ... that's why we make the big bucks, I guess. (My fingers initially typoed that as "that's why we make the big bugs". Just throwing that out there.) 

Currently that code has a hard-coded assumption that the only kind of exception-container is MultiError, but I guess it wouldn't be too hard to extend that into some kind of protocol that your  asymmetric CleanupError could also participate in? Like: an abstract exception container has 0 or 1 (predecessor exception, description) pairs [i.e., __cause__ or __context__], plus 0 or more (embedded exception, description) pairs, and then the printing code just walks the resulting tree using the above algorithm?

If this all works out at the library level, then one can start to imagine ways that the interpreter could potentially get benefits from participating:

- right now, if an exception that already has a __cause__ or __context__ is re-raised, then implicit exception chaining will  overwrite  the old __cause__ or __context__. Instead, it could check to see if there's already a non-None value there, and if so do exc.__context__ = MultiError([exc.__context__, new_exc]).

- standardizing the protocol for walking over an exception container would let us avoid fighting over who owns sys.excepthook, and make it easier for third-party exception printers (like IPython, pytest, etc.) to play along.

- MultiError.catch is a bit awkward. If we were going all-in on this then one can imagine some construct like

multitry:
    ... do stuff ...
    raise MultiError([ValueError(1), MultiError([TypeError(), ValueError(2)])])
multiexcept ValueError as exc:
    print("caught a ValueError", exc)
multiexcept TypeError:
    print("caught a TypeError and raising RuntimeError")
    raise RuntimeError

which prints

  caught a ValueError 1
  caught a TypeError and raising RuntimeError
  caught a ValueError 2

and then raises a RuntimeError with its __context__ pointing to the TypeError.

...but of course that's preeeeetty speculative at this point; definitely more python-ideas territory.

-n
-- 
Nathaniel J. Smith -- https://vorpus.org
">

(original) (raw)

On Tue, Jun 13, 2017 at 12:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

reporting failures from concurrent.futures.wait:

https://pythonhosted.org/futures/#concurrent.futures.wait


Yeah, and asyncio.gather is another example in the stdlib. Or there's twisted's DeferredList. Trio is unusual in effectively forcing all tasks to be run under a gather(), but the basic concept isn't unique at all.
Figuring out how to *display* an exception tree coherently is going to

be a pain (it's already problematic with just the linked list), but if

we can at least model exception trees consistently, then we'd be able

to share that display logic, even if the scenarios resulting in

MultiErrors varied.

It's true, it was a pain :-). And I'm sure it can be refined. But I think this is a reasonable first cut: https://github.com/python-trio/trio/blob/9e0df6159e55fe5e389ae5e24f9bbe51e9b77943/trio/_core/_multierror.py#L341-L388

Basically the algorithm there is:

if there's a __cause__ or __context__:
print it (recursively using this algorithm)
print the description line ("this exception was the direct cause" or whatever is appropriate)
print the traceback and str() attached to this exception itself
for each embedded exception:
print "Details of embedded exception {i}:"
with extra indentation:
print the embedded exception (recursively using this algorithm)
(+ some memoization to avoid infinite recursion on loopy structures)

Of course really complicated trainwrecks that send exception shrapnel flying everywhere can still be complicated to read, but ... that's why we make the big bucks, I guess. (My fingers initially typoed that as "that's why we make the big bugs". Just throwing that out there.)

Currently that code has a hard-coded assumption that the only kind of exception-container is MultiError, but I guess it wouldn't be too hard to extend that into some kind of protocol that your asymmetric CleanupError could also participate in? Like: an abstract exception container has 0 or 1 (predecessor exception, description) pairs [i.e., __cause__ or __context__], plus 0 or more (embedded exception, description) pairs, and then the printing code just walks the resulting tree using the above algorithm?

If this all works out at the library level, then one can start to imagine ways that the interpreter could potentially get benefits from participating:

- right now, if an exception that already has a __cause__ or __context__ is re-raised, then implicit exception chaining will overwrite the old __cause__ or __context__. Instead, it could check to see if there's already a non-None value there, and if so do exc.__context__ = MultiError([exc.__context__, new_exc]).

- standardizing the protocol for walking over an exception container would let us avoid fighting over who owns sys.excepthook, and make it easier for third-party exception printers (like IPython, pytest, etc.) to play along.

- MultiError.catch is a bit awkward. If we were going all-in on this then one can imagine some construct like

multitry:
... do stuff ...
raise MultiError([ValueError(1), MultiError([TypeError(), ValueError(2)])])
multiexcept ValueError as exc:
print("caught a ValueError", exc)
multiexcept TypeError:
print("caught a TypeError and raising RuntimeError")
raise RuntimeError

which prints

caught a ValueError 1
caught a TypeError and raising RuntimeError
caught a ValueError 2

and then raises a RuntimeError with its __context__ pointing to the TypeError.

...but of course that's preeeeetty speculative at this point; definitely more python-ideas territory.

-n

--
Nathaniel J. Smith -- https://vorpus.org