[Python-Dev] exec/with thunk-handling proposal (original) (raw)

Moore, Paul Paul.Moore@atosorigin.com
Tue, 4 Feb 2003 13:07:37 -0000

holger krekel wrote:

Michael Hudson wrote: > holger krekel <pyth@devel.trillke.net> writes: > > I think we can may get away with only a "weak" keyword > > and allow the aforementioned encapsulation of execution=20 > > events into an object like this: > > > > exec expr [with params]: suite >=20 > Gut reaction: ugh! =20 then i guess you rather prefer new keywords. I thought that there is a general reluctance to introduce new keywords and many people dislike 'exec' for its existence. So reusing it (and we are dealing with execution aspects IMO) makes some sense to me.

I can see your point. Personally, I don't have any great problems with = "exec", so I'd like to understand better how your proposal integrates this new = use of exec with the existing use. And at some point, it will be necessary to = confirm that the parser can be made to distinguish the various uses (it seems to = me that extra lookahead will be needed).

> > where the expression is evaluated to return a > > "thunk" handler with these optional "execution" hooks: > > > > def enter(self): =20 > > "before suite start" > > > > def except(self, type, value, tb):=20 > > "swallow given exception, reraise if neccessary" > > > > def leave(self): > > """upon suite finish (not called if except=20 > > exists and an exception happened) > > """ > > > > The above "with" parameters (of the form name=3Dexpr, = comma-separated)=20 > > are bound in local (or global/nested) and handler instance=20 > > namespace. The 'suite' is what we call "thunk". > > > > The above logic allows clean timely finalization for > > multiple ressources: > > > > exec autoclose() with f1=3Dopen(name1), f2=3Dopen(name2, 'w'): > > for line in f1: > > ... > > f2.write(...) >=20 > That looks messy. =20 as compared to? I think i have seen messier ideas with all kinds of brackets and double-colons :-)

I agree with you here. This looks OK to me in terms of syntax. I'm not = so sure of the semantics - that may be the bit Michael thought was "messy". = Binding the with parameters in two places feels odd, and in particular injecting = the parameter names into the instance namespace is unusual (I know of no = other construct in Python which implicitly changes an instance namespace like = this).

And as far as the local namespace is concerned, consider

exec something() with f1=3D12:
    suite
# Out of the exec now - what is the value of f1???

I assume that the assignment to f1 is still in existence, otherwise = you're inventing a new way of defining a scope. But if the binding to f1 does = still exist, then that looks odd, because "exec ... with f1=3D12" reads as = if f1 is only in scope for the duration of the statement. You seem to lose either way...

> > which would execute as follows > > > > a) autoclose() instance is created and stored as the=20 > > "thunk"-handler >=20 > We need a name for these. I've been using "monitor" for a while, = but > I'm not sure it's that apposite. =20 So far i'd liked "execution handler" because 'enter/exit/except' are = IMHO execution events which you can hook into. Do you agree with calling them 'execution events' or how would you call them?=20

The way you've defined things, I see the logic - it's hooks into the = execution model. I'm not sure I'd have thought of that way of looking at it for = myself, though...

> > b) f1/f2 are stored as attributes on the autoclose instance > > > > c) f1/f2 are put into the local/global namespace (and nested = ones > > if rebinding is allowed) > > > > d) thunk executes (for line ...) > > > > e) autoclose 'leave' hook is called (with or without = exception) > > and is implemented like this: > > =20 > > def leave(self): > > for obj in self.dict.values():=20 > > obj.close()

But what if the object was more complex, and needed its own state? How = could it tell which were "its own" variables and which had been injected? = (Other than by hard coding a list of "its own" variables to ignore, which is = utterly horrible).

> > f) thunk handler is removed=20 >=20 > "Too much magic!" =20 Hmmm. That seems a bit unfair to me as many other proposals didn't = care=20 to elaborate the exact sequence. You probably are refering to the=20 "namespace interactions" as the other stuff is probably the same with any other proposal so far.

No, I agree with Michael here. Just because other proposals handwave and = don't give proper semantics doesn't mean that they don't also have too much = magic. In my book "unclear semantics" is a worse accusation than "too much = magic". But both are bad :-)

> > Because computing 'f1' may succeed but 'f2' can subsequently > > fail the assignments have to execute within "autoclose" > > control. =20

Hmm. In the equivalent try...finally construct, the idiom is to open the = file outside the try statement, so that exceptions when opening the file = don't get accidentally caught. You need to address this - particularly in the = case of multiple variables.

Would the following be the correct way to achieve this with your = patch?=20 =20 f1=3Dopen(inputfn) with autoclose(f1): f2 =3D open(outputfn, 'w') with autoclose(f2): for line in f1: ... f2.write(line) =20 I think there should be a better solution for multiple ressources.

In many ways, I agree with you. I originally wanted multiple = expressions, and assignments within the "with" statement. But people argued against them = (look back over the thread if you want the details). I honestly don't believe = that the with statement will ever please everyone - the best we can hope for = is to let it handle the simple cases well, with clean, well-defined semantics = with as few surprises as possible, and leave the remaining cases to the = existing try statement (or to a more ambitious thunk-based mechanism).

There is a real risk of slimming the with statement down to uselessness = by this philosophy. But this may simply indicate that the need is perceived rather than real :-)

> > Now on to the usage of the except hook. Nice use cases might be > > > > exec retry(maxretry=3D3, on=3DIOError):=20 > > # do network io-stuff > > or > > exec skipon(AttributeError, TypeError): > > someobject.notifyhook() > > > > but i am sure there are more. Exception handling is often > > ugly when inlined with the code. I think that stating=20 > > 'exception behaviour' up-front allows to write nice=20 > > readable constructs.=20 >=20 > I am still not convinced that an except hook is worth the = pain. > Can you implement those for me? =20 Maybe, i haven't looked at your implementation yet and there are some other projects pending :-)

There's no implementation of any form of except hook in existence yet (excluding the IEXEC patch). The key point here is that your explanation = of semantics above didn't cover except hooks. I'd like to see = except integrated into the semantics you describe, and then with that as a = basis I's like to see definitions of retry() and skip_on().

The idea looks interesting, but I'd like to see concrete code.

> yields already can't go in blocks with finally statements, right? =20 right.=20 =20 > Would you propose calling the enter method each time the = generator > resumed? =20 yes, neccessary for locking (as the above example tried to indicate).

But not for files. It's arguable that you need enter, exit, = yield, and resume hooks for full generality.

On the other hand, YAGNI makes more sense to me...

> > If there is interest i can probably modify my patch=20 > > to allow all of the proposed syntax so that you could=20 > > play around with it. =20

Working code for any of these ideas could only be a good thing.

User code which benefits from the new features is also good. The = auto-closing file example has been worked to death, and its main effect for me so far = has been to make me wonder whether we need any of these mechanisms :-) The = lock example is better, but still pretty borderline. I'd really, really like = to see a substantial, non-toy use case.

Actually, my main experience is C/C++ based, where blocks with many = memory allocations and the occasional open file is the canonical example. = Freeing and closing in reverse order, under all error conditions, is a pain. But = for this, the C++ "resource acquisition is initialisation" idiom works brilliantly. It's a shame it can't be made to work in Python.

Paul.