Issue 1212921: option to allow reload to affect existing instances (original) (raw)
Created on 2005-06-01 18:18 by bdrosen, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (7)
Author: Brett Rosen (bdrosen)
Date: 2005-06-01 18:18
One of the commonly asked questions (especially when developing GUI apps) is how do I reload a modified python file and have the existing instances automatically get the changed behavior. (this greatly speeds up the development cycle.)
Unfortunately python doesn't support this without having to do the work yourself. To rectify this, I've implemented smart reloading as detailed below:
Smart reloading is defined as applying modified code to your already instantiated objects on reload.
Supporting smart reloading for python:
Make sure that you execute python with the -r flag if you want smart reloading to work by default. The semantics of reload work as normal
The normal model has been expanded by adding two special attributes to classes:
1 pqname : This is used to give a hint as to our context if we are a nested class (top level classes don't need to use it) It specifies the parent hierarchy, excluding the module name. ie:
class outer(object):
class middle(object):
__pqname__ = "outer"
class inner(object):
__pqname__ = "outer.middle"excluding this for inner classes can lead to confusion if you have multiple inner classes with the same name.
2 reloadMode : This can have one of the 3 values - "overlay", "clear" and "disable"
a) clear is the default if you are in reload mode (-r) and you don't specify anything. The behavior here is to clear the class dictionary before doing the reload of the class. b) overlay works like clear except that it doesn't clear the class dictionary first. c) disable gives you the classic python behavior and is the default if you are not in reload mode (no -r specified)
When using this you almost never want to reuse class names in a module file. For example:
class foo(object): ...
class foo(object): ...
would be bad. The second class foo replaces the first one, although with overlay it could be used as syntactic sugar for adding new methods to an existing class.
Author: Georg Brandl (georg.brandl) * 
Date: 2005-08-31 22:29
Logged In: YES user_id=1188172
Does anyone care to give a quick, obliterative judgement about this?
Author: Michael Hudson (mwh)
Date: 2005-09-01 09:38
Logged In: YES user_id=6656
My first thought is to plug my recipe:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
In fact, this seems pretty similar to what the patch does, on a quick skim.
Given that the bulk of the functionality can be implemented in Python, I'm not sure I support the further complication of type_new for this.
Author: Ted Czotter (tczotter)
Date: 2005-09-02 21:35
Logged In: YES user_id=1339137
Ahh - finally someone is interested in this one.
Michael's recipe does address some of the same problems solved by this patch, but not all.
Before I get into more detail, let's be clear about something. The reload wart is a problem of inconvenience, not the inability to do something. For ordinary development, when I make a change in one module, I save time and avoid mistakes if I don't have to: * reload other modules * restart all or part of my application and thereby have to recover my state Neither of these stop me from developing, they just slow me down.
So a good solution is going to be measured on three things: 1) How well it improves my efficiency as a developer. 2) The level of effort required to use the solution.
Ok.
So the core of the wart is that when you re-compile a class, a new class is created, instead of altering the definitions bound to the existing class. The symptoms are:
- Existing instances (like open dialogs or sockets) continue to use the old class and don't inherit new behavior.
- Existing subclasses refer to the old base class and any instances created in the future are still inheriting old behavior.
Michael's recipe addresses these symptoms.
Are there other important symptoms? Unfortunately, yes.
- OTHER REFERENCES TO THE OLD CLASS EXIST Where? All over the place. Consider this:
from mycompany.corporateframework.gui.widgets import Button, CheckBox
This module now points to the old Button and CheckBox.
Many calls to Button() and CheckBox() follow in methods
after this.
The recipe can't find and fix these references. If I was using the recipe, I'd have to reload all these modules. But then I compound the problem. Reloading these other modules creates still more new classes, causing dozens of other modules to need reloading. In the end, I'm better off just restarting the whole darn application.
I understand that I could take advantage of the fact that modules retain their identity and so if I use fully qualified names everywhere, I won't have this problem because I'd always be looking up the latest class. But realistically, who is going to type b = mycompany.corporateframework.gui.widgets.Button("OK") all the time instead of just: b = Button("OK") Making people use python "a certain way" that is not idiomatic (i.e. automatic, normal, how we do things every day) is a problem and lowers the usefulness of the solution. And I'm certainly not about to suggest that anybody go through our 30,000 lines of python code to make such idiomatic changes so the recipe can work.
Anyway, even if I did accept that tactic (and I don't) I'm still not out of the woods.
Consider warehouses and factories. Here you are storing pointers to classes in containers for later invocations. For example, In wx, the grid class requires that you register classes to be used later for in-place editing in a grid. All these registrations are pointing to old class definitions. To fix the problem, we have to re- register. Which usually means reloading some modules. Which means reloading some more modules - you get it by now.
So as a result, my first measures of success are not met very well by the recipe. Yes, I can type foo.DoLatestThing() at the console, but in real life I'm still going to have to restart my application a lot.
On the other hand, the patch addresses the cause of the problem instead of some of the symptoms.
****** The true cause is that re-compiled classes change their identity. ******
With this patch, they do not. They retain their identity just like modules do.
All the symptoms go away.
Now let's talk about the harm the recipe does. What harm you ask?
Well to make it work you have to make all your most-base classes use Michael's metaclass. Unfortunately, we already use metaclasses for a lot of things. And we use multiple inheritance a lot (we like mixins). How fast can you say "metaclass conflict"?
I like what metaclasses do and I'm happy that Python let's me extend "internal" behavior so neatly. But the metaclass conflict is another serious wart. If we can avoid it, good.
And I'm not happy that I seriously have to edit a few hundred class definitions in existing code to get the benefit of the recipe. **** The patch approach requires ZERO changes to existing applications. Z E R O ****
Brett and I work at the same company. He made this patch because we were wasting hundreds of hours debugging reload problems or just quitting the application and restarting. I've been using this patch now for 3 months. It has saved our company hundreds of man-hours and we didn't have to change one line of existing code to make it work.
If the community doesn't want to use it, I'm fine with that. I just want to make sure that you fully understand what you are giving up.
Ted Czotter CTO Lavastorm Technologies
Author: Michael Hudson (mwh)
Date: 2005-09-03 09:57
Logged In: YES user_id=6656
Oh, come on, my recipe could very clearly be changed to mutate the overridden class instead of assigning to its instances' class attribute.
My point is that there's very little the patch does that can't be done in Python. type_new is already far too complicated, this adds more complication.
The main advantage your patch has over my recipe is that it edits the default metaclass which obviates the need to edit any class statements and avoids metaclass conflict issues. Is that enough to override my objections above? Maybe, maybe not. I'll think about it.
(PS: a touch less bombast would be nice)
Author: Michael Hudson (mwh)
Date: 2005-09-03 09:57
Logged In: YES user_id=6656
Oh, come on, my recipe could very clearly be changed to mutate the overridden class instead of assigning to its instances' class attribute.
My point is that there's very little the patch does that can't be done in Python. type_new is already far too complicated, this adds more complication.
The main advantage your patch has over my recipe is that it edits the default metaclass which obviates the need to edit any class statements and avoids metaclass conflict issues. Is that enough to override my objections above? Maybe, maybe not. I'll think about it.
(PS: a touch less bombast would be nice)
Author: Martin v. Löwis (loewis) *
Date: 2006-04-15 09:02
Logged In: YES user_id=21627
I think a change of this nature requires a PEP. The current patch comes with no documentation, even though it introduces severa new APIs. From reading the patch, it is not clear to me what the consequences of this patch are (e.g. what happens if the layout of the type changes?); these consequences should be discussed in the PEP.
So I'm rejecting the patch now; not rejecting the feature per se, but just the path taken to get it into Python.