| msg148759 - (view) |
Author: Richard Oudkerk (sbt) *  |
Date: 2011-12-02 16:29 |
| The attached patch makes pickle use an object's __qualname__ attribute if __name__ does not work. This makes nested classes, unbound instance methods and static methods picklable (assuming that __module__ and __qualname__ give the correct "address"). BTW, It would not be hard to make instance methods picklable (and, as a by-product, class methods) by giving instance methods a __reduce__ method equivalent to def __reduce__(self): return (getattr, (self.__self__, self.__name__)) Is there any reason not to make such a change? |
|
|
| msg148770 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-02 19:10 |
| > The attached patch makes pickle use an object's __qualname__ attribute > if __name__ does not work. > > This makes nested classes, unbound instance methods and static methods > picklable (assuming that __module__ and __qualname__ give the correct > "address"). Thanks. I'm not sure that's a good approach for protocol 3: some pickles created by Python 3.3 would not be readable by Python 3.2. That's why I initially added that idea to PEP 3154, for a hypothetical protocol 4. However, perhaps it is possible to use the same reduce/getattr trick you are proposing for instance methods? > BTW, It would not be hard to make instance methods picklable (and, as > a by-product, class methods) by giving instance methods a __reduce__ > method equivalent to > > def __reduce__(self): > return (getattr, (self.__self__, self.__name__)) > > Is there any reason not to make such a change? I don't see any personally. Actually, multiprocessing has a Pickler subclass called ForkingPickler which does something similar (in Lib/multiprocessing/forking.py). |
|
|
| msg148903 - (view) |
Author: Alexandre Vassalotti (alexandre.vassalotti) * |
Date: 2011-12-06 02:36 |
| This might not be the case anymore, but __module__ can sometime be None. There is some discussion about this in Issue 3657. We should define the expected behavior when this happens. Also, I don't think backward-compatibility of the protocol is a big issue if we use the getattr approach. I feel bumping the protocol version is only necessary if we introduce new opcodes or change their interpretation. |
|
|
| msg148920 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2011-12-06 14:52 |
| It looks like Issue 3657 is really about builtin methods (i.e. builtin_function_or_method objects where __self__ is not a module). It causes no problem for normal python instance methods. If we tried the getattr approach for builtin methods too then it should only be used as a fallback when saving as a global will not work. |
|
|
| msg168820 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-08-21 23:16 |
| There is a cute way to use operator.attrgetter to produce backwards compatible pickles using the qualname: import pickle, copyreg, operator, sys, pickletools, types class AttrGetter(object): def __init__(self, name): self.name = name def __call__(self): # pretend to be callable raise RuntimeError def __reduce__(self): return operator.attrgetter, (self.name,) def reduce_by_qualname(obj): mod = sys.modules[obj.__module__] first, rest = obj.__qualname__.split('.', 1) firstobj = getattr(mod, first) assert operator.attrgetter(rest)(firstobj) is obj return AttrGetter(rest), (firstobj,) # FunctionType defaults to save_global but uses fallback if it fails copyreg.pickle(types.FunctionType, reduce_by_qualname) class A(object): class B(object): class C(object): @staticmethod def foo(): print("foo foo foo") def bar(): print("bar bar bar") for obj in [A.B.C.foo, bar]: data = pickle.dumps(obj, 2) pickletools.dis(data) func = pickle.loads(data) assert func is obj func() This produces 0: \x80 PROTO 2 2: c GLOBAL 'operator attrgetter' 23: q BINPUT 0 25: X BINUNICODE 'B.C.foo' 37: q BINPUT 1 39: \x85 TUPLE1 40: q BINPUT 2 42: R REDUCE 43: q BINPUT 3 45: c GLOBAL '__main__ A' 57: q BINPUT 4 59: \x85 TUPLE1 60: q BINPUT 5 62: R REDUCE 63: q BINPUT 6 65: . STOP highest protocol among opcodes = 2 foo foo foo 0: \x80 PROTO 2 2: c GLOBAL '__main__ bar' 16: q BINPUT 0 18: . STOP highest protocol among opcodes = 2 bar bar bar |
|
|