Issue 33652: Improve pickling of typing types (original) (raw)
Created on 2018-05-26 07:10 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (8)
Author: Serhiy Storchaka (serhiy.storchaka) * 
Date: 2018-05-26 07:10
The following PR makes pickles for typing types more portable. Type variables no longer use _find_name() and can be unpickled in 3.6. Subscripted generics no longer expose internals and can be unpickled in 3.6 and future versions with changed internal implementation.
Before this PR:
import pickle, pickletools, typing
pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))
0: \x80 PROTO 4 2: \x95 FRAME 30 11: \x8c SHORT_BINUNICODE 'typing' 19: \x94 MEMOIZE (as 0) 20: \x8c SHORT_BINUNICODE '_find_name' 32: \x93 STACK_GLOBAL 33: h BINGET 0 35: \x8c SHORT_BINUNICODE 'T' 38: \x86 TUPLE2 39: R REDUCE 40: . STOP highest protocol among opcodes = 4 pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4))) 0: \x80 PROTO 4 2: \x95 FRAME 198 11: \x8c SHORT_BINUNICODE 'copyreg' 20: \x8c SHORT_BINUNICODE '_reconstructor' 36: \x93 STACK_GLOBAL 37: \x8c SHORT_BINUNICODE 'typing' 45: \x94 MEMOIZE (as 0) 46: \x8c SHORT_BINUNICODE '_GenericAlias' 61: \x93 STACK_GLOBAL 62: \x8c SHORT_BINUNICODE 'builtins' 72: \x94 MEMOIZE (as 1) 73: \x8c SHORT_BINUNICODE 'object' 81: \x93 STACK_GLOBAL 82: N NONE 83: \x87 TUPLE3 84: R REDUCE 85: } EMPTY_DICT 86: ( MARK 87: \x8c SHORT_BINUNICODE '_inst' 94: \x88 NEWTRUE 95: \x8c SHORT_BINUNICODE '_special' 105: \x89 NEWFALSE 106: \x8c SHORT_BINUNICODE '_name' 113: N NONE 114: \x8c SHORT_BINUNICODE 'origin' 126: h BINGET 0 128: \x8c SHORT_BINUNICODE 'Union' 135: \x93 STACK_GLOBAL 136: \x8c SHORT_BINUNICODE 'args' 146: h BINGET 1 148: \x8c SHORT_BINUNICODE 'int' 153: \x93 STACK_GLOBAL 154: h BINGET 1 156: \x8c SHORT_BINUNICODE 'str' 161: \x93 STACK_GLOBAL 162: \x86 TUPLE2 163: \x8c SHORT_BINUNICODE 'parameters' 179: ) EMPTY_TUPLE 180: \x8c SHORT_BINUNICODE 'slots' 191: N NONE 192: \x8c SHORT_BINUNICODE 'module' 204: h BINGET 0 206: u SETITEMS (MARK at 86) 207: b BUILD 208: . STOP highest protocol among opcodes = 4
With this PR:
import pickle, pickletools, typing
pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))
0: \x80 PROTO 4 2: \x95 FRAME 13 11: \x8c SHORT_BINUNICODE 'typing' 19: \x8c SHORT_BINUNICODE 'T' 22: \x93 STACK_GLOBAL 23: . STOP highest protocol among opcodes = 4 pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))
0: \x80 PROTO 4 2: \x95 FRAME 66 11: \x8c SHORT_BINUNICODE '_operator' 22: \x8c SHORT_BINUNICODE 'getitem' 31: \x93 STACK_GLOBAL 32: \x8c SHORT_BINUNICODE 'typing' 40: \x8c SHORT_BINUNICODE 'Union' 47: \x93 STACK_GLOBAL 48: \x8c SHORT_BINUNICODE 'builtins' 58: \x94 MEMOIZE (as 0) 59: \x8c SHORT_BINUNICODE 'int' 64: \x93 STACK_GLOBAL 65: h BINGET 0 67: \x8c SHORT_BINUNICODE 'str' 72: \x93 STACK_GLOBAL 73: \x86 TUPLE2 74: \x86 TUPLE2 75: R REDUCE 76: . STOP highest protocol among opcodes = 4
If there is a chance it would be nice to merge these changes into 3.7. Otherwise either pickles created in 3.7.0 will be incompatible not only with older Python versions, but with future Python versions too, or we will need to add complex code for supporting specific 3.7.0 pickles in future versions.
Author: Ivan Levkivskyi (levkivskyi) *
Date: 2018-05-26 18:19
New changeset 09f3221fbbf72692308149054e4f7668b08b22eb by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master': bpo-33652: Improve pickle support in the typing module. (GH-7123) https://github.com/python/cpython/commit/09f3221fbbf72692308149054e4f7668b08b22eb
Author: miss-islington (miss-islington)
Date: 2018-05-26 18:38
New changeset d49862582ed3513debed6e919fd4f92e9d4eebbd by Miss Islington (bot) in branch '3.7': bpo-33652: Improve pickle support in the typing module. (GH-7123) https://github.com/python/cpython/commit/d49862582ed3513debed6e919fd4f92e9d4eebbd
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2018-05-26 21:19
There is a question -- what to do with all these getstate and setstate methods? They are part of the pickle protocol, but they are not used when define reduce. And they are not needed for supporting compatibility with older versions, because in 3.6 pickleable generic types were pickled by name. I think it is better to remove these methods.
Author: Ivan Levkivskyi (levkivskyi) *
Date: 2018-05-26 21:42
Yes, these are just legacy from times when TypeVars were serialized by value, not by identity like now. I think it should be safe to remove them. Would you like to make a PR?
Author: Ivan Levkivskyi (levkivskyi) *
Date: 2018-05-28 10:54
New changeset 97b523db7c79c18c48516fba9410014d9896abc4 by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master': bpo-33652: Remove getstate and setstate methods in typing. (GH-7144) https://github.com/python/cpython/commit/97b523db7c79c18c48516fba9410014d9896abc4
Author: miss-islington (miss-islington)
Date: 2018-05-28 11:21
New changeset 98b089e2a178a309f20ac9ff498b230407da1f47 by Miss Islington (bot) in branch '3.7': bpo-33652: Remove getstate and setstate methods in typing. (GH-7144) https://github.com/python/cpython/commit/98b089e2a178a309f20ac9ff498b230407da1f47
Author: Ivan Levkivskyi (levkivskyi) *
Date: 2018-05-28 12:57
I think this can be closed now.
History
Date
User
Action
Args
2022-04-11 14:59:00
admin
set
github: 77833
2018-05-28 12:57:46
levkivskyi
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2018-05-28 11:21:56
miss-islington
set
messages: +
2018-05-28 10:55:12
miss-islington
set
pull_requests: + <pull%5Frequest6782>
2018-05-28 10:54:59
levkivskyi
set
messages: +
2018-05-28 08:22:35
serhiy.storchaka
set
pull_requests: + <pull%5Frequest6778>
2018-05-26 21:42:26
levkivskyi
set
messages: +
2018-05-26 21:19:09
serhiy.storchaka
set
messages: +
2018-05-26 18:38:02
miss-islington
set
nosy: + miss-islington
messages: +
2018-05-26 18:21:00
miss-islington
set
pull_requests: + <pull%5Frequest6766>
2018-05-26 18:19:28
levkivskyi
set
messages: +
2018-05-26 07:17:13
serhiy.storchaka
set
keywords: + patch
stage: patch review
pull_requests: + <pull%5Frequest6758>
2018-05-26 07:10:15
serhiy.storchaka
create