[Python-Dev] Increasing the C-optimized pickle extensibility (original) (raw)
Pierre Glaser pierre.glaser at inria.fr
Fri Apr 26 08:32:31 EDT 2019
- Previous message (by thread): [Python-Dev] Any core dev event plans for EP19?
- Next message (by thread): [Python-Dev] Increasing the C-optimized pickle extensibility
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi All,
We (Antoine Pitrou, Olivier Grisel and myself) spent some efforts recently on enabling pickle extensions to extend the C-optimized Pickler instead of the pure Python one.
Pickle extensions have a crucial role in many distributed computing libraries: cloudpickle (https://github.com/cloudpipe/cloudpickle) for example is vendored in dask, pyspark, ray, and joblib. Early benchmarks show that relying on the C-optimized pickle yields significant serialization speed improvements (up to 30x faster). (draft PR of the CPickler-backed version of cloudpickle: https://github.com/cloudpipe/cloudpickle/pull/253)
To make extending the C Pickler possible, we are currently moving forward with a few enhancements to the public pickle API.
First, we are enabling Pickler subclasses to implement a reducer_override method, that will be have priority over the registered reducers in the dispatch_table and over the default handling of classes and functions. (PR link: https://github.com/python/cpython/pull/12499)
Then, we are adding a new keyword argument to save_reduce called state_setter. (consequently we allow a reducer's return value to have a new, 6th item). This state setter callable is useful to override programmatically the state updating behavior of an object, that would otherwise be restricted to its static
__setstate__method. (PR link: https://github.com/python/cpython/pull/12588)
The PR review process of these changes is in progress, and anyone is welcomed to chime in and share some thoughts.
The first addition is very non-invasive. We estimated that the second point did not require introducing a new opcode, as this change could be implemented as simple sequence of standard pickle instructions. We therefore think that it is not necessary to make this change dependent on the new protocol 5 proposed in PEP 574.
The key advantage in not creating a new opcode that this makes our change backward-compatible, meaning that 3.8-written pickles will not break because of our change if read using earlier Python versions.
OTOH, one might argue that a new OPCODE might
- make the code a little bit cleaner
- make it easier to interpret disassembled pickle strings.
If you are interested, here is an example of a disassembled pickle string using our currently proposed solution: https://github.com/pierreglaser/cpython/pull/2#issuecomment-486243350
Does anyone have an opinion on this?
Thanks,
Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20190426/6578a59e/attachment-0001.html>
- Previous message (by thread): [Python-Dev] Any core dev event plans for EP19?
- Next message (by thread): [Python-Dev] Increasing the C-optimized pickle extensibility
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]