[Python-Dev] pickling of large arrays (original) (raw)
Ralf W. Grosse-Kunstleve rwgk@yahoo.com
Thu, 20 Feb 2003 07:05:29 -0800 (PST)
- Previous message: [Python-Dev] pickling of large arrays
- Next message: [Python-Dev] pickling of large arrays
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
--- Scott Gilbert <xscottg@yahoo.com> wrote:
Have your reduce method return a 4-tuple (function, arguments, state, listitems) with:
function = a constructing function that takes the length of your array in bytes, and the type of the data in the array arguments = a 2-tuple specifying the bytes and type state = None listitems = an iterator that returns small chunks of memory at a time.
Hey, this is great! I am beginning to see the light.
I've been gone for a while, is this PEP going to be included in the final version of 2.3?
My little prototype below works with Python 2.3a2!
This is almost perfect. In C++ we can have two overloads. Highly simplified:
template class array { void append(T const& value); // the regular append void append(std::string const& value); // for unpickling };
This will work for all T (e.g. int, float, etc.) ... except T == std::string.
This leads me to find it unfortunate that append() is re-used for unpickling. How about:
If the object has a (say) unpickle_append method this is used by the unpickler instead of append or extend.
Ralf
import pickle
class int_array(object):
def init(self, elems): self.elems = list(elems)
def reduce(self): return (int_array_factory, (len(self.elems),), None, int_array_iter(self.elems))
def append(self, value): values = [int(x) for x in value.split(",")] self.elems.extend(values)
class int_array_iter(object):
def init(self, elems, buf_size=4): self.elems = elems self.buf_size = 4 self.i = 0
def iter(self): return self
def next(self): if (self.i >= len(self.elems)): raise StopIteration result = "" for i in xrange(self.buf_size): result+= str(self.elems[self.i]) + "," self.i += 1 if (self.i == len(self.elems)): break return result[:-1]
def int_array_factory(size): print "reserve:", size return int_array([])
f = int_array(range(11)) print "f.elems:", f.elems s = pickle.dumps(f) print s g = pickle.loads(s) print "g.elems:", g.elems
Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/
- Previous message: [Python-Dev] pickling of large arrays
- Next message: [Python-Dev] pickling of large arrays
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]